Corpus comparison¶

compare_features(table_a, table_b, labels=('A', 'B'), n_bootstrap=1000, seed=0)

Compares two corpora feature by feature. Each corpus is a table of the features of its windows, parts of the texts of about equal size, so the features do not depend on text length. The result has one row per column, by descending absolute Cliff's delta, the features without statistics last. The bootstrap resamples whole texts by the index level text; without it every row is a text of its own. Every feature has a generator of its own, from the seed and the name of the feature, so its interval does not change with the other columns. A column missing from one table gives nan.

CodeResult

import pandas as pd

from anyts.corpus import compare_features

short = pd.DataFrame({"length": [14.0, 15.0, 13.0, 16.0]})
long = pd.DataFrame({"length": [33.0, 35.0, 31.0, 29.0]})
result = compare_features(short, long, labels=("short", "long"), n_bootstrap=200)
result.loc[
    "length", ["mean_short", "mean_long", "median_diff", "ci_low", "ci_high", "cliff_delta"]
].round(2).tolist()

[14.5, 32.0, -17.5, -20.5, -14.99, -1.0]

Parameters¶

Parameter	Type	Default	Description
`table_a`	`DataFrame`	-	Features of the windows of the first corpus, one row per window
`table_b`	`DataFrame`	-	Features of the windows of the second corpus
`labels`	`tuple[str, str]`	`('A', 'B')`	Names of the corpora for the columns, two strings that give distinct columns
`n_bootstrap`	`int`	`1000`	Number of bootstrap samples
`seed`	`int` / `Generator`	`0`	Seed of the random number generator, a non-negative integer, or a numpy `Generator`; `None` - a random one

A table that is not a DataFrame, repeats the name of a column or has a column that is not numeric raises SourceTypeError; the other parameters are checked by check_comparison_params.

check_comparison_params(labels=('A', 'B'), n_bootstrap=1000, seed=0)

Checks the parameters of compare_features before the texts are processed: it raises ParameterError unless the labels are two strings that give distinct columns (("diff", "B") would repeat median_diff), the number of bootstrap samples is an integer of at least one and the seed is None, a non-negative integer or a numpy Generator.

Statistics¶

For a feature with the values \(x_1 \dots x_{n_A}\) in corpus A and \(y_1 \dots y_{n_B}\) in corpus B (undefined and infinite values dropped; with fewer than two values on a side - nan):

Column	Description
`mean_A`, `mean_B`, `median_A`, `median_B`	means and medians
`median_diff`, `ci_low`, `ci_high`	the difference of the medians and its 95% percentile bootstrap interval: both sets are resampled `n_bootstrap` times (`bootstrap_median_diff`)
`cohen_d`	\(d = (\bar{x} - \bar{y}) / s\), \(s\) - the pooled standard deviation; 0.2 - a small effect, 0.5 - medium, 0.8 - large (`calc_cohen_d`)
`cliff_delta`	\(\delta = P(x > y) - P(x < y)\) from −1 to 1; \(\lvert\delta\rvert\) < 0.147 - a negligible effect, < 0.33 - small, < 0.474 - medium, large otherwise (Romano et al. 2006; `calc_cliff_delta`)
`auc`	the feature as a classifier: the share of the pairs of windows with the value in A greater than in B, ties counted as half; \(\delta = 2 \cdot AUC - 1\), 0.5 - the feature does not tell the corpora apart
`u`, `p_value`	the Mann-Whitney U statistic and the two-sided p-value (`scipy.stats.mannwhitneyu`)
`p_holm`	the p-value with Holm's correction for the number of features (`holm_correction`)
`n_A`, `n_B`	number of windows with a defined value
`n_texts_A`, `n_texts_B`	number of texts behind those windows

The columns are anyts.corpus.COMPARISON_COLUMNS with the suffixes a and b replaced by the labels of the corpora. Cliff's delta and the AUC come from the same U statistic and agree; Cohen's d is sensitive to outliers and non-normality, so it is best read next to the delta.

Windows of one text are not independent

The test and the effect sizes take the windows for independent observations, which the windows of one text are not: with few texts the p-values are too small and reflect the chosen texts as much as the corpora. The bootstrap resamples whole texts (a cluster bootstrap), so its interval covers the spread between texts; it needs at least two texts on each side and is rough with few.

Functions of the statistics¶

compare_values(values_a, values_b, n_bootstrap=1000, rng=None, texts_a=None, texts_b=None)

Compares two sets of values of one feature and returns one row in the order of anyts.corpus.COMPARISON_COLUMNS, p_holm left nan. Undefined and infinite values, missing ones (None, NA) included, are dropped with their texts.

Parameter	Type	Default	Description
`values_a`	`list[float]`	-	Values in the first corpus
`values_b`	`list[float]`	-	Values in the second corpus
`n_bootstrap`	`int`	`1000`	Number of bootstrap samples
`rng`	`Generator`	`None`	Random number generator of numpy; `None` - a new one without a seed
`texts_a`	`list`	`None`	Texts of the values in the first corpus for the bootstrap; `None` - every value a text of its own
`texts_b`	`list`	`None`	Texts of the values in the second corpus

A value that is neither a number nor a missing one, values that are not a flat list and texts that are not labels of one type raise SourceTypeError; an n_bootstrap that is not an integer of at least one, texts not as many as the values and an rng that is not a numpy Generator raise ParameterError.

calc_cohen_d(values_a, values_b)

Cohen's d with the pooled sample variances (ddof=1); nan with fewer than two values on a side or without spread.

Parameter	Type	Default	Description
`values_a`	`list[float]`	-	Values in the first corpus
`values_b`	`list[float]`	-	Values in the second corpus

A value that is neither a number nor a missing one (None, NA) and values that are not a flat list raise SourceTypeError.

calc_cliff_delta(values_a, values_b)

Cliff's delta, the share of the pairs where the first value is greater minus the share where it is smaller; nan for an empty set or an undefined value. Computed by sorting, in \(O((n_A + n_B) \log n_B)\) time and linear memory.

Parameter	Type	Default	Description
`values_a`	`list[float]`	-	Values in the first corpus
`values_b`	`list[float]`	-	Values in the second corpus

A value that is neither a number nor a missing one (None, NA) and values that are not a flat list raise SourceTypeError.

bootstrap_median_diff(values_a, values_b, n_bootstrap=1000, rng=None, confidence=0.95, texts_a=None, texts_b=None)

The percentile bootstrap interval of the difference of the medians. With texts_a and texts_b whole texts are resampled, the median of a draw taken over the values of its texts; with fewer than two texts on a side the interval is nan.

Parameter	Type	Default	Description
`values_a`	`list[float]`	-	Values in the first corpus
`values_b`	`list[float]`	-	Values in the second corpus
`n_bootstrap`	`int`	`1000`	Number of bootstrap samples
`rng`	`Generator`	`None`	Random number generator of numpy; `None` - a new one without a seed
`confidence`	`float`	`0.95`	Confidence level
`texts_a`	`list`	`None`	Texts of the values in the first corpus; `None` - every value is resampled on its own
`texts_b`	`list`	`None`	Texts of the values in the second corpus

A value that is neither a number nor a missing one, values that are not a flat list and texts that are not labels of one type raise SourceTypeError; an n_bootstrap that is not an integer of at least one, a confidence that is not a number in (0, 1), texts not as many as the values and an rng that is not a numpy Generator raise ParameterError.

holm_correction(p_values)

Holm's correction for multiple comparisons: the p-values are sorted in ascending order, the i-th is multiplied by (m − i + 1), where m is the number of defined values, then the running maximum is taken and capped at one; nan stays nan.

Parameter	Type	Default	Description
`p_values`	`list[float]`	-	p-values

A p-value that is neither a number nor a missing one and p-values that are not a flat list raise SourceTypeError, and a p-value outside [0, 1] ParameterError.

CodeResult

from anyts.corpus import calc_cliff_delta, calc_cohen_d, holm_correction

round(calc_cohen_d([2, 4, 6, 8], [1, 3, 5, 7]), 3)
calc_cliff_delta([2, 4, 6, 8], [1, 3, 5, 7])
holm_correction([0.01, 0.04, 0.03]).round(3).tolist()

0.387
0.25
[0.03, 0.06, 0.06]

Windows of a text¶

split_windows(text, window=1000, min_words=None, *, tokenize=None, opening_marks=OPENING_MARKS, symmetric_marks=SYMMETRIC_MARKS, closing_marks=CLOSING_MARKS, messages=MESSAGES)

Cuts a text into consecutive windows of exactly window words, so the features that depend on length are comparable between texts. The rest is a window only with at least min_words words (by default a whole window, so it is dropped). A boundary goes before the first word of a window and its opening marks; symmetric marks, such as a straight quote or the dash of an aside, close the previous window when glued to its last word, or after a closing mark when whitespace follows them; a dash glued to the next word opens a line of dialogue, and a run of such marks goes as a whole. Windows are stripped of whitespace; window=None gives the whole text. The words and their positions come from tokenize, by default the words of WordsExtractor.

Parameter	Type	Default	Description
`text`	`str`	-	Text string
`window`	`int`	`1000`	Size of a window in words; `None` - the whole text
`min_words`	`int`	`None`	Smallest number of words in a window; `None` - a whole window, or one word without a window
`tokenize`	`Callable`	`None`	Words of a text with their start and end positions
`opening_marks`	`Collection[str]`	`OPENING_MARKS`	Marks that open a phrase
`symmetric_marks`	`Collection[str]`	`SYMMETRIC_MARKS`	Opening marks that close a phrase as well
`closing_marks`	`Collection[str]`	`CLOSING_MARKS`	Marks that close a phrase
`messages`	`dict[str, str]`	`MESSAGES`	Messages of the errors by key

A text that is not a string, a tokenizer that is not callable or does not give the words in order as triples of integer bounds within the text and a string, and marks that are not a collection of strings raise SourceTypeError, a window or min_words that is not an integer of at least one, min_words above the window and a mark of more than one character ParameterError. A text without words or shorter than min_words gives an empty list.

The marks of the core are those of anyts.constants: quotes, brackets and dashes open a phrase, the straight quotes and the dashes are symmetric, and the final marks, the comma, the semicolon, the closing brackets, guillemet and quotes close one.

CodeResult

from anyts.corpus import split_windows

split_windows("One two three four five", 2, min_words=1)
split_windows('He said: "Go home." Then he left the room', 4)

['One two', 'three four', 'five']
['He said: "Go home."', 'Then he left the']

Rhythm of the sentences¶

sentence_rhythm(lengths, *, messages=MESSAGES)

Gives the rhythm of the sentences from their lengths in words: the mean sents_mean, the standard deviation sents_std, the coefficient of variation sents_cv and the lag-1 autocorrelation of neighbouring lengths sents_autocorr (Yule, 1939). The deviation needs two sentences, the autocorrelation three of unequal lengths, the coefficient of variation a mean above zero; otherwise, and without sentences, a feature is nan.

Parameter	Type	Default	Description
`lengths`	`list[int]`	-	Lengths of the sentences in words
`messages`	`dict[str, str]`	`MESSAGES`	Messages of the errors by key

Lengths that are not a list of integers raise SourceTypeError, a negative length ParameterError.

CodeResult

from anyts.corpus import sentence_rhythm

{key: round(value, 3) for key, value in sentence_rhythm([4, 8, 2, 7]).items()}

{'sents_mean': 5.25, 'sents_std': 2.754, 'sents_cv': 0.525, 'sents_autocorr': -0.794}