Corpus comparison¶

compare_corpora(a, b, window=1000, features=None, labels=('A', 'B'), n_bootstrap=1000, seed=0, min_words=None)

Compares two corpora of Spanish texts by every feature of a text at once: which statistics tell the corpora apart, and by how much. For single words keyness does the same, for the distances between texts - delta.

The texts are cut in a row into windows of exactly window words by split_windows, so the windows of all texts have one length; an incomplete remainder at the end of a text and a text shorter than a window are dropped unless min_words allows shorter windows. The features of every window are computed by text_features or a function of one's own, and the tables of the two corpora are compared by compare_features of the anyTS core.

Galdós against Unamuno, three novels each from the corpus of literature: Marianela, Misericordia and Torquemada en la hoguera against Niebla, Abel Sánchez and La tía Tula.

CodeResult

from ests.corpus import compare_corpora
from ests.datasets import SpanishLiterature

sl = SpanishLiterature()
sl.download()
galdos = [
    record["text"]
    for record in sl.get_records(author="galdos")
    if record["title"] in ("Marianela", "Misericordia", "Torquemada en la hoguera")
]
unamuno = [
    record["text"]
    for record in sl.get_records(author="unamuno")
    if record["title"] in ("Niebla", "Abel Sánchez", "La tía Tula")
]

result = compare_corpora(galdos, unamuno, window=1000, labels=("Galdós", "Unamuno"))
columns = [
    "median_Galdós",
    "median_Unamuno",
    "ci_low",
    "ci_high",
    "cohen_d",
    "cliff_delta",
    "auc",
    "p_holm",
]
result[columns].head(5).round(3)

features = [
    "basic_letters_per_word",
    "readability_lix",
    "morph_p_gerund",
    "morph_polarity_Neg",
    "sents_mean",
    "punct_dash",
    "punct_exclamation",
    "diversity_yule_k",
]
result.loc[features, columns].round(3)

result.loc["sents_mean", ["n_Galdós", "n_Unamuno", "n_texts_Galdós", "n_texts_Unamuno"]].to_dict()

                  median_Galdós  median_Unamuno  ci_low  ci_high  cohen_d  cliff_delta    auc  p_holm
diversity_mtldw         105.455          66.179  37.104   43.799    3.260        0.988  0.994     0.0
diversity_mamtld        103.316          64.814  35.107   45.657    3.083        0.981  0.990     0.0
diversity_mtld          103.655          63.851  38.546   47.911    3.110        0.981  0.990     0.0
diversity_mattr           0.817           0.769   0.046    0.057    2.952        0.979  0.990     0.0
diversity_msttr           0.819           0.769   0.047    0.059    2.929        0.979  0.989     0.0

                        median_Galdós  median_Unamuno  ci_low  ci_high  cohen_d  cliff_delta    auc  p_holm
basic_letters_per_word          4.441           4.130   0.243    0.358    1.695        0.784  0.892     0.0
readability_lix                37.906          29.721   7.097   11.023    1.084        0.694  0.847     0.0
morph_p_gerund                  0.069           0.037   0.026    0.035    1.533        0.742  0.871     0.0
morph_polarity_Neg              0.019           0.028  -0.011   -0.009   -1.122       -0.559  0.221     0.0
sents_mean                     16.393          11.494   3.078    6.688    0.833        0.610  0.805     0.0
punct_dash                     19.000          46.000 -39.000  -14.000   -1.287       -0.605  0.197     0.0
punct_exclamation               9.000          24.000 -23.000   -2.000   -1.251       -0.637  0.181     0.0
diversity_yule_k              103.620         112.500 -17.620   -2.660   -0.676       -0.351  0.324     0.0

{'n_Galdós': 155, 'n_Unamuno': 117, 'n_texts_Galdós': 3, 'n_texts_Unamuno': 3}

Galdós has the richer vocabulary: the measures built on the number of distinct words - TTR and its transformations, MATTR, MTLD, the hapaxes - tell the authors apart with a delta near or above 0.9, while Yule's K, which weighs the frequent words, gives a medium effect (0.35), so the difference lies mostly in the rare vocabulary. His words and sentences are longer, with nearly twice as many gerunds among the verb forms. Unamuno writes in dialogue: two and a half times as many dashes and exclamation marks per 1000 words, and more negations. The cohesion tells them apart as well (result.filter(like="cohesion_", axis=0)): per 1000 words Unamuno opens more than three times as many sentences with a connector (cohesion_connectors_inter, 20 against 6) - about 2.4 times as many per sentence, his sentences being shorter - and he repeats more of the content words of a sentence in the next two (0.108 against 0.065). Part of the cohesion follows the length of the sentences: Spearman's ρ with sents_mean over the windows is 0.76 for Galdós and 0.71 for Unamuno for cohesion_content_overlap_all, the binary overlap of all the pairs of sentences, and -0.49 and -0.61 for the connectors opening a sentence, while the share of the content words repeated in the next two sentences is not tied to it (0.02 and -0.01).

The p-values take the 272 windows of six novels for independent (see the warning below), and by the same test two novels of Galdós, Marianela and Misericordia, differ in 52 features. The interval of the difference of the medians resamples whole novels and is the safer guide: for the length of a sentence it spans 3.1 to 6.7 words, where the windows alone would give 3.9 to 6.1.

Parameters¶

Parameter	Type	Default	Description
`a`	`list[str]`	-	Texts of the first corpus
`b`	`list[str]`	-	Texts of the second corpus
`window`	`int`	`1000`	Size of a window in words; `None` - the whole texts
`features`	`Callable`	`None`	Function of the features of a text; `None` - `text_features`
`min_words`	`int`	`None`	Smallest number of words in a window, at most `window`; `None` - a whole window, or one word when `window` is `None`

labels, n_bootstrap and seed go on to compare_features. compare_corpora checks them with check_comparison_params first, so a wrong name of a corpus or seed fails before any text is parsed.

Raises:

SourceTypeError - texts that are not a list of strings, features that cannot be called or give no mapping of numbers by name;
SourceError - a corpus without a window of min_words words or more, a text with a character without a UTF-8 form;
ParameterError - a window or min_words that is not an integer of at least one, min_words above window, labels, a number of samples or a seed that check_comparison_params rejects;
DatasetNotFoundError - the default features without the model installed.

Features¶

text_features(text, nlp=None)

Returns 163 features of a text prefixed by their source, 209 with a pipeline that has a parser; an undefined feature is nan.

Prefix	Features	Source
`basic_`	shares of long, complex, simple, mono- and polysyllabic words, of letters, spaces and punctuation marks; letters and syllables per word	`BasicStats`
`readability_`	every readability formula and the consensus grade	`ReadabilityStats`
`diversity_`	every measure of lexical diversity but Orlov's Z, unstable on a window	`DiversityStats`
`morph_`	shares of the values of every morphological feature (`morph_pos_NOUN`, `morph_mood_Sub`, `morph_polarity_Neg`) and the markers of Spanish (`morph_p_gerund`, `morph_p_mente_adverbs`)	`MorphStats` by the spaCy model
`sents_`	mean length of a sentence in words, standard deviation, coefficient of variation, autocorrelation of neighbouring lengths (`sentence_rhythm`) - the rhythm of the text	`SentsExtractor`
`punct_`	frequencies of the punctuation marks by type per 1000 words and the share of the inverted marks	`punctuation_profile`
`cohesion_`	overlaps of the nouns, arguments and content words of the sentences, givenness, temporal cohesion and the densities of the connectors by class, kind and place, but the overlaps between paragraphs, which the windows cut	`CohesionStats` by the spaCy model
`syntax_`	every syntactic statistic, with a pipeline that has a parser	`SyntaxStats`

Left out are the reading time, which only follows the number of words in a window, and the features that repeat others: the share of unique words of BasicStats, which repeats diversity_ttr, its words per sentence, which repeat sents_mean, and the markers of the moods of get_markers, which repeat morph_mood_*.

The parts of speech and the features of a single value - polarity, polite, poss, reflex - are shares of all the words (morph_polarity_Neg is the share of the negations), the other features shares of the words that carry them (morph_mood_Sub is the subjunctive among the moods). A value that does not occur in a window gives 0, a feature absent from the window altogether (no verbs - no tense) gives nan. A value of several values (PronType=Int,Rel of que) is shared equally among its parts, so the shares of a feature still sum to one.

The morphology and the cohesion are parsed by the model es_core_news_sm without its parser, so the marker morph_p_ser, which reads the copulas from the dependencies, and the group syntax_ are left out; a pipeline passed in nlp runs whole but for the entity recognizer, so functools.partial(text_features, nlp=get_nlp()) brings the parser, morph_p_ser and the syntax in, and another model goes into the comparison the same way. The parse takes the sentences of the group sents_ (set_sent_starts of the core), so the cohesion and the syntax count the same sentences as the rhythm. A novel is compared by windows and not whole, as the pipeline takes at most max_length characters.

The shares of spaces, letters and punctuation marks (basic_p_spaces, basic_p_letters, basic_p_punctuations) count the characters as they are: indents, double and non-breaking spaces of the files reflect the typesetting of an edition and not the text. In a corpus from different sources collapse them beforehand, for instance with re.sub(r"[^\S\n]+", " ", text).

Parameter	Type	Default	Description
`text`	`str`	-	Text string
`nlp`	`Language`	`None`	Pipeline for the morphology, the cohesion and, with a parser, the syntax; `None` - the default model without its parser

Raises:

SourceTypeError - a text that is not a string, a pipeline that is not a spaCy Language;
SourceError - a text without words, with a character without a UTF-8 form or longer than the max_length of the pipeline, a pipeline that gives no parts of speech or no lemmas;
DatasetNotFoundError - the default model is not installed.

corpus_features(texts, window=1000, features=text_features, min_words=None)

Returns the matrix of the features of the windows indexed by (number of the text, number of the window), for classifiers of one's own as well; the level text of the index is what the bootstrap of compare_features resamples. Texts without words and windows shorter than min_words are skipped. compare_corpora is corpus_features for each corpus followed by compare_features, and the split lets the features of several corpora be computed once and compared in pairs.

Parameter	Type	Default	Description
`texts`	`list[str]`	-	Texts of the corpus
`window`	`int`	`1000`	Size of a window in words; `None` - the whole texts
`features`	`Callable`	`text_features`	Function of the features of a text
`min_words`	`int`	`None`	Smallest number of words in a window, at most `window`; `None` - a whole window, or one word when `window` is `None`

Raises:

SourceTypeError - texts that are not a list of strings, features that cannot be called or give no mapping of numbers by name;
SourceError - a corpus without a window of min_words words or more, a text with a character without a UTF-8 form;
ParameterError - a window or min_words that is not an integer of at least one, min_words above window;
DatasetNotFoundError - the default features without the model installed.

The syntactic features come with a pipeline that has a parser, passed in a function of the features; features of one's own are passed the same way:

CodeResult

from functools import partial

from ests.corpus import compare_corpora, text_features
from ests.utils import get_nlp

features = partial(text_features, nlp=get_nlp())
result = compare_corpora(
    galdos, unamuno, window=1000, features=features, labels=("Galdós", "Unamuno")
)
result.filter(like="syntax_", axis=0)[columns].head(5).round(3)

                                    median_Galdós  median_Unamuno  ci_low  ci_high  cohen_d  cliff_delta    auc  p_holm
syntax_gerund_clauses_per_sent              0.145           0.045   0.088    0.119    1.460        0.848  0.924     0.0
syntax_complex_nominals_per_clause          0.809           0.550   0.213    0.307    1.158        0.688  0.844     0.0
syntax_tree_depth                           3.727           2.861   0.529    1.057    1.064        0.659  0.830     0.0
syntax_t_units_per_sent                     1.073           1.009   0.028    0.156    1.365        0.657  0.829     0.0
syntax_subtrees_per_sent                    6.742           4.494   1.484    2.921    0.932        0.656  0.828     0.0

Comparison of feature tables¶

compare_features(table_a, table_b, labels=('A', 'B'), n_bootstrap=1000, seed=0)

Compares two corpora feature by feature. Each corpus is a table of the features of its windows, parts of the texts of about equal size, so the features do not depend on text length. The result has one row per column, by descending absolute Cliff's delta, the features without statistics last. The bootstrap resamples whole texts by the index level text; without it every row is a text of its own. Every feature has a generator of its own, from the seed and the name of the feature, so its interval does not change with the other columns. A column missing from one table gives nan.

Parameter	Type	Default	Description
`table_a`	`DataFrame`	-	Features of the windows of the first corpus, one row per window
`table_b`	`DataFrame`	-	Features of the windows of the second corpus
`labels`	`tuple[str, str]`	`('A', 'B')`	Names of the corpora for the columns, two strings that give distinct columns
`n_bootstrap`	`int`	`1000`	Number of bootstrap samples
`seed`	`int` / `Generator`	`0`	Seed of the random number generator, a non-negative integer, or a numpy `Generator`; `None` - a random one

A table that is not a DataFrame, repeats the name of a column or has a column that is not numeric raises SourceTypeError; the other parameters are checked by check_comparison_params.

check_comparison_params(labels=('A', 'B'), n_bootstrap=1000, seed=0)

Checks the parameters of compare_features before the texts are processed: it raises ParameterError unless the labels are two strings that give distinct columns (("diff", "B") would repeat median_diff), the number of bootstrap samples is an integer of at least one and the seed is None, a non-negative integer or a numpy Generator.

Statistics¶

For a feature with the values \(x_1 \dots x_{n_A}\) in corpus A and \(y_1 \dots y_{n_B}\) in corpus B (undefined and infinite values dropped; with fewer than two values on a side - nan):

Column	Description
`mean_A`, `mean_B`, `median_A`, `median_B`	means and medians
`median_diff`, `ci_low`, `ci_high`	the difference of the medians and its 95% percentile bootstrap interval: both sets are resampled `n_bootstrap` times (`bootstrap_median_diff`)
`cohen_d`	\(d = (\bar{x} - \bar{y}) / s\), \(s\) - the pooled standard deviation; 0.2 - a small effect, 0.5 - medium, 0.8 - large (`calc_cohen_d`)
`cliff_delta`	\(\delta = P(x > y) - P(x < y)\) from −1 to 1; \(\lvert\delta\rvert\) < 0.147 - a negligible effect, < 0.33 - small, < 0.474 - medium, large otherwise (Romano et al. 2006; `calc_cliff_delta`)
`auc`	the feature as a classifier: the share of the pairs of windows with the value in A greater than in B, ties counted as half; \(\delta = 2 \cdot AUC - 1\), 0.5 - the feature does not tell the corpora apart
`u`, `p_value`	the Mann-Whitney U statistic and the two-sided p-value (`scipy.stats.mannwhitneyu`)
`p_holm`	the p-value with Holm's correction for the number of features (`holm_correction`)
`n_A`, `n_B`	number of windows with a defined value
`n_texts_A`, `n_texts_B`	number of texts behind those windows

The columns are anyts.corpus.COMPARISON_COLUMNS with the suffixes a and b replaced by the labels of the corpora. Cliff's delta and the AUC come from the same U statistic and agree; Cohen's d is sensitive to outliers and non-normality, so it is best read next to the delta.

Windows of one text are not independent

The test and the effect sizes take the windows for independent observations, which the windows of one text are not: with few texts the p-values are too small and reflect the chosen texts as much as the corpora. The bootstrap resamples whole texts (a cluster bootstrap), so its interval covers the spread between texts; it needs at least two texts on each side and is rough with few.

Functions of the statistics¶

The statistics of a row are available one by one from ests.corpus:

calc_cohen_d(values_a, values_b)

Cohen's d with the pooled sample variances (ddof=1); nan with fewer than two values on a side or without spread.

Parameter	Type	Default	Description
`values_a`	`list[float]`	-	Values in the first corpus
`values_b`	`list[float]`	-	Values in the second corpus

A value that is neither a number nor a missing one (None, NA) and values that are not a flat list raise SourceTypeError.

calc_cliff_delta(values_a, values_b)

Cliff's delta, the share of the pairs where the first value is greater minus the share where it is smaller; nan for an empty set or an undefined value. Computed by sorting, in \(O((n_A + n_B) \log n_B)\) time and linear memory.

Parameter	Type	Default	Description
`values_a`	`list[float]`	-	Values in the first corpus
`values_b`	`list[float]`	-	Values in the second corpus

A value that is neither a number nor a missing one (None, NA) and values that are not a flat list raise SourceTypeError.

bootstrap_median_diff(values_a, values_b, n_bootstrap=1000, rng=None, confidence=0.95, texts_a=None, texts_b=None)

The percentile bootstrap interval of the difference of the medians. With texts_a and texts_b whole texts are resampled, the median of a draw taken over the values of its texts; with fewer than two texts on a side the interval is nan.

Parameter	Type	Default	Description
`values_a`	`list[float]`	-	Values in the first corpus
`values_b`	`list[float]`	-	Values in the second corpus
`n_bootstrap`	`int`	`1000`	Number of bootstrap samples
`rng`	`Generator`	`None`	Random number generator of numpy; `None` - a new one without a seed
`confidence`	`float`	`0.95`	Confidence level
`texts_a`	`list`	`None`	Texts of the values in the first corpus; `None` - every value is resampled on its own
`texts_b`	`list`	`None`	Texts of the values in the second corpus

A value that is neither a number nor a missing one, values that are not a flat list and texts that are not labels of one type raise SourceTypeError; an n_bootstrap that is not an integer of at least one, a confidence that is not a number in (0, 1), texts not as many as the values and an rng that is not a numpy Generator raise ParameterError.

holm_correction(p_values)

Holm's correction for multiple comparisons: the p-values are sorted in ascending order, the i-th is multiplied by (m − i + 1), where m is the number of defined values, then the running maximum is taken and capped at one; nan stays nan.

Parameter	Type	Default	Description
`p_values`	`list[float]`	-	p-values

A p-value that is neither a number nor a missing one and p-values that are not a flat list raise SourceTypeError, and a p-value outside [0, 1] ParameterError.

CodeResult

from ests.corpus import calc_cliff_delta, calc_cohen_d, holm_correction

round(calc_cohen_d([2, 4, 6, 8], [1, 3, 5, 7]), 3)
calc_cliff_delta([2, 4, 6, 8], [1, 3, 5, 7])
holm_correction([0.01, 0.04, 0.03]).round(3).tolist()

0.387
0.25
[0.03, 0.06, 0.06]

Windows of a text¶

split_windows(text, window=1000, min_words=None)

Cuts a text into consecutive windows of exactly window words, so the features that depend on length are comparable between texts. The rest is a window only with at least min_words words (by default a whole window, so it is dropped). A boundary goes before the first word of a window and its opening marks; a straight quote or a dash glued to the last word of the previous window stays with it, and so does one after a closing mark when whitespace follows it ("cuatro", —dijo Juan—, "adiós."); a dash glued to the next word opens its line of dialogue (cuatro.—¿Cinco?). Windows are stripped of whitespace; window=None gives the whole text. It is the split_windows of the core with the words of WordsExtractor and the marks of ests.constants: quotes, brackets, dashes, ¿ and ¡ open a phrase (OPENING_MARKS), the straight quote and the dashes are symmetric (SYMMETRIC_MARKS); the marks that close a phrase are those of the core.

Parameter	Type	Default	Description
`text`	`str`	-	Text string
`window`	`int`	`1000`	Size of a window in words; `None` - the whole text
`min_words`	`int`	`None`	Smallest number of words in a window; `None` - a whole window, or one word when `window` is `None`

A text without words or shorter than min_words gives an empty list.

Raises:

SourceTypeError - a text that is not a string;
SourceError - a character without a UTF-8 form;
ParameterError - a window or min_words that is not an integer of at least one, min_words above the window.

CodeResult

from ests.corpus import split_windows

split_windows("Uno dos tres cuatro cinco", 2, min_words=1)
split_windows("El gato duerme mucho. ¿Dónde está el perro?", 4)
split_windows('Dijo "adiós" y se fue ya', 2)

['Uno dos', 'tres cuatro', 'cinco']
['El gato duerme mucho.', '¿Dónde está el perro?']
['Dijo "adiós"', 'y se', 'fue ya']

Rhythm of the sentences¶

sentence_rhythm(lengths, *, messages=MESSAGES)

Gives the rhythm of the sentences from their lengths in words: the mean sents_mean, the standard deviation sents_std, the coefficient of variation sents_cv and the lag-1 autocorrelation of neighbouring lengths sents_autocorr (Yule, 1939). The deviation needs two sentences, the autocorrelation three of unequal lengths, the coefficient of variation a mean above zero; otherwise, and without sentences, a feature is nan.

Parameter	Type	Default	Description
`lengths`	`list[int]`	-	Lengths of the sentences in words
`messages`	`dict[str, str]`	`MESSAGES`	Messages of the errors by key

Lengths that are not a list of integers raise SourceTypeError, a negative length ParameterError.

CodeResult

from ests.corpus import sentence_rhythm

{key: round(value, 3) for key, value in sentence_rhythm([4, 8, 2, 7]).items()}

{'sents_mean': 5.25, 'sents_std': 2.754, 'sents_cv': 0.525, 'sents_autocorr': -0.794}