Corpus comparison¶

compare_corpora(a, b, window=1000, features=None, labels=('A', 'B'), n_bootstrap=1000, seed=0, min_words=None)

Compares two corpora of Russian texts by every feature of a text at once: which statistics tell authors, genres, translations, human and generated texts apart - and by how much. For single words keyness does the same, for the distances between texts - delta.

The texts are cut into windows of window words by split_windows, the features of every window come from text_features or a function of one's own, and compare_features of the anyTS core compares the tables of the two corpora.

Chekhov versus Tolstoy over the prose of the RussianLiterature dataset: 77 and 42 works, 249 and 1422 windows of 1000 words, about a minute.

CodeResult

from ruts.corpus import compare_corpora
from ruts.datasets import RussianLiterature

rl = RussianLiterature()
chekhov = list(rl.get_texts(genre="prose", author="Чехов"))
tolstoy = list(rl.get_texts(genre="prose", author="Толстой"))

result = compare_corpora(chekhov, tolstoy, window=1000, labels=("Чехов", "Толстой"))
columns = [
    "median_Чехов",
    "median_Толстой",
    "ci_low",
    "ci_high",
    "cohen_d",
    "cliff_delta",
    "auc",
    "p_holm",
]
result[columns].head(10).round(3)
result.loc["sents_mean", ["n_Чехов", "n_Толстой", "n_texts_Чехов", "n_texts_Толстой"]].to_dict()

                                        median_Чехов  median_Толстой  ci_low  ci_high  cohen_d  cliff_delta    auc  p_holm
cohesion_content_overlap_2_sent                0.067           0.109  -0.047   -0.035   -1.067       -0.837  0.081     0.0
cohesion_noun_overlap_2_sent                   0.081           0.141  -0.069   -0.050   -1.077       -0.759  0.121     0.0
cohesion_content_overlap_prop_all              0.015           0.027  -0.014   -0.009   -0.531       -0.751  0.125     0.0
cohesion_content_overlap_adjacent              0.190           0.342  -0.189   -0.126   -1.254       -0.717  0.141     0.0
punct_ellipsis                                15.000           2.000   6.000   25.000    1.868        0.706  0.853     0.0
cohesion_content_overlap_all                   0.089           0.181  -0.120   -0.069   -1.081       -0.706  0.147     0.0
cohesion_noun_overlap_adjacent                 0.106           0.218  -0.138   -0.087   -1.049       -0.668  0.166     0.0
cohesion_noun_overlap_all                      0.045           0.095  -0.071   -0.035   -0.846       -0.658  0.171     0.0
punct_exclamation                             14.000           4.000   7.000   14.000    1.671        0.657  0.829     0.0
cohesion_content_overlap_prop_adjacent         0.037           0.060  -0.026   -0.018   -0.838       -0.643  0.178     0.0
{'n_Чехов': 249, 'n_Толстой': 1422, 'n_texts_Чехов': 57, 'n_texts_Толстой': 41}

Cohesion sets the corpora apart most: Tolstoy repeats nouns and content words in adjacent and following sentences more often, and Chekhov has several times more ellipses and exclamations per 1000 words. The overlaps in adjacent and in all pairs of sentences partly repeat the sentence length (the Spearman ρ of cohesion_content_overlap_all with sents_mean over the windows is 0.90 for Chekhov and 0.75 for Tolstoy), while the share of the lemmas repeated in the next two sentences (cohesion_content_overlap_2_sent) repeats it only weakly (0.23 and 0.31). Twenty works of Chekhov and one of Tolstoy are shorter than a window, so the windows come from 57 and 41 works.

The p-values take the 1671 windows for independent (see the warning below): 111 of the 147 features have a corrected p-value below 0.01, but only 22 show a large effect by Cliff's delta. The interval of the difference of the medians resamples whole works and is the safer guide: it excludes zero for 81 of those 111 features, and for the length of a sentence it spans −5.2 to −1.6 words, where the windows alone would give −4.1 to −2.8.

Parameters¶

Parameter	Type	Default	Description
`a`	`list[str]`	-	Texts of the first corpus
`b`	`list[str]`	-	Texts of the second corpus
`window`	`int`	`1000`	Window size in words; `None` - whole texts
`features`	`Callable`	`None`	Text feature function; `None` - `text_features`
`min_words`	`int`	`None`	Smallest number of words in a window, at most `window`; `None` - a whole window, one word with `window=None`

labels, n_bootstrap and seed go on to compare_features and are checked before any text is processed.

Raises:

SourceTypeError - texts that are not a list of strings, features that cannot be called;
SourceError - a corpus without a window of min_words words or more;
ParameterError - a window or min_words that is not an integer of at least one, min_words above window, labels, a number of samples or a seed that check_comparison_params rejects.

Features¶

text_features(text, nlp=None)

Returns 147 features of a text prefixed by source, 192 with the pipeline nlp; an undefined feature is nan.

Prefix	Features	Source
`basic_`	shares of long, complex, simple, mono- and polysyllabic words, letters, spaces and punctuation marks; letters and syllables per word	`BasicStats`
`readability_`	all readability formulas and the consensus grade	`ReadabilityStats`
`diversity_`	lexical diversity measures, but for Orlov's Z, unstable on a window	`DiversityStats`
`morph_`	shares of parts of speech and of single-valued features among the words (`morph_pos_NOUN`, `morph_polarity_Neg`) and shares of values within the other features (`morph_case_Gen`; one share for a two-valued feature, `morph_number_Plur`)	`MorphStats` by ruMorph
`sents_`	mean sentence length in words, standard deviation, coefficient of variation, autocorrelation of adjacent lengths (`sentence_rhythm`)	`sentence_lengths`
`punct_`	frequencies of punctuation marks by type per 1000 words and the share of the letter ё	`punctuation_profile`
`cohesion_`	the overlap of the sentences, pronouns, givenness, temporal cohesion and connectors, without the overlaps between paragraphs (`*_para`)	`CohesionStats` by ruMorph
`syntax_`	all syntactic statistics of the parse, only with the pipeline `nlp`; the leaves, the subtrees, the longest dependency and the depth of the tree nearly repeat `sents_mean`	`SyntaxStats`

A 1000-word window takes about 0.04 s. The shares of spaces, letters and punctuation marks (basic_p_spaces, basic_p_letters, basic_p_punctuations) count the characters as they are, with the indents and double spaces of the edition; in a corpus from different sources collapse them beforehand, for example re.sub(r"[^\S\n]+", " ", text).

Parameter	Type	Default	Description
`text`	`str`	-	Text string
`nlp`	`Language`	`None`	spaCy pipeline with a parser and parts of speech for the `syntax_` features; `None` - without them

Raises:

SourceTypeError - a text that is not a string, an nlp that is not a spaCy Language object;
SourceError - a text without words, a text longer than the max_length of the pipeline, a pipeline without a parser or parts of speech.

corpus_features(texts, window=1000, features=text_features, min_words=None)

Returns the feature matrix of the windows indexed by (text number, window number), for your own classifiers as well; texts without words and windows shorter than min_words are skipped. compare_corpora is corpus_features for each corpus followed by compare_features; apart they compute the features once to compare several pairs of corpora, for example all authors pairwise.

Parameter	Type	Default	Description
`texts`	`list[str]`	-	Texts of the corpus
`window`	`int`	`1000`	Window size in words; `None` - whole texts
`features`	`Callable`	`text_features`	Text feature function
`min_words`	`int`	`None`	Smallest number of words in a window, at most `window`; `None` - a whole window, one word with `window=None`

Raises:

SourceTypeError - texts that are not a list of strings, features that cannot be called;
SourceError - a corpus without a window of min_words words or more;
ParameterError - a window or min_words that is not an integer of at least one, min_words above window.

Features of your own go to features as a function of a text, text_features with a pipeline among them; with the parse the same comparison takes about five minutes:

CodeResult

from functools import partial

from ruts import load_pipeline
from ruts.corpus import compare_corpora, text_features

nlp = load_pipeline()
result = compare_corpora(
    chekhov,
    tolstoy,
    window=1000,
    features=partial(text_features, nlp=nlp),
    labels=("Чехов", "Толстой"),
)
result.filter(like="syntax_", axis=0)[columns].head(5).round(3)

                                    median_Чехов  median_Толстой  ci_low  ci_high  cohen_d  cliff_delta    auc  p_holm
syntax_converb_clauses_per_sent            0.101           0.207  -0.138   -0.086   -1.034       -0.608  0.196     0.0
syntax_participle_clauses_per_sent         0.040           0.117  -0.098   -0.056   -0.877       -0.586  0.207     0.0
syntax_nominal_chains_per_sent             0.075           0.164  -0.121   -0.062   -0.819       -0.579  0.210     0.0
syntax_tree_depth                          3.023           3.634  -0.906   -0.405   -1.012       -0.541  0.229     0.0
syntax_mean_clause_len                     4.965           5.808  -1.257   -0.479   -0.886       -0.534  0.233     0.0

Comparison of feature tables¶

compare_features(table_a, table_b, labels=('A', 'B'), n_bootstrap=1000, seed=0)

Compares two corpora feature by feature. Each corpus is a table of the features of its windows, parts of the texts of about equal size, so the features do not depend on text length. The result has one row per column, by descending absolute Cliff's delta, the features without statistics last. The bootstrap resamples whole texts by the index level text; without it every row is a text of its own. Every feature has a generator of its own, from the seed and the name of the feature, so its interval does not change with the other columns. A column missing from one table gives nan.

Parameter	Type	Default	Description
`table_a`	`DataFrame`	-	Features of the windows of the first corpus, one row per window
`table_b`	`DataFrame`	-	Features of the windows of the second corpus
`labels`	`tuple[str, str]`	`('A', 'B')`	Names of the corpora for the columns, two strings that give distinct columns
`n_bootstrap`	`int`	`1000`	Number of bootstrap samples
`seed`	`int` / `Generator`	`0`	Seed of the random number generator, a non-negative integer, or a numpy `Generator`; `None` - a random one

A table that is not a DataFrame, repeats the name of a column or has a column that is not numeric raises SourceTypeError; the other parameters are checked by check_comparison_params.

check_comparison_params(labels=('A', 'B'), n_bootstrap=1000, seed=0)

Checks the parameters of compare_features before the texts are processed: it raises ParameterError unless the labels are two strings that give distinct columns (("diff", "B") would repeat median_diff), the number of bootstrap samples is an integer of at least one and the seed is None, a non-negative integer or a numpy Generator.

Statistics¶

For a feature with the values \(x_1 \dots x_{n_A}\) in corpus A and \(y_1 \dots y_{n_B}\) in corpus B (undefined and infinite values dropped; with fewer than two values on a side - nan):

Column	Description
`mean_A`, `mean_B`, `median_A`, `median_B`	means and medians
`median_diff`, `ci_low`, `ci_high`	the difference of the medians and its 95% percentile bootstrap interval: both sets are resampled `n_bootstrap` times (`bootstrap_median_diff`)
`cohen_d`	\(d = (\bar{x} - \bar{y}) / s\), \(s\) - the pooled standard deviation; 0.2 - a small effect, 0.5 - medium, 0.8 - large (`calc_cohen_d`)
`cliff_delta`	\(\delta = P(x > y) - P(x < y)\) from −1 to 1; \(\lvert\delta\rvert\) < 0.147 - a negligible effect, < 0.33 - small, < 0.474 - medium, large otherwise (Romano et al. 2006; `calc_cliff_delta`)
`auc`	the feature as a classifier: the share of the pairs of windows with the value in A greater than in B, ties counted as half; \(\delta = 2 \cdot AUC - 1\), 0.5 - the feature does not tell the corpora apart
`u`, `p_value`	the Mann-Whitney U statistic and the two-sided p-value (`scipy.stats.mannwhitneyu`)
`p_holm`	the p-value with Holm's correction for the number of features (`holm_correction`)
`n_A`, `n_B`	number of windows with a defined value
`n_texts_A`, `n_texts_B`	number of texts behind those windows

The columns are anyts.corpus.COMPARISON_COLUMNS with the suffixes a and b replaced by the labels of the corpora. Cliff's delta and the AUC come from the same U statistic and agree; Cohen's d is sensitive to outliers and non-normality, so it is best read next to the delta.

Windows of one text are not independent

The test and the effect sizes take the windows for independent observations, which the windows of one text are not: with few texts the p-values are too small and reflect the chosen texts as much as the corpora. The bootstrap resamples whole texts (a cluster bootstrap), so its interval covers the spread between texts; it needs at least two texts on each side and is rough with few.

Functions of the statistics¶

The statistics of a row are available one by one from ruts.corpus:

calc_cohen_d(values_a, values_b)

Cohen's d with the pooled sample variances (ddof=1); nan with fewer than two values on a side or without spread.

Parameter	Type	Default	Description
`values_a`	`list[float]`	-	Values in the first corpus
`values_b`	`list[float]`	-	Values in the second corpus

A value that is neither a number nor a missing one (None, NA) and values that are not a flat list raise SourceTypeError.

calc_cliff_delta(values_a, values_b)

Cliff's delta, the share of the pairs where the first value is greater minus the share where it is smaller; nan for an empty set or an undefined value. Computed by sorting, in \(O((n_A + n_B) \log n_B)\) time and linear memory.

Parameter	Type	Default	Description
`values_a`	`list[float]`	-	Values in the first corpus
`values_b`	`list[float]`	-	Values in the second corpus

A value that is neither a number nor a missing one (None, NA) and values that are not a flat list raise SourceTypeError.

bootstrap_median_diff(values_a, values_b, n_bootstrap=1000, rng=None, confidence=0.95, texts_a=None, texts_b=None)

The percentile bootstrap interval of the difference of the medians. With texts_a and texts_b whole texts are resampled, the median of a draw taken over the values of its texts; with fewer than two texts on a side the interval is nan.

Parameter	Type	Default	Description
`values_a`	`list[float]`	-	Values in the first corpus
`values_b`	`list[float]`	-	Values in the second corpus
`n_bootstrap`	`int`	`1000`	Number of bootstrap samples
`rng`	`Generator`	`None`	Random number generator of numpy; `None` - a new one without a seed
`confidence`	`float`	`0.95`	Confidence level
`texts_a`	`list`	`None`	Texts of the values in the first corpus; `None` - every value is resampled on its own
`texts_b`	`list`	`None`	Texts of the values in the second corpus

A value that is neither a number nor a missing one, values that are not a flat list and texts that are not labels of one type raise SourceTypeError; an n_bootstrap that is not an integer of at least one, a confidence that is not a number in (0, 1), texts not as many as the values and an rng that is not a numpy Generator raise ParameterError.

holm_correction(p_values)

Holm's correction for multiple comparisons: the p-values are sorted in ascending order, the i-th is multiplied by (m − i + 1), where m is the number of defined values, then the running maximum is taken and capped at one; nan stays nan.

Parameter	Type	Default	Description
`p_values`	`list[float]`	-	p-values

A p-value that is neither a number nor a missing one and p-values that are not a flat list raise SourceTypeError, and a p-value outside [0, 1] ParameterError.

CodeResult

from ruts.corpus import calc_cliff_delta, calc_cohen_d, holm_correction

round(calc_cohen_d([2, 4, 6, 8], [1, 3, 5, 7]), 3)
calc_cliff_delta([2, 4, 6, 8], [1, 3, 5, 7])
holm_correction([0.01, 0.04, 0.03]).round(3).tolist()

0.387
0.25
[0.03, 0.06, 0.06]

Windows of a text¶

split_windows(text, window=1000, min_words=None)

Cuts a text into consecutive windows of exactly window words, the words of WordsExtractor, so the features that depend on length are comparable between texts. The rest is a window only with at least min_words words (by default a whole window, so it is dropped). A boundary goes before the first word of a window and the quotes, brackets and dashes that open it, so the punctuation stays with its phrase.

Parameter	Type	Default	Description
`text`	`str`	-	Text string
`window`	`int`	`1000`	Size of a window in words; `None` - the whole text
`min_words`	`int`	`None`	Smallest number of words in a window; `None` - a whole window, or one word without a window

A text without words or shorter than min_words gives an empty list.

Raises:

SourceTypeError - a text that is not a string;
ParameterError - a window or min_words that is not an integer of at least one, min_words above the window.

CodeResult

from ruts.corpus import split_windows

split_windows("Раз два три четыре пять", 2, min_words=1)
split_windows("Он сказал: «Иди домой». Потом он вышел из комнаты", 4)

['Раз два', 'три четыре', 'пять']
['Он сказал: «Иди домой».', 'Потом он вышел из']

Rhythm of the sentences¶

sentence_rhythm(lengths)

Gives the rhythm of the sentences from their lengths in words: the mean sents_mean, the standard deviation sents_std, the coefficient of variation sents_cv and the lag-1 autocorrelation of neighbouring lengths sents_autocorr (Yule, 1939). The deviation needs two sentences, the autocorrelation three of unequal lengths, the coefficient of variation a mean above zero; otherwise, and without sentences, a feature is nan.

Parameter	Type	Default	Description
`lengths`	`list[int]`	-	Lengths of the sentences in words

Raises:

SourceTypeError - lengths that are not a list of integers;
ParameterError - a negative length.

CodeResult

from ruts.corpus import sentence_rhythm

{key: round(value, 3) for key, value in sentence_rhythm([4, 8, 2, 7]).items()}

{'sents_mean': 5.25, 'sents_std': 2.754, 'sents_cv': 0.525, 'sents_autocorr': -0.794}