Stylometry¶

Measures of stylometry and authorship attribution: distances between texts by the frequencies of the most frequent words (Burrows's Delta and its variants, as in stylo) and along a text by its windows (rolling Delta), markers of preferred and avoided words (Zeta) and their shares in the segments of corpora, Kilgarriff's chi-square distance between corpora and the Mendenhall curve. The functions work on lists of units of a text: lower-case word forms (the usual choice for Delta), lemmas or character N-grams - case and lemmatization belong to the extractors.

The functions are those of the anyTS core, with the profile of the function words of Russian added as a feature of an author. The units of a text come from WordsExtractor - word forms and lemmas - and CharNgramsExtractor.

Burrows's Delta¶

frequency_table(corpus, n_mfw=100, culling=0.0)

Builds the table of relative frequencies of a corpus, a dictionary "name of a text → units": rows are the texts, columns the n_mfw most frequent units by descending mean relative frequency (alphabetically when equal); culling keeps the units that occur in at least the given share of the texts, as in stylo.

Parameter	Type	Default	Description
`corpus`	`dict[str, list[str]]`	-	Units of the texts by the names of the texts
`n_mfw`	`int`	`100`	Number of the most frequent units; `None` - all of them
`culling`	`float`	`0.0`	Smallest share of the texts a unit occurs in

A corpus that is not a mapping of lists of strings raises SourceTypeError; an empty corpus, a text without units and no unit left after culling raise SourceError, and an n_mfw that is not an integer of at least one and a culling that is not a number within [0, 1] ParameterError.

z_scores(table)

Standardizes the columns of a table of frequencies with the sample standard deviation, like scale() of R; a column with the same frequency in every text gives zeros.

Parameter	Type	Default	Description
`table`	`DataFrame`	-	Table of relative frequencies

delta(corpus, n_mfw=100, variant='burrows', culling=0.0)

Computes a symmetric matrix of distances from the z-scores (a DataFrame with the names of the texts); at least three texts are needed - with two, the z-scores degenerate to ±1/√2 and the distances do not depend on the frequencies.

The variants (anyts.constants.DELTA_VARIANTS), with the formulas of the sources of stylo; \(n\) is the number of units, \(z_A\) and \(z_B\) the vectors of z-scores of the texts:

Variant	Key	Formula	Source
Burrows's Delta	`burrows`	\(\frac{1}{n} \sum_i \lvert z_{A,i} - z_{B,i} \rvert\)	Burrows (2002), `dist.delta`
Quadratic Delta	`quadratic`	\(\frac{1}{n} \sqrt{\sum_i (z_{A,i} - z_{B,i})^2}\)	Argamon (2008), `dist.argamon`
Eder's Delta	`eder`	\(\sum_i \frac{n - i + 2}{n} \lvert z_{A,i} - z_{B,i} \rvert\), \(i\) - rank of the unit by frequency	Eder, `dist.eder`
Cosine Delta	`cosine`	\(1 - \frac{z_A \cdot z_B}{\lVert z_A \rVert \lVert z_B \rVert}\)	Smith and Aldridge (2011), Evert et al. (2015), `dist.wurzburg`

Cosine Delta clusters the texts by author best in the experiments of Evert et al. The usual number of units is 100 to 500 most frequent words, 100-200 for character N-grams.

Parameter	Type	Default	Description
`corpus`	`dict[str, list[str]]`	-	Units of the texts by the names of the texts
`n_mfw`	`int`	`100`	Number of the most frequent units; `None` - all of them
`variant`	`str`	`'burrows'`	Variant of Delta of `anyts.constants.DELTA_VARIANTS`
`culling`	`float`	`0.0`	Smallest share of the texts a unit occurs in

The corpus, n_mfw and culling raise the errors of frequency_table; fewer than three texts raise SourceError, and an unknown variant ParameterError.

delta_profiles(reference, samples, n_mfw=100, variant='burrows', culling=0.0, statistics=None)

Distances by Delta for authorship attribution. The most frequent units, the culling and the statistics of the z-scores come from the reference texts (the profiles of the authors) or from a separate set statistics, such as training windows when the profiles are too few to estimate the spread of the frequencies; the texts under test samples get the same units and scaling. The result is the distances from the samples (rows) to the references (columns); the nearest reference is the presumed author. Unlike in delta, a sample affects neither the units nor the scaling, so its result does not depend on the other samples.

Parameter	Type	Default	Description
`reference`	`dict[str, list[str]]`	-	Units of the reference texts by the names of the texts
`samples`	`dict[str, list[str]]`	-	Units of the texts under test by the names of the texts
`n_mfw`	`int`	`100`	Number of the most frequent units; `None` - all of them
`variant`	`str`	`'burrows'`	Variant of Delta of `anyts.constants.DELTA_VARIANTS`
`culling`	`float`	`0.0`	Smallest share of the texts a unit occurs in
`statistics`	`dict[str, list[str]]`	`None`	Texts of the units and of the statistics of the scaling; `None` - the reference texts

The corpora, n_mfw and culling raise the errors of frequency_table; fewer than three texts for the statistics raise SourceError, and an unknown variant ParameterError.

CodeResult

from ruts import WordsExtractor
from ruts.corpus import delta, delta_profiles, frequency_table

texts = {
    "А": "Кот сидел на окне и смотрел на птиц. Птицы улетели, и кот уснул на окне.",
    "Б": "Собака лежала на полу и дремала. Потом собака ела и снова дремала на полу.",
    "В": "Завтра кот снова будет сидеть на окне и смотреть на птиц, а собака будет дремать.",
}
we = WordsExtractor(lowercase=True)
corpus = {name: we.extract(text) for name, text in texts.items()}

frequency_table(corpus, n_mfw=5).round(3)
delta(corpus, n_mfw=5).round(3)
delta(corpus, n_mfw=5, variant="cosine").round(3)

sample = {"?": we.extract("Кот проснулся на окне и снова смотрел на птиц.")}
delta_profiles(corpus, sample, n_mfw=5).round(3)

      на      и  собака    кот   окне
А  0.200  0.133   0.000  0.133  0.133
Б  0.143  0.143   0.143  0.000  0.000
В  0.133  0.067   0.067  0.067  0.067

       А      Б      В
А  0.000  1.563  1.277
Б  1.563  0.000  1.033
В  1.277  1.033  0.000

       А      Б      В
А  0.000  1.782  1.452
Б  1.782  0.000  1.202
В  1.452  1.202  0.000

       А     Б     В
?  0.364  1.66  1.16

Rolling Delta¶

rolling_delta(reference, text, window=5000, step=1000, n_mfw=100, variant='burrows', culling=0.0, statistics=None)

Delta along a text, after rolling.delta of stylo: the reference texts and the text under test are cut into windows of window units every step units, the rest shorter than a window dropped. The most frequent units, the culling and the statistics of the z-scores come from the windows of the reference texts, or of a separate set statistics, and every window of the text is compared by the variant of Delta with the centroid of the windows of every reference text. The result is the distances from the windows (rows, by the position of their middle in units) to the references (columns); where the nearest reference changes along the text, so does the style - another author of a part of a text written together, a borrowed passage. Unlike stylo, which sums the differences of the raw frequencies, the distances are those of delta_profiles over z-scores.

Scaled by their own windows, the centroids of the references balance each other, and Cosine Delta loses its sense: the cosine distances of a window to the references average 1 (weighted by their windows and the lengths of their centroids), with two references they add up to 2 and the lines are mirrored, so a passage of a third author looks halfway between them. With few references, Cosine Delta needs statistics of more texts - other authors, other works; Burrows's and Eder's Delta are not bound so. With statistics a single reference is enough.

Parameter	Type	Default	Description
`reference`	`dict[str, list[str]]`	-	Units of the reference texts by the names of the texts
`text`	`list[str]`	-	Units of the text under test
`window`	`int`	`5000`	Size of a window in units
`step`	`int`	`1000`	Distance between the starts of the windows in units
`n_mfw`	`int`	`100`	Number of the most frequent units; `None` - all of them
`variant`	`str`	`'burrows'`	Variant of Delta of `anyts.constants.DELTA_VARIANTS`
`culling`	`float`	`0.0`	Smallest share of the windows a unit occurs in
`statistics`	`dict[str, list[str]]`	`None`	Texts whose windows give the units and the statistics of the scaling; `None` - the reference texts

A corpus that is not a mapping of lists of strings and units of the text that are not a list of strings raise SourceTypeError. No reference texts, fewer than two without statistics, a text without units or shorter than a window, fewer than three windows of the statistics and no unit left after culling raise SourceError. An unknown variant, a window, a step or an n_mfw that is not an integer of at least one, a step greater than the window and a culling that is not a number within [0, 1] raise ParameterError.

CodeResult

from ruts.corpus import rolling_delta

rolling_delta(corpus, corpus["А"] + corpus["Б"], window=6, step=3, n_mfw=10).round(3)

        А      Б      В
3   0.197  1.132  0.801
6   0.296  1.132  0.702
9   0.292  1.138  0.801
12  0.197  1.132  0.801
15  0.988  1.230  1.048
18  0.804  0.330  1.015
21  1.298  0.165  1.509
24  1.298  0.165  1.509

Zeta¶

zeta(target, comparison, segment_size=2000, top_n=None)

Markers of preferred and avoided words after Burrows (2007) and Craig and Kinney (2009). Every text of both corpora is split into segments of about segment_size words (the length over the size rounded half up, at least one segment); \(DP\) is the share of the segments of a corpus with the word. Zeta is \(DP_{target} - DP_{comparison}\), from −1 to 1 (zeta.craig in stylo; the classic Zeta of Craig, \(DP_{target} + (1 - DP_{comparison})\), is greater by one); the logarithmic Zeta is \(\log_2 \frac{DP_{target}}{DP_{comparison}}\) (Schöch et al. 2018), a zero share replaced by half a segment. The list goes from the words the target corpus prefers to those it avoids.

Parameter	Type	Default	Description
`target`	`list[str]` / `list[list[str]]`	-	Words of the target corpus - one text or a list of texts
`comparison`	`list[str]` / `list[list[str]]`	-	Words of the comparison corpus
`segment_size`	`int`	`2000`	Size of a segment in words
`top_n`	`int`	`None`	Number of words from the start of the list; `None` - all of them

The result is a list of ZetaScore(word, dp_target, dp_comparison, zeta, log_zeta) named tuples by descending Zeta, by descending logarithmic Zeta and alphabetically when equal. A corpus that is neither a list of strings nor a list of such lists raises SourceTypeError, and a corpus without words SourceError; a segment_size or a top_n that is not an integer of at least one raises ParameterError.

CodeResult

from ruts.corpus import zeta

zeta(corpus["А"], corpus["Б"], segment_size=5, top_n=2)
zeta(corpus["А"], corpus["Б"], segment_size=5)[-1]

[ZetaScore(word='кот', dp_target=0.6666666666666666, dp_comparison=0.0, zeta=0.6666666666666666, log_zeta=2.0), ZetaScore(word='окне', dp_target=0.6666666666666666, dp_comparison=0.0, zeta=0.6666666666666666, log_zeta=2.0)]
ZetaScore(word='собака', dp_target=0.0, dp_comparison=0.6666666666666666, zeta=-0.6666666666666666, log_zeta=-2.0)

zeta_segments(corpora, preferred, avoided, segment_size=2000)

The shares of the markers of Zeta in the segments of corpora, as oppose of stylo places the segments: every text is split into segments as in zeta, and a segment gets the shares of the words of the lists preferred and avoided found in it. With the markers of a target corpus against a comparison corpus, the segments of the target lie high on the left of the plane of the two shares and those of the comparison low on the right; the segments of a disputed text show which group it is near (zeta_scatter_plot).

Parameter	Type	Default	Description
`corpora`	`dict[str, list[str] / list[list[str]]]`	-	Words of the corpora by name - one text or a list of texts each
`preferred`	`Collection[str]`	-	Words the target corpus prefers - the start of the list of `zeta`
`avoided`	`Collection[str]`	-	Words the target corpus avoids - the end of the list
`segment_size`	`int`	`2000`	Size of a segment in words

The result is a DataFrame of the segments in the order of the corpora and of their texts, with the columns corpus (the name of the corpus), preferred and avoided (the shares of the lists). Corpora that are not a mapping and a corpus or a list of markers that is not a list of strings raise SourceTypeError; no corpora, an empty list of markers and a corpus without words SourceError, and a segment_size that is not an integer of at least one ParameterError.

CodeResult

from ruts.corpus import zeta_segments

scores = zeta(corpus["А"], corpus["Б"], segment_size=5)
preferred = [score.word for score in scores if score.zeta > 0]
avoided = [score.word for score in scores if score.zeta < 0]
zeta_segments(corpus, preferred, avoided, segment_size=5).round(3)

  corpus  preferred  avoided
0      А      0.444    0.000
1      А      0.556    0.000
2      А      0.444    0.000
3      Б      0.111    0.429
4      Б      0.000    0.571
5      Б      0.111    0.429
6      В      0.111    0.143
7      В      0.222    0.000
8      В      0.111    0.143

Kilgarriff's chi-square¶

kilgarriff_chi2(words_a, words_b, n_mfw=500)

The distance between two corpora after Kilgarriff (2001): for the n_mfw most frequent words of the joint corpus the expected frequencies in the corpora are proportional to their sizes, \(\chi^2 = \sum (O - E)^2 / E\) over the words and both corpora. The greater the value, the more the corpora differ; it grows with the size of the corpora, so only pairs of equal sizes are comparable.

Parameter	Type	Default	Description
`words_a`	`list[str]`	-	Words of the first corpus
`words_b`	`list[str]`	-	Words of the second corpus
`n_mfw`	`int`	`500`	Number of the most frequent words of the joint corpus

Words that are not a list of strings raise SourceTypeError, a corpus without words SourceError, and an n_mfw that is not an integer of at least one ParameterError.

CodeResult

from ruts.corpus import kilgarriff_chi2

round(kilgarriff_chi2(corpus["А"], corpus["Б"], n_mfw=5), 3)

6.018

Mendenhall curve¶

mendenhall_curve(words)

The shares of the words of every length in characters by ascending length (Mendenhall 1887), a profile of the author comparable between texts whatever their size.

Parameter	Type	Default	Description
`words`	`list[str]`	-	Words of the text

Words that are not a list of strings raise SourceTypeError, and no words SourceError.

mendenhall_distance(words_a, words_b)

The Jensen-Shannon distance with base 2 between the Mendenhall curves of two texts, from 0 (the distributions coincide) to 1.

Parameter	Type	Default	Description
`words_a`	`list[str]`	-	Words of the first text
`words_b`	`list[str]`	-	Words of the second text

Words that are not a list of strings raise SourceTypeError, and a text without words SourceError.

CodeResult

from ruts.corpus import mendenhall_curve, mendenhall_distance

{length: round(share, 3) for length, share in mendenhall_curve(corpus["А"]).items()}
round(mendenhall_distance(corpus["А"], corpus["Б"]), 3)

{1: 0.133, 2: 0.2, 3: 0.133, 4: 0.2, 5: 0.2, 7: 0.133}
0.353

Function word profile¶

function_words_profile(source)

The shares of adpositions, coordinating and subordinating conjunctions, particles, pronouns, determiners and interjections (ruts.constants.FUNCTION_UD_POS: ADP, CCONJ, SCONJ, PART, PRON, DET, INTJ) among the words of the text - by the first ruMorph analysis for a list of words and by the annotation for a Doc with parts of speech; punctuation is not counted. Function words do not depend on the topic, so their profile is a classic authorship feature (the Marusenko school).

Parameter	Type	Default	Description
`source`	`list[str]` / `Doc`	-	Words of the text or Doc object

Raises:

SourceTypeError - a source that is neither a list of strings nor a Doc;
SourceError - a source without words.

CodeResult

from ruts.corpus import function_words_profile

{pos: round(share, 3) for pos, share in function_words_profile(corpus["А"]).items()}

{'ADP': 0.2, 'CCONJ': 0.133, 'SCONJ': 0.0, 'PART': 0.0, 'PRON': 0.0, 'DET': 0.0, 'INTJ': 0.0}