Word dispersion¶
dispersion(words, parts=10, word=None, min_freq=1)
The dispersion of a word is how evenly it is spread over the parts of a text or of a corpus. Frequency does not tell a word occurring once in every chapter from one gathered in a single chapter; the measures of dispersion (Gries 2008, 2020) complement it.
parts is the number of parts of about equal size or the sizes of the parts in order (sentences, paragraphs, chapters, documents of a corpus), adding up to the number of words; split_paragraphs of anyts.extractors gives the paragraphs of a text between blank lines, whose words make the sizes (for a text with a paragraph on every line, its lines are the paragraphs, as in CohesionStats); the paragraphs without words, such as a scene break of asterisks, are dropped first, since a part needs a word. Six measures are computed from the frequencies of every word by part; Gries recommends DP as the main one.
Words are compared as they are: case and lemmatization belong to the word extractor.
The function wraps dispersion of the anyTS core and strips stress marks from the words and from word. Words are extracted with WordsExtractor.
from ruts import SentsExtractor, WordsExtractor
from ruts.corpus import dispersion
text = (
"Кот сидел на окне и смотрел на птиц. Птицы улетели, и кот уснул на окне. "
"Завтра кот снова будет сидеть на окне и смотреть на птиц."
)
we = WordsExtractor(use_lexemes=True, lowercase=True)
words = we.extract(text)
sizes = [len(we.extract(sent)) for sent in SentsExtractor().extract(text)]
sizes
dispersion(words, parts=sizes, word="кот")
# Three equal parts: «птица» only in the first and the last
[(d.word, round(d.dp, 2)) for d in dispersion(words, parts=3, min_freq=3)]
[8, 7, 11]
[Dispersion(word='кот', freq=3, dp=0.08974358974358976, dp_norm=0.12280701754385966, juilland_d=0.8725780285943101, carroll_d2=0.984770130157433, rosengren_s=0.9907464277073554, kl_divergence=0.0265483705216355)]
[('на', 0.15), ('кот', 0.32), ('окно', 0.03), ('и', 0.03), ('птица', 0.35)]
Parameters¶
| Parameter | Type | Default | Description |
|---|---|---|---|
words |
list[str] |
- | Words of the text in order |
parts |
int / list[int] |
10 |
Number of parts (from 2 to the number of words) or sizes of the parts |
word |
str |
None |
Word whose dispersion is needed; None - every word |
min_freq |
int |
1 |
Minimum frequency of a word |
Words that are not a list of strings raise SourceTypeError, and no words SourceError. A parts that is neither an integer from 2 to the number of words nor two or more positive integers adding up to it, a word that is not a string and a min_freq that is not an integer raise ParameterError.
Result¶
Dispersion(word, freq, dp, dp_norm, juilland_d, carroll_d2, rosengren_s, kl_divergence)
A list of Dispersion named tuples by descending frequency: the word, its frequency and the measures; pd.DataFrame(result) gives a table.
Measures¶
The measures, the fields of Dispersion, with their names in anyts.constants.DISPERSION_STATS_DESC:
| Field | Measures | Range | Higher means |
|---|---|---|---|
dp |
Deviation of the shares of the occurrences from the shares of the parts | 0 to \(1 - \min s_i\) | More uneven |
dp_norm |
DP over its greatest possible value | 0-1 | More uneven |
juilland_d |
Variation of the relative frequencies by part | 0-1 | More even |
carroll_d2 |
Entropy of the relative frequencies by part | 0-1 | More even |
rosengren_s |
Spread of the occurrences against the sizes of the parts | \(\min s_i\) to 1 | More even |
kl_divergence |
Divergence of the shares of the occurrences from the shares of the parts, in bits | 0 to \(-\log_2 \min s_i\) | More uneven |
Every measure is a function of the module anyts.corpus.dispersion of the frequencies of the word by part and the sizes of the parts. For \(n\) parts of sizes \(n_i\) and shares \(s_i\) of the text, frequencies of the word by part \(v_i\) and a total frequency \(f = \sum v_i\), \(p_i = v_i / n_i\) is the relative frequency in a part:
| Parameter | Type | Default | Description |
|---|---|---|---|
frequencies |
list[int] |
- | Frequencies of the word by part |
sizes |
list[int] |
- | Sizes of the parts |
For a word of zero frequency every measure is nan.
The module ruts.corpus.dispersion re-exports the functions of the measures (from ruts.corpus.dispersion import calc_dp).
Deviation of proportions DP¶
calc_dp(frequencies, sizes)
How far the shares of the occurrences of the word by part are from the shares of the parts: 0 means the word is spread in proportion to the sizes of the parts, \(1 - s_i\) that it is all in part \(i\).
Normalized DP¶
calc_dp_norm(frequencies, sizes)
DP divided by its greatest possible value, so the maximum is one whatever the split.
Juilland's D¶
calc_juilland_d(frequencies, sizes)
One less the coefficient of variation \(V\) of the relative frequencies \(p_i\) over its greatest value \(\sqrt{n - 1}\): 1 means even, 0 all in one part. The standard deviation \(\sigma\) is that of the population, as in Gries (2020); Gries (2008) takes the sample one.
Carroll's D2¶
calc_carroll_d2(frequencies, sizes)
The entropy of the distribution of the relative frequencies \(p_i\) over the parts, divided by its greatest value: 1 means even, 0 all in one part.
Rosengren's S¶
calc_rosengren_s(frequencies, sizes)
1 means the word is spread in proportion to the sizes of the parts; all in part \(i\), it gives \(s_i\), so \(1/n\) in one of \(n\) equal parts.
Kullback-Leibler divergence¶
calc_kl_divergence(frequencies, sizes)
The divergence of the shares of the occurrences of the word by part from the shares of the parts: 0 means in proportion; it grows when the word is gathered in small parts.