Word dispersion¶
dispersion(words, parts=10, word=None, min_freq=1)
The dispersion of a word is how evenly it is spread over the parts of a text or of a corpus. Frequency does not tell a word occurring once in every chapter from one gathered in a single chapter; the measures of dispersion (Gries 2008, 2020) complement it.
parts is the number of parts of about equal size or the sizes of the parts in order (sentences, paragraphs, chapters, documents of a corpus), adding up to the number of words; split_paragraphs of anyts.extractors gives the paragraphs of a text between blank lines, whose words make the sizes (for a text with a paragraph on every line, its lines are the paragraphs, as in CohesionStats); the paragraphs without words, such as a scene break of asterisks, are dropped first, since a part needs a word. Six measures are computed from the frequencies of every word by part; Gries recommends DP as the main one.
Words are compared as they are: case and lemmatization belong to the word extractor.
ests.corpus re-exports the function of the anyTS core. Words are extracted with WordsExtractor.
from ests import SentsExtractor, WordsExtractor
from ests.corpus import dispersion
text = (
"El gato estaba en la ventana y miraba a los pájaros. Los pájaros se fueron y el gato "
"se durmió en la ventana. Mañana el gato volverá a estar en la ventana y mirará a los pájaros."
)
we = WordsExtractor(use_lexemes=True, lowercase=True)
words = we.extract(text)
sizes = [len(we.extract(sent)) for sent in SentsExtractor().extract(text)]
sizes
dispersion(words, parts=sizes, word="gato")
# Three equal parts: a and pájaro are gathered at the edges of the text
[(d.word, round(d.dp, 2)) for d in dispersion(words, parts=3, min_freq=3)]
[11, 12, 14]
[Dispersion(word='gato', freq=3, dp=0.04504504504504503, dp_norm=0.06410256410256408, juilland_d=0.9307654905484509, carroll_d2=0.9955884389001025, rosengren_s=0.9974825285744621, kl_divergence=0.007241184435774255)]
[('el', 0.1), ('gato', 0.02), ('en', 0.02), ('ventana', 0.02), ('y', 0.02), ('a', 0.34), ('pájaro', 0.32)]
The paragraphs of a text come from split_paragraphs, imported from ests.extractors; a scene break of asterisks has no words and is dropped:
from ests.extractors import split_paragraphs
text = (
"Capítulo I\n\nEl gato estaba en la ventana y miraba a los pájaros.\n\n* * *\n\n"
"Los pájaros se fueron y el gato se durmió en la ventana.\n\n"
"Mañana el gato volverá a estar en la ventana."
)
sizes = [n for n in (len(we.extract(part)) for part in split_paragraphs(text)) if n]
sizes
[(d.word, round(d.dp, 2)) for d in dispersion(we.extract(text), parts=sizes, min_freq=3)]
[2, 11, 12, 9]
[('el', 0.07), ('gato', 0.08), ('en', 0.08), ('ventana', 0.08)]
Parameters¶
| Parameter | Type | Default | Description |
|---|---|---|---|
words |
list[str] |
- | Words of the text in order |
parts |
int / list[int] |
10 |
Number of parts (from 2 to the number of words) or sizes of the parts |
word |
str |
None |
Word whose dispersion is needed; None - every word |
min_freq |
int |
1 |
Minimum frequency of a word |
Words that are not a list of strings raise SourceTypeError, and no words SourceError. A parts that is neither an integer from 2 to the number of words nor two or more positive integers adding up to it, a word that is not a string and a min_freq that is not an integer raise ParameterError.
Result¶
Dispersion(word, freq, dp, dp_norm, juilland_d, carroll_d2, rosengren_s, kl_divergence)
A list of Dispersion named tuples by descending frequency: the word, its frequency and the measures; pd.DataFrame(result) gives a table.
Measures¶
The measures, the fields of Dispersion, with their names in anyts.constants.DISPERSION_STATS_DESC:
| Field | Measures | Range | Higher means |
|---|---|---|---|
dp |
Deviation of the shares of the occurrences from the shares of the parts | 0 to \(1 - \min s_i\) | More uneven |
dp_norm |
DP over its greatest possible value | 0-1 | More uneven |
juilland_d |
Variation of the relative frequencies by part | 0-1 | More even |
carroll_d2 |
Entropy of the relative frequencies by part | 0-1 | More even |
rosengren_s |
Spread of the occurrences against the sizes of the parts | \(\min s_i\) to 1 | More even |
kl_divergence |
Divergence of the shares of the occurrences from the shares of the parts, in bits | 0 to \(-\log_2 \min s_i\) | More uneven |
Every measure is a function of the module anyts.corpus.dispersion of the frequencies of the word by part and the sizes of the parts. For \(n\) parts of sizes \(n_i\) and shares \(s_i\) of the text, frequencies of the word by part \(v_i\) and a total frequency \(f = \sum v_i\), \(p_i = v_i / n_i\) is the relative frequency in a part:
| Parameter | Type | Default | Description |
|---|---|---|---|
frequencies |
list[int] |
- | Frequencies of the word by part |
sizes |
list[int] |
- | Sizes of the parts |
For a word of zero frequency every measure is nan.
The module ests.corpus.dispersion re-exports the functions of the measures (from ests.corpus.dispersion import calc_dp).
Deviation of proportions DP¶
calc_dp(frequencies, sizes)
How far the shares of the occurrences of the word by part are from the shares of the parts: 0 means the word is spread in proportion to the sizes of the parts, \(1 - s_i\) that it is all in part \(i\).
Normalized DP¶
calc_dp_norm(frequencies, sizes)
DP divided by its greatest possible value, so the maximum is one whatever the split.
Juilland's D¶
calc_juilland_d(frequencies, sizes)
One less the coefficient of variation \(V\) of the relative frequencies \(p_i\) over its greatest value \(\sqrt{n - 1}\): 1 means even, 0 all in one part. The standard deviation \(\sigma\) is that of the population, as in Gries (2020); Gries (2008) takes the sample one.
Carroll's D2¶
calc_carroll_d2(frequencies, sizes)
The entropy of the distribution of the relative frequencies \(p_i\) over the parts, divided by its greatest value: 1 means even, 0 all in one part.
Rosengren's S¶
calc_rosengren_s(frequencies, sizes)
1 means the word is spread in proportion to the sizes of the parts; all in part \(i\), it gives \(s_i\), so \(1/n\) in one of \(n\) equal parts.
Kullback-Leibler divergence¶
calc_kl_divergence(frequencies, sizes)
The divergence of the shares of the occurrences of the word by part from the shares of the parts: 0 means in proportion; it grows when the word is gathered in small parts.