Keywords¶

keyness(target, reference, measure='log_likelihood', min_freq=1, positive=True, top_n=None, n=1)

Keyword extraction (keyness) for a target corpus against a reference one: the words that occur significantly more often in the target corpus than in the reference.

Every word gets two values that Gabrielatos and Marchi and Hardie recommend reading together: the log-likelihood \(G^2\) with its p-value (whether there is a difference) and Log Ratio (how large it is), plus the chosen measure score, which sorts the list. The measures of significance (\(G^2\), chi-square, BIC) and ELL, the effect size of \(G^2\), are negative when the word is more frequent in the reference; %DIFF, Log Ratio and the odds ratio are directional by construction.

The reference may be a list of words, a mapping of frequencies (its size is the sum of the counts) or a FrequencyReference. Words are compared as they are: case, lemmatization and stop words belong to the word extractor, and both corpora have to be extracted the same way. A corpus of several texts is given as a list of their lists of words. With n above 1 the keywords are the n-grams of the words of every text, joined by underscores as the n-grams of WordsExtractor (black_cat), so an n-gram never runs across two texts; the counts and the sizes are those of the n-grams, and the frequencies per million are per million n-grams. Frequencies of n-grams - a sum of the Counter of the n-grams of every text, or a frequency dictionary of n-grams - go with n=1: a mapping and a FrequencyReference are refused with n above 1.

The function wraps keyness of the anyTS core and also takes the frequency dictionary FreqDict as the reference; FrequencyReference is in ests.corpus next to it. Words are extracted with WordsExtractor.

CodeResult

from ests import WordsExtractor
from ests.corpus import keyness

we = WordsExtractor(use_lexemes=True, lowercase=True)
target = we.extract(
    "El gato estaba en la ventana y miraba a los pájaros. Los pájaros se fueron y el gato "
    "se durmió en la ventana. Mañana el gato volverá a estar en la ventana y mirará a los pájaros."
)
reference = we.extract(
    "El perro estaba en el suelo y dormía. Después el perro comió y volvió a dormir. "
    "Mañana el perro saldrá a pasear."
)

keyness(target, reference, top_n=1)
[(k.word, round(k.g2, 2)) for k in keyness(target, reference, positive=False, top_n=2)]

[Keyword(word='gato', freq_target=3, freq_reference=0.0, ipm_target=81081.08108108108, ipm_reference=0.0, g2=2.79971718756897, p_value=0.09428093556593176, log_ratio=1.8349407537295037, score=2.79971718756897)]
[('perro', -5.92), ('comer', -1.97)]

Parameters¶

Parameter	Type	Default	Description
`target`	`list[str]` / `list[list[str]]` / `dict[str, int]`	-	Words of the target corpus - one text or a list of texts - or their frequencies
`reference`	`list[str]` / `list[list[str]]` / `dict[str, float]` / `FrequencyReference`	-	Words of the reference corpus - one text or a list of texts - their frequencies or a reference by frequencies
`measure`	`str`	`'log_likelihood'`	Measure of `anyts.constants.KEYNESS_MEASURES` for `score` and the sorting
`min_freq`	`int`	`1`	Minimum frequency of a keyword in its own corpus
`positive`	`bool`	`True`	Positive keywords (more frequent in the target corpus) or negative ones (more frequent in the reference)
`top_n`	`int`	`None`	Number of keywords; `None` - all of them
`n`	`int`	`1`	Length of the n-grams of the words of every text; `1` - the words

The parameters and the target corpus are checked by check_keyness_params; an n above 1 with frequencies or a FrequencyReference raises ParameterError. A reference that is not a list of strings, a list of such lists, a counter or a FrequencyReference raises SourceTypeError, and a negative, nan or infinite frequency of a counter and a corpus without words, or shorter than n words, raise SourceError.

reference may also be FreqDict, against which target is word forms. A frequency dictionary that is not downloaded raises DatasetNotFoundError. keyness checks its parameters and the target with check_keyness_params before it reads FreqDict, so a wrong argument fails before the dictionary is loaded; the dictionary holds words, so n above 1 with it raises ParameterError before it is read too.

check_keyness_params(measure='log_likelihood', min_freq=1, top_n=None, target=None, n=1, positive=True)

Checks the parameters of keyness before a frequency dictionary is read: it raises ParameterError unless the measure is one of KEYNESS_MEASURES, the minimum frequency is an integer, the number of keywords is None or an integer of at least one, the length of the n-grams n is an integer of at least one and positive is a bool. With target it also raises SourceTypeError unless the target corpus is a list of strings, a list of such lists or a mapping of strings to whole frequencies, and SourceError for a negative, nan or infinite frequency.

Reference by frequencies¶

FrequencyReference(counts, size, missing=0.0, key=None, keep=None)

A reference corpus described by its frequencies alone, such as a frequency dictionary built from a corpus too large to pass as words:

Field	Type	Default	Description
`counts`	`dict[str, float]`	-	Frequencies of the keys in the reference corpus
`size`	`float`	-	Size of the reference corpus in words
`missing`	`float`	`0.0`	Frequency of a key missing from `counts`
`key`	`Callable`	`None`	Key of a word of the target corpus in `counts`, such as its lemma; `None` - the word itself
`keep`	`Callable`	`None`	Whether a word of the target corpus is counted; `None` - every word

Only the words of the target that keep passes are counted, under their key, in the frequencies and in the size alike. A key missing from counts has the frequency missing, such as the least frequency of a dictionary: an upper bound, so its word may be a positive keyword but never a negative one. Negative keywords come only from the keys of counts that keep passes. A frequency is checked as it is read, so a large dictionary is not gone through for positive keywords: one that is not a number raises SourceTypeError, and a negative, nan or infinite one SourceError. A size or a missing that is not a finite number or is negative raises ParameterError, and a key or a keep that cannot be called SourceTypeError.

The frequency dictionary FreqDict of Google Books Ngram passed as reference is turned into such a reference:

Field	`FreqDict`
`counts`	the ipm of `FreqDict.word_ipm`, the proper nouns included, converted to occurrences in the corpus of the dictionary
`size`	`CORPUS_SIZE`, 63 billion words of the books of 1980-2019
`missing`	the least frequency of the dictionary `FreqDict.min_ipm`, 0.1 ipm (about 6300 occurrences)
`key`	`lemma_key`: a keyword may carry the label of another lemma, Roma is counted under `romo`
`keep`	the forms made of the letters of `WORD_PATTERN`; numbers and words with a hyphen or a dot are left out

The target has to be counted the way the dictionary was: word forms with the stop words kept, not lemmas - lemma_key is not idempotent (estado - estar), so WordsExtractor(use_lexemes=True) does not fit. The dictionary describes the register of the books, so the negative keywords of a text are the words of scholarly prose (de, social, país).

Against the dictionary the words of a play of Lope de Vega from the corpus of literature give its characters:

CodeResult

from ests import WordsExtractor
from ests.corpus import keyness
from ests.datasets import FreqDict, SpanishLiterature

text = next(SpanishLiterature().get_texts(author="lope", genre="drama"))
words = WordsExtractor(lowercase=True).extract(text)
for k in keyness(words, FreqDict(), top_n=5):
    print(k.word, k.freq_target, round(k.ipm_reference, 2), round(k.g2, 1), round(k.log_ratio, 2))

laurencia 135 0.32 2528.7 14.96
mengo 105 0.13 2102.5 15.9
comendador 154 5.33 2059.9 11.09
frondoso 113 3.84 1515.6 11.12
barrildo 52 0.1 995.7 15.26

Result¶

Keyword(word, freq_target, freq_reference, ipm_target, ipm_reference, g2, p_value, log_ratio, score)

A list of Keyword named tuples by descending keyness (ties broken by descending frequency and alphabetically, words of an undefined measure last); pd.DataFrame(keywords) gives a table.

Field	Type	Description
`word`	`str`	Word
`freq_target`	`int`	Frequency in the target corpus
`freq_reference`	`float`	Frequency in the reference corpus
`ipm_target`	`float`	Frequency in the target corpus per million words, or n-grams
`ipm_reference`	`float`	Frequency in the reference corpus per million words, or n-grams
`g2`	`float`	Signed \(G^2\)
`p_value`	`float`	p-value of \(G^2\)
`log_ratio`	`float`	Log Ratio
`score`	`float`	Value of the chosen measure

Measures¶

The measures of measure by key, with their names and descriptions in anyts.constants.KEYNESS_MEASURES:

Key	Measures	Range	Higher means
`log_likelihood`	Significance of the difference of the frequencies	Any number	A more significant excess in the target corpus
`chi2`	Significance of the difference, with Yates's correction	Any number	A more significant excess in the target corpus
`diff`	Difference of the frequencies per million, in percent of the reference	Above −100	A larger excess in the target corpus
`log_ratio`	Binary logarithm of the ratio of the frequencies per million	Any number	A larger excess in the target corpus
`bic`	Evidence of the difference: \(G^2\) penalized by the size of the corpora	Any number	Stronger evidence of an excess in the target corpus, for a word more frequent there
`ell`	Effect size of \(G^2\)	−1 to 1 as a rule, `nan` for small expected frequencies	A larger excess in the target corpus
`odds_ratio`	Ratio of the odds of the word in the corpora	0 to `inf`, `nan` if the word fills both corpora	Higher odds in the target corpus

Every measure is a function of the module anyts.corpus.keyness of the frequencies of a word and the sizes of the corpora, \(N = c + d\):

Parameter	Type	Default	Description
`a`	`float`	-	Frequency of the word in the target corpus
`b`	`float`	-	Frequency of the word in the reference corpus
`c`	`float`	-	Size of the target corpus
`d`	`float`	-	Size of the reference corpus

A zero frequency in one of the corpora is replaced with 0.5 for %DIFF, Log Ratio and the odds ratio (Hardie 2014).

The module ests.corpus.keyness re-exports the functions of the measures (from ests.corpus.keyness import calc_log_likelihood).

Log-likelihood¶

log_likelihood · g2 · p_value · Rayson and Garside (2000)

calc_log_likelihood(a, b, c, d)

The log-likelihood of the frequencies of the word against the expected ones \(E_1\) and \(E_2\); a term of zero frequency is zero. The critical values are anyts.constants.G2_CRITICAL_VALUES: 3.84 for p < 0.05, 6.63 for p < 0.01, 10.83 for p < 0.001, 15.13 for p < 0.0001. The sign is negative when the word is more frequent in the reference.

\[ G^2 = 2 \left(a \ln \frac{a}{E_1} + b \ln \frac{b}{E_2}\right), \quad E_1 = \frac{c\,(a+b)}{N}, \quad E_2 = \frac{d\,(a+b)}{N} \]

calc_p_value(g2)

The p-value of a \(G^2\) or a chi-square, whatever its sign, by the chi-square distribution with one degree of freedom; an array of values gives an array.

Chi-square¶

chi2

calc_chi2(a, b, c, d)

The chi-square with Yates's correction over the 2×2 contingency table of the word and the other words of each corpus. When the correction exceeds the difference, the statistic is zero. The sign is negative when the word is more frequent in the reference.

\[ \chi^2 = \frac{N\,\max(\lvert a(d-b) - b(c-a) \rvert - N/2,\ 0)^2}{(a+b)(N-a-b)\,c\,d} \]

%DIFF¶

diff · Gabrielatos and Marchi (2011)

calc_diff(a, b, c, d)

The difference of the frequencies \(NF\) per million words, in percent of the frequency in the reference.

\[ \frac{NF_a - NF_b}{NF_b} \cdot 100 \]

Log Ratio¶

log_ratio · Hardie (2014)

calc_log_ratio(a, b, c, d)

The binary logarithm of the ratio of the frequencies \(NF\) per million words: one means the word is twice as frequent in the target corpus.

\[ \log_2 \frac{NF_a}{NF_b} \]

BIC¶

bic · Wilson (2013)

calc_bic(a, b, c, d)

The Bayesian information criterion of \(G^2\). In absolute value above 2 it is positive evidence of a difference, above 6 strong, above 10 very strong; a value of the sign opposite to \(G^2\), when \(\lvert G^2 \rvert < \ln N\), means no evidence, not the opposite direction.

\[ \operatorname{sign}(G^2) \cdot (\lvert G^2 \rvert - \ln N) \]

ELL¶

ell · Johnston, Berry and Mielke (2006)

calc_ell(a, b, c, d)

The effect size of \(G^2\): the share of the greatest possible departure from the expected frequencies \(E_1\) and \(E_2\). It grows without bound as the least expected frequency nears one and is nan when that frequency is at most one (a logarithm of zero or below). The sign is negative when the word is more frequent in the reference.

\[ \frac{G^2}{N \ln \min(E_1, E_2)} \]

Odds ratio¶

odds_ratio

calc_odds_ratio(a, b, c, d)

The ratio of the odds of the word in the two corpora: one means equal odds. A word that fills the whole target corpus gives inf, the whole reference 0, both nan.

\[ \frac{a / (c - a)}{b / (d - b)} \]