Keywords¶

keyness(target, reference, measure='log_likelihood', min_freq=1, positive=True, top_n=None, n=1, lemmatize=True)

Keyword extraction (keyness) for a target corpus against a reference one: the words that occur significantly more often in the target corpus than in the reference.

Every word gets two values that Gabrielatos and Marchi and Hardie recommend reading together: the log-likelihood \(G^2\) with its p-value (whether there is a difference) and Log Ratio (how large it is), plus the chosen measure score, which sorts the list. The measures of significance (\(G^2\), chi-square, BIC) and ELL, the effect size of \(G^2\), are negative when the word is more frequent in the reference; %DIFF, Log Ratio and the odds ratio are directional by construction.

The reference may be a list of words, a mapping of frequencies (its size is the sum of the counts) or a FrequencyReference. Words are compared as they are: case, lemmatization and stop words belong to the word extractor, and both corpora have to be extracted the same way. A corpus of several texts is given as a list of their lists of words. With n above 1 the keywords are the n-grams of the words of every text, joined by underscores as the n-grams of WordsExtractor (black_cat), so an n-gram never runs across two texts; the counts and the sizes are those of the n-grams, and the frequencies per million are per million n-grams. Frequencies of n-grams - a sum of the Counter of the n-grams of every text, or a frequency dictionary of n-grams - go with n=1: a mapping and a FrequencyReference are refused with n above 1.

The function wraps keyness of the anyTS core and also takes the Lyashevskaya and Sharoff frequency dictionary FreqDict as the reference; FrequencyReference is in ruts.corpus next to it. Words are extracted with WordsExtractor.

CodeResult

from ruts import WordsExtractor
from ruts.corpus import keyness

we = WordsExtractor(use_lexemes=True, lowercase=True)
target = we.extract(
    "Кот сидел на окне и смотрел на птиц. Птицы улетели, и кот уснул на окне. "
    "Завтра кот снова будет сидеть на окне и смотреть на птиц."
)
reference = we.extract(
    "Собака лежала на полу и дремала. Потом собака ела и снова дремала. Завтра собака будет гулять."
)

keyness(target, reference, top_n=1)
[(k.word, round(k.g2, 2)) for k in keyness(target, reference, positive=False, top_n=2)]

[Keyword(word='кот', freq_target=3, freq_reference=0.0, ipm_target=115384.61538461539, ipm_reference=0.0, g2=2.8774384815713177, p_value=0.08982881315854577, log_ratio=1.8845227825800641, score=2.8774384815713177)]
[('собака', -5.79), ('дремать', -3.86)]

Parameters¶

Parameter	Type	Default	Description
`target`	`list[str]` / `list[list[str]]` / `dict[str, int]`	-	Words of the target corpus - one text or a list of texts - or their frequencies
`reference`	`list[str]` / `list[list[str]]` / `dict[str, float]` / `FrequencyReference`	-	Words of the reference corpus - one text or a list of texts - their frequencies or a reference by frequencies
`measure`	`str`	`'log_likelihood'`	Measure of `anyts.constants.KEYNESS_MEASURES` for `score` and the sorting
`min_freq`	`int`	`1`	Minimum frequency of a keyword in its own corpus
`positive`	`bool`	`True`	Positive keywords (more frequent in the target corpus) or negative ones (more frequent in the reference)
`top_n`	`int`	`None`	Number of keywords; `None` - all of them
`n`	`int`	`1`	Length of the n-grams of the words of every text; `1` - the words

The parameters and the target corpus are checked by check_keyness_params; an n above 1 with frequencies or a FrequencyReference raises ParameterError. A reference that is not a list of strings, a list of such lists, a counter or a FrequencyReference raises SourceTypeError, and a negative, nan or infinite frequency of a counter and a corpus without words, or shorter than n words, raise SourceError.

reference may also be FreqDict, and one more parameter tells what the target corpus is against it:

Parameter	Type	Default	Description
`lemmatize`	`bool`	`True`	The target corpus is word forms, `False` - lemmas (`FreqDict` only)

A frequency dictionary that is not downloaded raises DatasetNotFoundError, and an n above 1 with it ParameterError.

check_keyness_params(measure='log_likelihood', min_freq=1, top_n=None, target=None, n=1, positive=True)

Checks the parameters of keyness before a frequency dictionary is read: it raises ParameterError unless the measure is one of KEYNESS_MEASURES, the minimum frequency is an integer, the number of keywords is None or an integer of at least one, the length of the n-grams n is an integer of at least one and positive is a bool. With target it also raises SourceTypeError unless the target corpus is a list of strings, a list of such lists or a mapping of strings to whole frequencies, and SourceError for a negative, nan or infinite frequency.

Reference by frequencies¶

FrequencyReference(counts, size, missing=0.0, key=None, keep=None)

A reference corpus described by its frequencies alone, such as a frequency dictionary built from a corpus too large to pass as words:

Field	Type	Default	Description
`counts`	`dict[str, float]`	-	Frequencies of the keys in the reference corpus
`size`	`float`	-	Size of the reference corpus in words
`missing`	`float`	`0.0`	Frequency of a key missing from `counts`
`key`	`Callable`	`None`	Key of a word of the target corpus in `counts`, such as its lemma; `None` - the word itself
`keep`	`Callable`	`None`	Whether a word of the target corpus is counted; `None` - every word

Only the words of the target that keep passes are counted, under their key, in the frequencies and in the size alike. A key missing from counts has the frequency missing, such as the least frequency of a dictionary: an upper bound, so its word may be a positive keyword but never a negative one. Negative keywords come only from the keys of counts that keep passes. A frequency is checked as it is read, so a large dictionary is not gone through for positive keywords: one that is not a number raises SourceTypeError, and a negative, nan or infinite one SourceError. A size or a missing that is not a finite number or is negative raises ParameterError, and a key or a keep that cannot be called SourceTypeError.

The frequency dictionary FreqDict passed as reference is turned into such a reference:

Field	`FreqDict`
`counts`	the frequencies of `FreqDict.word_ipm` converted to occurrences in the corpus of the dictionary
`size`	`CORPUS_SIZE` of `ruts.datasets.freq2011`, 92 million tokens of the modern subcorpus of the Russian National Corpus
`missing`	the least frequency of the dictionary `FreqDict.min_ipm`, 0.4 ipm (about 37 occurrences)
`key`	the lemma of the dictionary: the ruMorph lemma of a word form (with `lemmatize=False` the word itself) brought to the conventions of the dictionary, as in `LexicalStats`
`keep`	words of Cyrillic letters; numbers and words with Latin letters are left out

A dictionary entry that no word form is reduced to is never a negative keyword: the dictionary keeps «во» and «со» as entries of their own, while ruMorph reduces these word forms to «в» and «с».

With lemmatize=False a lemma loses the conventions of the dictionary - the short form, the plural, the spelling variant («должный» instead of «должен», «деньга» instead of «деньги») - and such entries come out as false keywords, so compare word forms with the dictionary.

CodeResult

from ruts import WordsExtractor
from ruts.corpus import keyness
from ruts.datasets import FreqDict

# The dictionary is downloaded once with FreqDict().download()
words = WordsExtractor().extract(
    "Кот сидел на окне и смотрел на птиц. Птицы улетели, и кот уснул на окне. "
    "Завтра кот снова будет сидеть на окне и смотреть на птиц."
)
[(k.word, round(k.log_ratio, 1)) for k in keyness(words, FreqDict(), min_freq=3, top_n=2)]

[('кот', 11.5), ('птица', 10.3)]

Result¶

Keyword(word, freq_target, freq_reference, ipm_target, ipm_reference, g2, p_value, log_ratio, score)

A list of Keyword named tuples by descending keyness (ties broken by descending frequency and alphabetically, words of an undefined measure last); pd.DataFrame(keywords) gives a table.

Field	Type	Description
`word`	`str`	Word
`freq_target`	`int`	Frequency in the target corpus
`freq_reference`	`float`	Frequency in the reference corpus
`ipm_target`	`float`	Frequency in the target corpus per million words, or n-grams
`ipm_reference`	`float`	Frequency in the reference corpus per million words, or n-grams
`g2`	`float`	Signed \(G^2\)
`p_value`	`float`	p-value of \(G^2\)
`log_ratio`	`float`	Log Ratio
`score`	`float`	Value of the chosen measure

Measures¶

The measures of measure by key, with their names and descriptions in anyts.constants.KEYNESS_MEASURES:

Key	Measures	Range	Higher means
`log_likelihood`	Significance of the difference of the frequencies	Any number	A more significant excess in the target corpus
`chi2`	Significance of the difference, with Yates's correction	Any number	A more significant excess in the target corpus
`diff`	Difference of the frequencies per million, in percent of the reference	Above −100	A larger excess in the target corpus
`log_ratio`	Binary logarithm of the ratio of the frequencies per million	Any number	A larger excess in the target corpus
`bic`	Evidence of the difference: \(G^2\) penalized by the size of the corpora	Any number	Stronger evidence of an excess in the target corpus, for a word more frequent there
`ell`	Effect size of \(G^2\)	−1 to 1 as a rule, `nan` for small expected frequencies	A larger excess in the target corpus
`odds_ratio`	Ratio of the odds of the word in the corpora	0 to `inf`, `nan` if the word fills both corpora	Higher odds in the target corpus

Every measure is a function of the module anyts.corpus.keyness of the frequencies of a word and the sizes of the corpora, \(N = c + d\):

Parameter	Type	Default	Description
`a`	`float`	-	Frequency of the word in the target corpus
`b`	`float`	-	Frequency of the word in the reference corpus
`c`	`float`	-	Size of the target corpus
`d`	`float`	-	Size of the reference corpus

A zero frequency in one of the corpora is replaced with 0.5 for %DIFF, Log Ratio and the odds ratio (Hardie 2014).

The module ruts.corpus.keyness re-exports the functions of the measures (from ruts.corpus.keyness import calc_log_likelihood).

Log-likelihood¶

log_likelihood · g2 · p_value · Rayson and Garside (2000)

calc_log_likelihood(a, b, c, d)

The log-likelihood of the frequencies of the word against the expected ones \(E_1\) and \(E_2\); a term of zero frequency is zero. The critical values are anyts.constants.G2_CRITICAL_VALUES: 3.84 for p < 0.05, 6.63 for p < 0.01, 10.83 for p < 0.001, 15.13 for p < 0.0001. The sign is negative when the word is more frequent in the reference.

\[ G^2 = 2 \left(a \ln \frac{a}{E_1} + b \ln \frac{b}{E_2}\right), \quad E_1 = \frac{c\,(a+b)}{N}, \quad E_2 = \frac{d\,(a+b)}{N} \]

calc_p_value(g2)

The p-value of a \(G^2\) or a chi-square, whatever its sign, by the chi-square distribution with one degree of freedom; an array of values gives an array.

Chi-square¶

chi2

calc_chi2(a, b, c, d)

The chi-square with Yates's correction over the 2×2 contingency table of the word and the other words of each corpus. When the correction exceeds the difference, the statistic is zero. The sign is negative when the word is more frequent in the reference.

\[ \chi^2 = \frac{N\,\max(\lvert a(d-b) - b(c-a) \rvert - N/2,\ 0)^2}{(a+b)(N-a-b)\,c\,d} \]

%DIFF¶

diff · Gabrielatos and Marchi (2011)

calc_diff(a, b, c, d)

The difference of the frequencies \(NF\) per million words, in percent of the frequency in the reference.

\[ \frac{NF_a - NF_b}{NF_b} \cdot 100 \]

Log Ratio¶

log_ratio · Hardie (2014)

calc_log_ratio(a, b, c, d)

The binary logarithm of the ratio of the frequencies \(NF\) per million words: one means the word is twice as frequent in the target corpus.

\[ \log_2 \frac{NF_a}{NF_b} \]

BIC¶

bic · Wilson (2013)

calc_bic(a, b, c, d)

The Bayesian information criterion of \(G^2\). In absolute value above 2 it is positive evidence of a difference, above 6 strong, above 10 very strong; a value of the sign opposite to \(G^2\), when \(\lvert G^2 \rvert < \ln N\), means no evidence, not the opposite direction.

\[ \operatorname{sign}(G^2) \cdot (\lvert G^2 \rvert - \ln N) \]

ELL¶

ell · Johnston, Berry and Mielke (2006)

calc_ell(a, b, c, d)

The effect size of \(G^2\): the share of the greatest possible departure from the expected frequencies \(E_1\) and \(E_2\). It grows without bound as the least expected frequency nears one and is nan when that frequency is at most one (a logarithm of zero or below). The sign is negative when the word is more frequent in the reference.

\[ \frac{G^2}{N \ln \min(E_1, E_2)} \]

Odds ratio¶

odds_ratio

calc_odds_ratio(a, b, c, d)

The ratio of the odds of the word in the two corpora: one means equal odds. A word that fills the whole target corpus gives inf, the whole reference 0, both nan.

\[ \frac{a / (c - a)}{b / (d - b)} \]