Collocations¶

collocations(words, window=5, measure='logdice', min_freq=2, node=None, top_n=None, side=None)

Collocation extraction - pairs of words that occur together more often than independence would give: fixed expressions, terminology, the combinatorics of a word. The measures of association are those of Sketch Engine and of nltk.metrics.association.

Pairs are ordered, as in NLTK: the right word occurs at most window words after the left one, each pair of positions counted once; window=1 gives bigrams. The measures take the pair frequency divided by the window (Church and Hanks 1990, as in NLTK), which keeps the expected frequency independent of the window and Dice and the minimum sensitivity at most one; freq_pair stays undivided. So with window > 1 an always adjacent pair gets a logDice of \(14 - \log_2 window\) (13 at a window of 2), not 14 as in Sketch Engine with its raw co-occurrence; 14 needs a pair at every distance within the window. node keeps the pairs with the given word on either side, and side on one of them: "left" the collocates before the node, "right" those after it.

Words are compared as they are: case, lemmatization and stop words belong to the word extractor; lemmas suit fixed expressions, word forms suit grammatical constructions.

The function wraps collocations of the anyTS core and strips stress marks from the words and from node. Words are extracted with WordsExtractor; for Russian fixed expressions such as «точка зрения» or «рабочий класс» take its lemmas (use_lexemes=True).

CodeResult

from ruts import WordsExtractor
from ruts.corpus import collocations

words = WordsExtractor(use_lexemes=True, lowercase=True).extract(
    "Кот сидел на окне и смотрел на птиц. Птицы улетели, и кот уснул на окне. "
    "Завтра кот снова будет сидеть на окне и смотреть на птиц."
)

collocations(words, window=2, top_n=1)
[
    (c.left, c.right, round(c.score, 2))
    for c in collocations(words, window=1, node="кот", min_freq=1, measure="mi")[:2]
]

[Collocation(left='птица', right='улететь', freq_left=3, freq_right=1, freq_pair=2, score=13.0)]
[('завтра', 'кот', 3.12), ('кот', 'снова', 3.12)]

Parameters¶

Parameter	Type	Default	Description
`words`	`list[str]`	-	Words of the text in order
`window`	`int`	`5`	Greatest distance between the words of a pair
`measure`	`str`	`'logdice'`	Measure of `anyts.constants.COLLOCATION_MEASURES`
`min_freq`	`int`	`2`	Minimum frequency of a pair
`node`	`str`	`None`	Word whose combinatorics is needed; `None` - every pair
`top_n`	`int`	`None`	Number of collocations; `None` - all of them
`side`	`str`	`None`	With a node, the side of its collocates: `"left"` or `"right"`; `None` - both

Words that are not a list of strings raise SourceTypeError, and no words SourceError. An unknown measure, a window or a top_n that is not an integer of at least one, a min_freq that is not an integer, a node that is not a string, and a side other than "left" and "right" or without a node raise ParameterError.

Result¶

Collocation(left, right, freq_left, freq_right, freq_pair, score)

A list of Collocation named tuples by descending measure and frequency of the pair (ties broken alphabetically); pd.DataFrame(found) gives a table.

Field	Type	Description
`left`	`str`	Left word
`right`	`str`	Right word, occurring within the window after the left one
`freq_left`	`int`	Frequency of the left word
`freq_right`	`int`	Frequency of the right word
`freq_pair`	`int`	Frequency of the co-occurrence
`score`	`float`	Value of the chosen measure

Measures¶

The measures of measure by key, with their names and descriptions in anyts.constants.COLLOCATION_MEASURES:

Key	Measures	Range	Higher means
`mi`	Frequency of the pair against the expected one, in bits	Any number	A stronger association
`mi3`	MI with the frequency of the pair cubed	Any number	A stronger association
`t_score`	Difference of the observed and the expected frequency of the pair	Any number	A surer association
`dice`	Share of the pair in the occurrences of the two words	0-1	A stronger association
`logdice`	Dice coefficient on a logarithmic scale	14 and below	A stronger association
`log_likelihood`	Significance of the association	0 and above, `nan` when a word fills the text or a cell is negative	A more significant departure from independence, for attracted and avoided pairs alike
`npmi`	MI normalized by the information of the pair	−1 to 1	A stronger association
`min_sensitivity`	Smaller share of the pair in the occurrences of either word	0-1	A stronger association

Every measure is a function of the module anyts.corpus.collocations of the frequencies of the words and of the pair and the number of words:

Parameter	Type	Default	Description
`freq_a`	`int`	-	Frequency \(f_a\) of the first word
`freq_b`	`int`	-	Frequency \(f_b\) of the second word
`freq_ab`	`float`	-	Frequency \(f_{ab}\) of the pair
`n`	`int`	-	Number of words \(N\) of the text

A pair of zero frequency gives nan for MI, MI³, the t-score, logDice and NPMI.

The module ruts.corpus.collocations re-exports the functions of the measures (from ruts.corpus.collocations import calc_logdice).

Mutual information¶

mi · Church and Hanks (1990)

calc_mi(freq_a, freq_b, freq_ab, n)

The pointwise mutual information of the two words; it overrates rare pairs.

\[ \log_2 \frac{f_{ab} N}{f_a f_b} \]

MI³¶

mi3 · Oakes (1998)

calc_mi3(freq_a, freq_b, freq_ab, n)

The mutual information with the frequency of the pair cubed, which favours frequent pairs.

\[ \log_2 \frac{f_{ab}^3 N}{f_a f_b} \]

t-score¶

t_score · Church et al. (1991)

calc_t_score(freq_a, freq_b, freq_ab, n)

The difference of the observed and the expected frequency of the pair over the root of the observed one; it favours frequent pairs.

\[ \frac{f_{ab} - f_a f_b / N}{\sqrt{f_{ab}}} \]

Dice coefficient¶

dice

calc_dice(freq_a, freq_b, freq_ab, n)

The share of the pair in the occurrences of the two words; it does not depend on the size of the text.

\[ \frac{2 f_{ab}}{f_a + f_b} \]

logDice¶

logdice · Rychlý (2008)

calc_logdice(freq_a, freq_b, freq_ab, n)

The Dice coefficient on a binary logarithmic scale, the default measure, as in Sketch Engine. It does not depend on the size of the text; below zero the link is weak.

\[ 14 + \log_2 \frac{2 f_{ab}}{f_a + f_b} \]

Log-likelihood¶

log_likelihood · Dunning (1993)

calc_log_likelihood(freq_a, freq_b, freq_ab, n)

\(G^2\) over the 2×2 contingency table of the two words: the observed frequencies \(O\) - \(f_{ab}\), \(f_a - f_{ab}\), \(f_b - f_{ab}\) and \(N - f_a - f_b + f_{ab}\) - against the expected ones \(E\) under independence. A word that fills the whole text, or a negative cell, as of a word paired with itself in a window above one, gives nan.

\[ G^2 = 2 \sum O \ln \frac{O}{E} \]

NPMI¶

npmi · Bouma (2009)

calc_npmi(freq_a, freq_b, freq_ab, n)

The mutual information normalized by the information of the pair: one means the words occur only together. A pair as frequent as the text is long gives nan.

\[ \frac{MI}{-\log_2 (f_{ab} / N)} \]

Minimum sensitivity¶

min_sensitivity · Pedersen (1998)

calc_min_sensitivity(freq_a, freq_b, freq_ab, n)

The smaller of the shares of the pair in the occurrences of each word.

\[ \min\left(\frac{f_{ab}}{f_a}, \frac{f_{ab}}{f_b}\right) \]