Lyashevskaya and Sharoff frequency dictionary¶

ruts.datasets.FreqDict

Description¶

A module for working with the frequency dictionary of modern Russian by O. N. Lyashevskaya and S. A. Sharoff (Freq2011): 52,138 lemmas of the modern subcorpus of the Russian National Corpus (1950-2007, 92 million tokens) with the ipm frequency, the range R (the number of corpus segments out of 100 in which the lemma occurs), Juilland's D, the number of documents and the part of speech in MyStem tags (s, v, a, adv, s.PROP for proper names).

The dictionary is downloaded from dict.ruslang.ru (HTTP only, 493 KB). No license is stated; the authors ask to cite the original source:

Citation

Ляшевская О. Н., Шаров С. А. Частотный словарь современного русского языка (на материалах Национального корпуса русского языка). М.: Азбуковник, 2009.

For lookup, a lemma with several parts of speech (а - conjunction, particle, interjection) is merged: ipm is summed, R, D and the number of documents are taken as maxima; the lemma is lowercased and ё is replaced with е, as in the dictionary. The parsed dictionary is cached by file path and read once per process, so creating FreqDict() for every text is cheap. The dictionary is used by LexicalStats and as a reference corpus in keyness (the dictionary corpus size is CORPUS_SIZE, 92 million tokens).

Parameters¶

Parameter	Type	Default	Description
`data_dir`	str	`DEFAULT_DATA_DIR.joinpath("dicts")`	Path to the dictionary directory

Attributes¶

Attribute	Type	Description
`entries`	dict[str, Entry]	Entries by normalized lemma
`min_ipm`	float	Minimum frequency in the dictionary (0.4)

An Entry is a named tuple with the fields lemma, pos (tuple of parts of speech), ipm, range, dispersion, docs.

Methods¶

download¶

Downloads the dictionary from the network and extracts the file. The downloaded archive is verified against a SHA-256 checksum (the file on the site has not changed since 2013); a corrupted or replaced file is deleted with a DownloadError, so that a repeated download is not skipped. If the archive is present but the dictionary file is not, the archive is extracted again.

Parameters:

Parameter	Type	Default	Description
`force`	bool	`-`	Download the dictionary even if it is already downloaded

Example

from ruts.datasets import FreqDict

fd = FreqDict()
fd.download()
fd.info["citation"]
# 'Ляшевская О. Н., Шаров С. А. Частотный словарь современного русского языка (на материалах Национального корпуса русского языка). М.: Азбуковник, 2009.'

lookup¶

Returns the dictionary entry for a lemma in any case, with or without ё, None for a lemma outside the dictionary.

Parameters:

Parameter	Type	Default	Description
`lemma`	str	`-`	Lemma

Example

fd.lookup("кот")
# Entry(lemma='кот', pos=('s',), ipm=40.3, range=98, dispersion=90, docs=947)
fd.lookup("Ещё")
# Entry(lemma='еще', pos=('adv', 'part'), ipm=2409.4, range=100, dispersion=97, docs=22654)

ipm¶

Returns the lemma frequency per million tokens, 0 for a lemma outside the dictionary.

Example

fd.ipm("котёнок"), fd.ipm("котоведение")
# (14.5, 0.0)
"кот" in fd, len(fd)
# (True, 51682)

get_records¶

Returns a generator of dictionary records in file (alphabetical) order, filtered by part of speech and minimum frequency.

Parameters:

Parameter	Type	Default	Description
`pos`	str	`None`	Part of speech in MyStem tags
`min_ipm`	float	`None`	Minimum frequency
`limit`	int	`None`	Number of records

Example

for record in fd.get_records(pos="s", min_ipm=2700, limit=2):
    print(record)
# {'lemma': 'год', 'pos': 's', 'ipm': 3727.5, 'range': 100, 'dispersion': 94, 'docs': 29477}
# {'lemma': 'человек', 'pos': 's', 'ipm': 2723.0, 'range': 100, 'dispersion': 97, 'docs': 20423}

get_texts¶

Returns a generator of dictionary lemmas with the same parameters as get_records.