Lyashevskaya and Sharoff frequency dictionary

FreqDict(data_dir=DEFAULT_DATASET_DIR)

The frequency dictionary of modern Russian by O. N. Lyashevskaya and S. A. Sharoff (Freq2011): 52,138 lemmas of the modern subcorpus of the Russian National Corpus (1950-2007, 92 million tokens) with the ipm frequency, the range R, Juilland's D, the number of documents and the part of speech in MyStem tags (s, v, a, adv, s.PROP for proper names).

The dictionary is downloaded from dict.ruslang.ru (HTTP only, 493 KB). No license is stated; the authors ask to cite the original source:

Citation

Ляшевская О. Н., Шаров С. А. Частотный словарь современного русского языка (на материалах Национального корпуса русского языка). М.: Азбуковник, 2009.

For lookup, the entries of one lemma with different parts of speech (а - conjunction, particle, interjection) are merged: ipm is summed, R, D and the number of documents are taken as maxima. The dictionary is read once per process, so creating FreqDict() for every text is cheap. It is used by LexicalStats and by keyness as a reference corpus (of size CORPUS_SIZE).

Raises:

• SourceTypeError - a data_dir that is neither a string nor a Path.

CodeResult

from ruts.datasets import FreqDict

fd = FreqDict()
fd.download()
fd.info["citation"]

'Ляшевская О. Н., Шаров С. А. Частотный словарь современного русского языка (на материалах Национального корпуса русского языка). М.: Азбуковник, 2009.'

Parameters

Parameter	Type	Default	Description
data_dir	str / Path	DEFAULT_DATASET_DIR	Path to the dictionary directory; dicts in DEFAULT_DATA_DIR by default

Attributes

Attribute	Type	Description
entries	dict[str, Entry]	Entries by normalized lemma
min_ipm	float	Minimum frequency in the dictionary (0.4)
word_ipm	dict[str, float]	Frequencies by the entry a word form reaches without a part of speech - the reference of keyness
info	dict[str, str]	Name of the dictionary and reference information about it

Entry(lemma, pos, ipm, range, dispersion, docs)

An entry of the dictionary: a named tuple.

Field	Type	Description
lemma	str	Lemma
pos	tuple[str, ...]	Parts of speech of the lemma in MyStem tags
ipm	float	Frequency per million tokens, summed over the parts of speech
range	int	Number of corpus segments out of 100 in which the lemma occurs
dispersion	int	Juilland's D
docs	int	Number of documents in which the lemma occurs

Methods

download(force=False) downloads the dictionary from the network and extracts the file; if the archive is present but the dictionary file is not, the archive is extracted again. The archive is verified against a SHA-256 checksum, a corrupted or replaced file is deleted.

Parameter	Type	Default	Description
force	bool	False	Download the dictionary even if it is already downloaded

A file that cannot be downloaded or fails the checksum raises DownloadError, an archive that cannot be extracted DataFileError.

check_data() checks that the dictionary file is in place and returns True; a dictionary that is not downloaded raises DatasetNotFoundError with the commands to download it, and so does reading its entries and frequencies.

lookup(lemma) returns the dictionary entry for a lemma in any case, with or without ё, None for a lemma outside the dictionary; ipm(lemma) returns the lemma frequency per million tokens, 0 for a lemma outside the dictionary.

Parameter	Type	Default	Description
lemma	str	-	Lemma

A lemma that is not a string raises SourceTypeError.

CodeResult

fd.lookup("кот")
fd.lookup("Ещё")
fd.ipm("котёнок"), fd.ipm("котоведение")
"кот" in fd, len(fd)

Entry(lemma='кот', pos=('s',), ipm=40.3, range=98, dispersion=90, docs=947)
Entry(lemma='еще', pos=('adv', 'part'), ipm=2409.4, range=100, dispersion=97, docs=22654)
(14.5, 0.0)
(True, 51682)

get_records(pos=None, min_ipm=None, limit=None) returns a generator of dictionary records in file (alphabetical) order, filtered by part of speech and minimum frequency; get_texts with the same parameters returns a generator of lemmas.

Parameter	Type	Default	Description
pos	str	None	Part of speech in MyStem tags
min_ipm	float	None	Minimum frequency
limit	int	None	Number of records

An unknown part of speech and a limit that is not a non-negative integer raise ParameterError.

CodeResult

for record in fd.get_records(pos="s", min_ipm=2700, limit=2):
    print(record)
list(fd.get_texts(pos="s", min_ipm=2700, limit=2))

{'lemma': 'год', 'pos': 's', 'ipm': 3727.5, 'range': 100, 'dispersion': 94, 'docs': 29477}
{'lemma': 'человек', 'pos': 's', 'ipm': 2723.0, 'range': 100, 'dispersion': 97, 'docs': 20423}
['год', 'человек']