Skip to content

Analyzer

Analyzer(path=None)

Morphological analyzer: for a word form it finds all analyses in the OpenCorpora dictionary - the initial form, the part of speech and the Universal Dependencies features - and orders them by probability. The dictionary is loaded when the first analyzer is created: analyzers with the same dictionary file in a process share it, so a new analyzer is created instantly. The last loaded dictionary stays in the process even without analyzers, so process pool tasks and an analyzer created for a single call do not load the dictionary again.

Exceptions:

  • FileNotFoundError - the dictionary file does not exist;
  • OSError - the dictionary file cannot be read: no access, the path points to a directory;
  • ValueError - the file is not a ruMorph dictionary or is damaged;
  • TypeError - a path that is neither a string nor PathLike.
from rumorph import Analyzer

morph = Analyzer()
morph.analyze("стали")[0]

Analysis(word='стали', lemma='стать', pos='VERB', feats='Aspect=Perf|Mood=Ind|Number=Plur|Tense=Past|VerbForm=Fin|Voice=Mid', labels='intr', prob=0.9753459013836056, source='dictionary')

Parameters

Parameter Type Default Description
path str / PathLike None Path to the dictionary file; by default, the dictionary from the package

The object keeps the dictionary path in path of type Path, read-only. The dictionary file is mapped into memory: processes with the same file share its memory, so every further process with an analyzer takes about 1.4 MB. While an analyzer exists, the file must not be overwritten in place, but it can be replaced with a new file; on Windows an open file can be neither replaced nor deleted. The last loaded dictionary stays open even after the analyzers are deleted - until a dictionary from another file is loaded or the process exits. A dictionary is trusted data, like code: loading rejects a damaged file but not every deliberately crafted one, so dictionaries from untrusted sources should not be loaded.

Methods

Analyses of a word form

analyze · list[Analysis]

analyze(word)

Analyses of the word form in descending order of probability; the probabilities sum to 1. Case does not matter, and the letter «е» may stand for «ё»: analyze("ЕЖ") returns the analyses of the word form «ёж». A token that is not in the dictionary - a number, a symbol, a Latin word, an abbreviation - is analyzed by rules, and the analyses of an unknown Cyrillic word are guessed from its ending and parts. A token without spaces and invisible characters gets at least one analysis: if nothing fits, X without features.

Parameter Type Default Description
word str - Word form

Exceptions:

  • TypeError - a word that is not a string.
[(a.lemma, a.pos, a.feats.get("Case"), round(a.prob, 3)) for a in morph.analyze("стали")]
[(a.lemma, a.source) for a in morph.analyze("чатботу")][:2]

[('стать', 'VERB', None, 0.975),
 ('сталь', 'NOUN', 'Gen', 0.011),
 ('сталь', 'NOUN', 'Nom', 0.005),
 ('сталь', 'NOUN', 'Dat', 0.003),
 ('сталь', 'NOUN', 'Loc', 0.003),
 ('сталь', 'NOUN', 'Acc', 0.003)]
[('чатбот', 'guess'), ('чатбота', 'guess')]

Analyses of a list of word forms

analyze_many · list[list[Analysis]]

analyze_many(words)

Analyses of each word form, as with analyze; on a list the method is faster than analyze in a loop.

Parameter Type Default Description
words Iterable[str] - Word forms: a list, a tuple, a generator

Exceptions:

  • TypeError - a string instead of a collection of words, a word that is not a string.
[len(analyses) for analyses in morph.analyze_many(["мама", "мыла", "раму"])]

[1, 4, 2]

Lemmatization

lemmatize · list[str]

lemmatize(words)

Lemmas of the most probable analyses of the word forms; a token without analyses is returned in lowercase.

Parameter Type Default Description
words Iterable[str] - Word forms: a list, a tuple, a generator

Exceptions:

  • TypeError - a string instead of a collection of words, a word that is not a string.
morph.lemmatize("Мы стали друзьями в ковиде".split())

['мы', 'стать', 'друг', 'в', 'ковид']

Word form in the dictionary

is_known · bool

is_known(word)

Whether the word form is in the dictionary; as with analyze, case and «е» instead of «ё» do not matter. Words analyzed by rules or guessed are not in the dictionary.

Parameter Type Default Description
word str - Word form

Exceptions:

  • TypeError - a word that is not a string.
morph.is_known("Ежи"), morph.is_known("вайб")

(True, False)

Dictionary

The package dictionary is built from the OpenCorpora dictionary: about 3.1 million word forms. The probabilities of analyses come from the OpenCorpora corpus: a word written with «е» instead of «ё» gets the frequencies of the spelling with «ё», and word forms missing from the corpus get equally probable analyses. The dictionary is distributed under the CC BY-SA 3.0 license.

Threads and processes

One analyzer can be used from several threads: analyze_many and lemmatize release the GIL and process lists in parallel, and on the free-threaded Python 3.14t analyze, is_known, inflect and agree run in parallel too. A shared analyzer is not slower than a separate one per thread, so there is no need for an analyzer per thread. Without the GIL threads are limited by deleting the analysis objects: forms hardly speeds up in threads and analyze_many on long lists speeds up weakly, so such lists are better analyzed in chunks of a thousand or two words. The analyzer and analyses are pickled, so they work with multiprocessing and joblib: processes share the dictionary mapped into memory, however many analyses they receive. Passing analyses between processes costs more than the analysis itself, so in processes it is better to get lemmas and analysis fields, and analyses in threads.

from concurrent.futures import ThreadPoolExecutor

texts = [["мама", "мыла", "раму"], ["кошки", "ловят", "мышей"]]
with ThreadPoolExecutor(2) as pool:
    lemmas = list(pool.map(morph.lemmatize, texts))
lemmas

[['мама', 'мыло', 'рама'], ['кошка', 'ловить', 'мышь']]