Components

StatsComponent(nlp, name)

The base of the components of spaCy that put the statistics of a text into an extension of Doc: a component registers the extension when added to a pipeline and puts the statistics of every document into doc._.<name>. Writing components in general is described in the documentation of spaCy.

The components of ruTS are subclasses of the StatsComponent of the anyTS core, one for every class of statistics.

Note

The examples below use the recommended pipeline on the ru_core_news_sm model, which is installed separately: python -m spacy download ru_core_news_sm (see installation).

CodeResult

import ruts

nlp = ruts.load_pipeline()
nlp.add_pipe("ruts_basic", name="basic")
nlp("мама мыла раму")._.basic.c_letters
nlp("?!")._.basic is None

{4: 3}
True

Parameters

Parameter	Type	Default	Description
nlp	Language	-	Pipeline the component is added to
name	str	-	Name of the component in the pipeline and of the extension

spaCy sets both at nlp.add_pipe(factory, name=...); the other parameters of a component are passed in the config dictionary of the same call.

Names

The factories carry the prefix ruts_ (table below, and the component of sentence boundaries ruts_blank_lines). They are declared as entry points of spacy_factories, so spaCy finds them without import ruts, and a pipeline saved with them (nlp.to_disk(path)) loads with spacy.load(path).

The name of the pipe is the name of the extension: nlp.add_pipe(factory, name="basic") puts the statistics into doc._.basic. Without name both take the name of the factory. The extension follows the pipe when the pipeline renames it (nlp.rename_pipe). A name taken by an extension of another package raises ParameterError instead of replacing it: at add_pipe, or at the first document after rename_pipe, which renames the pipe before the component learns its name. A component taken from another pipeline (add_pipe(name, source=other)) is the same object in both, so under a new name it writes to that extension in the other pipeline too; a separate component comes from the factory, add_pipe(factory, name=...). The same component can be added twice under different names, with different parameters. A document with no words - an empty string, whitespace, punctuation alone - passes through a component untouched, its extension left at None.

In ruts_lexical a document of numbers alone passes untouched as well, and in ruts_verse a document without letters.

Serialization

A component keeps an object of statistics in doc._.<name>, and spaCy cannot serialize it: Doc.to_bytes(), DocBin(store_user_data=True) and nlp.pipe(..., n_process>1) fail with these components in the pipeline. To save a document, leave the user data out (doc.to_bytes(exclude=["user_data"])) or keep doc._.<name>.get_stats() separately; for multiprocessing compute the statistics in the main process after nlp.pipe, without the components.

Adding a component adds to the spaCy tokenizer the rules for the dialogue dashes glued to words (ruts.utils.add_dash_rules), so that a dialogue dash is split off a word as in a string; a pipeline without ruTS components needs this call of its own. The anySeg tokenizer of the recommended pipeline is left as it is.

ruTS removes stress marks from the words of a Doc, but the model parses such words worse: on a text after accentuate the tagger of ru_core_news_sm changes the part of speech of over half of the words, and the dependencies change in the recommended pipeline as well. Pass a text with stresses without them - nlp(ruts.utils.strip_marks(text)).

Recommended pipeline

load_pipeline(model='ru_core_news_sm', exclude=(), **kwargs)

A spaCy pipeline in which a Doc is annotated the same way as a string, except the hyphenated words such as муж-то and два-три: a string splits them into parts, a Doc keeps one word. The tokens and the sentences come from the anySeg segmenter, the parts of speech, features and lemmas from the ruMorph morphological analyzer in the context of the sentence, and the model gives the dependency parser and the named entities. A saved pipeline of this kind loads as it is. The same by hand:

nlp = spacy.load(
    "ru_core_news_sm",
    exclude=["morphologizer", "lemmatizer"],
    config={"nlp": {"tokenizer": {"@tokenizers": "anyseg.Tokenizer.v1"}}},
)
nlp.add_pipe("anyseg_sentencizer", first=True)
nlp.add_pipe("rumorph", after="anyseg_sentencizer")

On UD SynTagRus this pipeline is more accurate than the model itself: the part of speech is right for 97.3 % of the words against 93.3 %, the dependency with its relation (LAS) for 82.7 % against 82.1 %. A plain spacy.load("ru_core_news_sm") is supported too, but then the parts of speech, features and sentences come from the model, and the statistics of a Doc may differ from those of the string. ruTS does not work around the errors of its tagger: the model tags connectors such as значит and т.е. as a verb and a punctuation mark, so they are not found (11 in 71 thousand words of TextsByGrade and SovChLit), and боишься as an infinitive, so the subordinate clause with it counts as an infinitive clause.

Parameter	Type	Default	Description
model	str	"ru_core_news_sm"	Name or path of the spaCy model: ru_core_news_sm, ru_core_news_md, ru_core_news_lg
exclude	str, list[str]	()	More components of the model not to load, for example ner
**kwargs	-	-	Other parameters of spacy.load; the tokenizer in config is replaced with anySeg

CodeResult

from ruts import load_pipeline

pipeline = load_pipeline()
pipeline.pipe_names
[(token.text, token.pos_, token.lemma_) for token in pipeline("Нож из стали.")]

['anyseg_sentencizer', 'rumorph', 'tok2vec', 'parser', 'attribute_ruler', 'ner']
[('Нож', 'NOUN', 'нож'), ('из', 'ADP', 'из'), ('стали', 'NOUN', 'сталь'), ('.', 'PUNCT', '.')]

Components of the classes of statistics

Component	Factory	Class of statistics	Parameters
BasicStatsComponent	ruts_basic	BasicStats	-
MorphStatsComponent	ruts_morph	MorphStats	-
ReadabilityStatsComponent	ruts_readability	ReadabilityStats	preset, basic
UnadaptedReadabilityStatsComponent	ruts_unadapted_readability	UnadaptedReadabilityStats	basic
DiversityStatsComponent	ruts_diversity	DiversityStats	window_len, mtld_threshold, mtld_min_len, hdd_sample_size, log_base
StyleStatsComponent	ruts_style	StyleStats	stopwords, top_n, nausea_stopwords, cliches
PhonStatsComponent	ruts_phon	PhonStats	window_len
SyntaxStatsComponent	ruts_syntax	SyntaxStats	-
CohesionStatsComponent	ruts_cohesion	CohesionStats	blank_lines
LexicalStatsComponent	ruts_lexical	LexicalStats	data_dir
VerseStatsComponent	ruts_verse	VerseStats	data_dir

Basic statistics

BasicStatsComponent(nlp, name)

BasicStatsComponent puts the basic statistics of a document into its extension.

Morphological statistics

MorphStatsComponent(nlp, name)

MorphStatsComponent puts the morphological statistics into the extension; by default they need the parts of speech before it.

In ruTS the parts of speech and the features come from the annotation of the pipeline (token.pos_, token.morph), and without it from the ruMorph analysis by word.

CodeResult

nlp.add_pipe("ruts_morph", name="morph")
nlp("мама мыла раму")._.morph.case

('Nom', None, 'Acc')

Readability metrics

ReadabilityStatsComponent(nlp, name, preset=None, basic=None)

ReadabilityStatsComponent puts the readability metrics into the extension, computed from the document or, with basic, from the basic statistics another component put into it. A library sets basic_factory, the factory of its component of the basic statistics, which the error names when that extension holds none.

Parameter	Type	Default	Description
preset	str	None	Preset of the coefficients, one of the presets of the class; the default of the class if None
basic	str	None	Extension of a component of the basic statistics to reuse instead of computing them again

With preset left at None the config of a saved pipeline holds no preset, and the pipeline takes the default of the class when it is loaded; a library that wants its default written into the config gives it in the signature of its subclass.

Raises:

• ParameterError - a stats_class that is not concrete, a preset that is not a string or not one of the class, an extension not named by a string;
• SourceTypeError - a class without basic_stats_class and no basic;
• SourceError - a document whose extension basic holds no basic statistics of the class.

The presets of ruTS are plainrussian (the default), fiction and academic. In ruTS the errors of stats_class and SourceTypeError do not occur, and the message of SourceError names the component ruts_basic to add before it.

CodeResult

nlp.add_pipe("ruts_readability", name="readability")
config = {"preset": "fiction", "basic": "basic"}
nlp.add_pipe("ruts_readability", name="readability_fiction", config=config)
doc = nlp("Кот сидел на окне и смотрел на птиц, которые летали над крышами домов.")
doc._.readability.flesch_kincaid_grade
doc._.readability_fiction.flesch_kincaid_grade

-0.1506153846153886
6.41769230769231

Formulas of other languages

UnadaptedReadabilityStatsComponent(nlp, name, basic=None)

UnadaptedReadabilityStatsComponent puts the readability formulas of other languages into the extension, computed from the document or, with basic, from the basic statistics another component put into it. A library sets basic_factory, as for ReadabilityStatsComponent.

Parameter	Type	Default	Description
basic	str	None	Extension of a component of the basic statistics to reuse instead of computing them again

Raises:

• ParameterError - a stats_class that is not concrete, an extension not named by a string;
• SourceTypeError - a class without basic_stats_class and no basic;
• SourceError - a document whose extension basic holds no basic statistics of the class.

The formulas are those of UnadaptedReadabilityStats, with their original coefficients and without a calibration for Russian; the errors in ruTS are as for ReadabilityStatsComponent.

CodeResult

nlp.add_pipe("ruts_unadapted_readability", name="unadapted", config={"basic": "basic"})
doc = nlp("Кот сидел на окне и смотрел на птиц, которые летали над крышами домов.")
doc._.unadapted.gulpease, doc._.unadapted.forcast
doc._.unadapted.bs is doc._.basic

(69.0, 13.076923076923077)
True

Lexical diversity metrics

DiversityStatsComponent(nlp, name, window_len=50, mtld_threshold=0.72, mtld_min_len=10, hdd_sample_size=42, log_base=10)

DiversityStatsComponent puts the lexical diversity metrics into the extension, by a class of the library that reads a Doc. Its parameters are those of DiversityStats, with the same defaults.

Raises:

• ParameterError - a wrong parameter (check_params of the diversity).

Parameter	Type	Default	Description
window_len	int	50	Window size for MATTR and segment size for MSTTR
mtld_threshold	float	0.72	TTR threshold for MTLD, MA-MTLD and MTLD-W
mtld_min_len	int	10	Minimum factor length for MTLD, MA-MTLD and MTLD-W
hdd_sample_size	int	42	Sample size for HD-D
log_base	float	10	Logarithm base for the Summer, Maas and Dugast metrics

CodeResult

config = {"window_len": 100, "log_base": 2.718281828459045}
nlp.add_pipe("ruts_diversity", name="diversity", config=config)
nlp("мама мыла раму")._.diversity.rttr

1.7320508075688774

SEO style metrics

StyleStatsComponent(nlp, name, stopwords=None, top_n=10, nausea_stopwords=None, cliches=None)

StyleStatsComponent puts the style metrics into the extension.

Parameter	Type	Default	Description
stopwords	list[str]	None	Stopwords of the water content; those of the class if None
top_n	int	10	Number of the most frequent words of the academic nausea and of the naturalness by Zipf's law (NAUSEA_TOP_N)
nausea_stopwords	list[str]	None	Words left out of the most frequent words of the academic nausea; those of the class if None
cliches	list[str]	None	Cliches; those of the class (cliches_list) if None

Raises:

• SourceTypeError - stopwords, nausea stopwords or cliches that are not a list of strings;
• ParameterError - a top_n that is not an integer of at least one.

Without the lists the water words and the stopwords of the nausea are determined by the ruMorph tags, and the cliches come from OFFICIALESE_CLICHES.

CodeResult

nlp.add_pipe("ruts_style", name="style", config={"stopwords": ["и", "в", "не"], "top_n": 5})
nlp("мама мыла раму")._.style.water
nlp.add_pipe("ruts_style", name="style_cliches", config={"cliches": ["в целях", "в рамках"]})
nlp("В целях экономии в рамках проекта мы закрыли склад.")._.style_cliches.cliches

0.0
22.22222222222222

Phonostatistics

PhonStatsComponent(nlp, name, window_len=3)

PhonStatsComponent puts the phonetic statistics into the extension; window_len (PHON_WINDOW_LEN by default) is the window in words of the alliteration and the assonance.

Raises:

• ParameterError - a window_len that is not an integer of at least 2.

CodeResult

nlp.add_pipe("ruts_phon", name="phon", config={"window_len": 5})
nlp("мама мыла раму")._.phon.p_open_syllables

1.0

Syntactic statistics

SyntaxStatsComponent(nlp, name)

SyntaxStatsComponent puts the syntactic statistics into the extension; they need a parser and the parts of speech before it.

The recommended pipeline and the ru_core_news_sm, ru_core_news_md and ru_core_news_lg models have a parser and the parts of speech.

Raises:

• SourceError - a document without a dependency parse or parts of speech (rumorph of the recommended pipeline or the morphologizer of a model).

CodeResult

nlp.add_pipe("ruts_syntax", name="syntax")
nlp("Дом, построенный рабочими в прошлом году, был продан")._.syntax.tree_depth

4.0

Cohesion statistics

CohesionStatsComponent(nlp, name, blank_lines=False)

CohesionStatsComponent puts the cohesion statistics into the extension. It needs no parser: a Doc without sentence boundaries is split into sentences by SentsExtractor. The parts of speech come from the annotation, and the lemmas from ruMorph by the part of speech of the token; without annotation both come from the ruMorph analysis, as for a string.

Parameter	Type	Default	Description
blank_lines	bool	False	Start the paragraphs after the blank lines alone, for a text whose lines are wrapped inside its paragraphs; a paragraph is a line by default

Raises:

• ParameterError - a blank_lines that is not a bool.

CodeResult

nlp.add_pipe("ruts_cohesion", name="cohesion")
nlp.add_pipe("ruts_cohesion", name="paragraphs", config={"blank_lines": True})
doc = nlp("Кот сидел на окне. Он смотрел на птиц. Птицы улетели,\nи кот уснул.")
(
    doc._.cohesion.noun_overlap_adjacent,
    doc._.cohesion.n_paragraphs,
    doc._.paragraphs.n_paragraphs,
)

(0.5, 2, 1)

Lexical sophistication statistics

LexicalStatsComponent(nlp, name, data_dir=None)

LexicalStatsComponent puts the lexical sophistication statistics into the extension. The frequency dictionary metrics require a downloaded FreqDict; the bands and lexical density are computed without it.

Parameter	Type	Default	Description
data_dir	str	None	Path to the frequency dictionary directory; if not given, the default directory is used

Raises:

• DatasetNotFoundError - the frequency dictionary metrics and get_stats() when FreqDict is not downloaded.

CodeResult

nlp.add_pipe("ruts_lexical", name="lexical")
nlp("Кот сидел на окне и смотрел на птиц")._.lexical.p_top1000

0.75

Verse statistics

VerseStatsComponent(nlp, name, data_dir=None)

VerseStatsComponent puts the verse statistics into the extension; it requires a downloaded StressDict stress dictionary. The lines of a poem are the lines of the Doc text, so pass the poem to nlp without joining them. A text with letters but without Russian words gives empty statistics, as in VerseStats.

Parameter	Type	Default	Description
data_dir	str	None	Path to the stress dictionary directory; if not given, the default directory is used

Raises:

• DatasetNotFoundError - the stress dictionary is not downloaded; the first document with letters raises it.

CodeResult

nlp.add_pipe("ruts_verse", name="verse")
poem = (
    "Буря мглою небо кроет,\nВихри снежные крутя;\n"
    "То, как зверь, она завоет,\nТо заплачет, как дитя"
)
doc = nlp(poem)
doc._.verse.meter, doc._.verse.n_feet

('хорей', 4)

Sentences after blank lines

start_sents_after_blank_lines(doc)

A component that starts a sentence at the first token after a blank line - a line break, optional whitespace and another line break - so that a heading, a list item or a paragraph without a final mark is a sentence of its own, as in the sentence extractor. It goes before the parser, senter or sentencizer, which keeps its starts and places the others. The whitespace before such a start stays with the sentence before it, and the marks that open a paragraph (a dash of a dialogue, ¡, «) stay with its first word. Alone, the component leaves the sentences within a paragraph unsplit and sets partial boundaries (has_sents), so the statistics take the sentences of such a document from its text. A parsed document, such as one passed through the pipeline again, passes untouched, since spaCy keeps its sentences. A language library registers the component under its prefix, Language.component("<prefix>_blank_lines", func=start_sents_after_blank_lines).

Parameter	Type	Default	Description
doc	Doc	-	Doc object

In ruTS the component is registered as ruts_blank_lines. Without it the parser of the ru_core_news_sm model glues a paragraph without a final mark to the next sentence; the recommended pipeline does not need it.

CodeResult

import spacy

nlp = spacy.load("ru_core_news_sm")
nlp.add_pipe("ruts_blank_lines", before="parser")
[sent.text.strip() for sent in nlp("Что нового\n\nБыстрый импорт. Меньше ошибок").sents]

['Что нового', 'Быстрый импорт.', 'Меньше ошибок']