Text highlighting¶

HighlightedText(source, layers=None, long_sent_word_factor=20, complex_syl_factor=4, stopwords=None, cliches=None, alliteration_threshold=0.001)

Highlighting of a text by layers, in the manner of the style checkers: every layer marks the fragments of a text that a statistic counts - long sentences, complex words, the passive. The data source is a text or a Doc object of spaCy. The result shows in Jupyter as HTML with styles and a legend; to_html returns the markup, and highlights keeps the fragments for a rendering of one's own. Fragments of different layers may overlap.

Raises:

SourceTypeError - a source that is neither a string nor a Doc;
SourceError - a source without words;
ParameterError - layers that are not a name or a list of names, a layer that is unknown or one that needs an annotation the source lacks.

highlight(source, layers=None, long_sent_word_factor=20, complex_syl_factor=4, stopwords=None, cliches=None, alliteration_threshold=0.001)

The function highlight returns a HighlightedText object of ruTS with the layers in the style of the Glavred and Turgenev services:

Group	Layer	What it marks	Statistic
Readability	`long_sents`	Sentences with at least `long_sent_word_factor` words	BasicStats, ReadabilityStats
	`complex_words`	Words with at least `complex_syl_factor` syllables	BasicStats, ReadabilityStats
	`rare_words`	Words whose lemma is outside the embedded top-10000 list; numbers, Latin script, abbreviations and stop words are not highlighted	LexicalStats
Syntax	`passive`	Passive verb forms together with the auxiliary verb	SyntaxStats
	`participle_clauses`	Participial clauses	SyntaxStats
	`converb_clauses`	Adverbial participle (converb) clauses	SyntaxStats
	`genitive_chains`	Genitive chains together with the governing word	SyntaxStats
	`split_predicates`	Split predicates from the verb to the nominal part	SyntaxStats
Officialese	`verbal_nouns`	Verbal nouns	StyleStats
	`compound_prepositions`	Compound prepositions from the `COMPOUND_PREPOSITIONS` list	StyleStats
	`cliches`	Clichés from the `OFFICIALESE_CLICHES` list or the `cliches` parameter	StyleStats
Style	`stopwords`	Water words by part of speech (`is_water_word`) or from the given list	StyleStats
	`parentheticals`	Parenthetical words and phrases	StyleStats
	`connectors`	Connectives by part of speech from the `Doc` annotation or ruMorph, the tooltip shows the class and type	CohesionStats
Phonics	`alliteration`	Repeats of a consonant in adjacent words that are unlikely given Russian letter frequencies	PhonStats

The groups are defined in ruts.constants.HIGHLIGHT_LAYER_GROUPS. The layers of the "Syntax" group need a Doc object with a dependency parse (the recommended pipeline or the ru_core_news_sm, ru_core_news_md, ru_core_news_lg models). The default layers are those of HIGHLIGHT_DEFAULT_LAYERS: long sentences, complex words, passive voice, genitive chains, split predicates, clichés. All fifteen layers at once overlap each other (a compound preposition consists of stop words), so pick the ones you need.

Alliteration

Alliteration is searched within a sentence as a chain of two or more adjacent words each of whose stems contains the same consonant letter. The stem is the common initial part of the word form and its ruMorph lemma (крупных → крупны, руках → рука), so the endings, which repeat consonants by grammar (своим целям и нуждам), do not count. The probability of a chain under an independent letter distribution is the product over the words of the probabilities of meeting the consonant among the stem letters, \(1 - (1 - f)^n\), where \(f\) is the frequency of the consonant in Russian texts and \(n\) the number of stem letters; a chain is highlighted if the probability is below the alliteration_threshold. The threshold is strict, since about twenty consonants are checked at every position: in prose about 5% of words are highlighted at 0.001, about 20% at 0.01, mostly random coincidences of frequent letters. Words shorter than three letters, stop words and words without vowels neither break nor continue a chain, and the letter й is not counted. The alliteration index of PhonStats measures how clustered the repeats are across the whole text; the highlighting shows where they are.

The class extends the HighlightedText of the anyTS core; the prefix of the CSS classes is ruts. The words of a string are those of WordsExtractor, and the parts of a hyphenated word of a Doc make one word. The sentences come from anySeg for a string and for a Doc without sentence boundaries (spacy.blank, a pipeline without parser and senter), so the long_sents layer is available for it too.

CodeResult

from ruts import load_pipeline
from ruts.visualizers import highlight

nlp = load_pipeline()
text = (
    "Проект, подготовленный за неделю, был одобрен советом без обсуждения. "
    "Повышение эффективности использования бюджетных средств обсуждалось, не выходя за рамки регламента. "
    "Участники, представлявшие региональные министерства, не смогли согласовать позиции по вопросам "
    "финансирования и распределения ответственности между ведомствами, поскольку каждое из них "
    "настаивало на собственной трактовке положений соглашения. "
    "Споры стихли, в кулуарах шумно шептались и шушукались, а решение было отложено "
    "до следующего заседания."
)
ht = highlight(nlp(text))
ht.counts
ht.highlights[:2]
highlight(nlp(text), layers="all").counts

{'long_sents': 1,
 'complex_words': 26,
 'passive': 4,
 'genitive_chains': 2,
 'split_predicates': 0,
 'cliches': 1}
(Highlight(start=8, end=22, layer='complex_words', note='сложное слово, 5 слогов'),
 Highlight(start=8, end=22, layer='passive', note='пассив без агенса'))
{'long_sents': 1,
 'complex_words': 26,
 'rare_words': 3,
 'passive': 4,
 'participle_clauses': 2,
 'converb_clauses': 1,
 'genitive_chains': 2,
 'split_predicates': 0,
 'verbal_nouns': 10,
 'compound_prepositions': 0,
 'cliches': 1,
 'stopwords': 7,
 'parentheticals': 0,
 'connectors': 4,
 'alliteration': 1}

In Jupyter the object ht displays like this (hover over a fragment to see the explanation):

Длинные предложения1Сложные слова26Пассив4Цепочки родительных2Расщеплённые сказуемые0Штампы1

Проект, подготовленный за неделю, был одобрен советом без обсуждения. Повышение эффективности использования бюджетных средств обсуждалось, не выходя за рамки регламента. Участники, представлявшие региональные министерства, не смогли согласовать позиции по вопросам финансирования и распределения ответственности между ведомствами, поскольку каждое из них настаивало на собственной трактовке положений соглашения. Споры стихли, в кулуарах шумно шептались и шушукались, а решение было отложено до следующего заседания.

A string without a spaCy model allows all layers but the syntactic ones, by default long sentences, complex words and clichés.

CodeResult

text = "Чуть слышно, бесшумно шуршат камыши. Повышение эффективности использования ресурсов обсуждалось."
highlight(text, layers=["complex_words", "alliteration"]).highlights

(Highlight(start=5, end=35, layer='alliteration', note='аллитерация на «ш»'),
 Highlight(start=37, end=46, layer='complex_words', note='сложное слово, 5 слогов'),
 Highlight(start=47, end=60, layer='complex_words', note='сложное слово, 5 слогов'),
 Highlight(start=61, end=74, layer='complex_words', note='сложное слово, 6 слогов'),
 Highlight(start=84, end=95, layer='complex_words', note='сложное слово, 4 слога'))

Parameters¶

Parameter	Type	Default	Description
`source`	`str` / `Doc`	-	Data source (a string or a Doc object)
`layers`	`list[str]` / `str`	`None`	Layers of the highlighting; if not given, the default layers the source allows; `"all"` - every layer allowed

The parameters of the layers of ruTS:

Parameter	Type	Default	Description
`long_sent_word_factor`	`int`	`20`	Minimum number of words in a long sentence
`complex_syl_factor`	`int`	`4`	Minimum number of syllables in a complex word
`stopwords`	`list[str]` / `set[str]`	`None`	List or set of water words; if not given, they are determined by part of speech with ruMorph
`cliches`	`list[str]` / `set[str]`	`None`	List or set of clichés; if not given, `OFFICIALESE_CLICHES` is used
`alliteration_threshold`	`float`	`0.001`	Probability threshold of a consonant repeat under an independent letter distribution, below which the repeat counts as alliteration

A threshold that is not an integer of at least one or a probability outside (0, 1] raises ParameterError, stop words or clichés that are not strings SourceTypeError.

Attributes¶

Attribute	Type	Description
`text`	`str`	Text of the data source
`layers`	`tuple[str]`	Layers turned on, in the order of drawing
`highlights`	`tuple[Highlight]`	Highlighted fragments, sorted by their start and then by descending end
`counts`	`dict[str, int]`	Number of fragments of every layer

Highlight(start, end, layer, note='')

A fragment of the highlighting: an immutable object with the parameters as its fields.

Field	Type	Default	Description
`start`	`int`	-	Position of the first character of the fragment in the text
`end`	`int`	-	Position after the last character of the fragment
`layer`	`str`	-	Layer of the highlighting
`note`	`str`	`''`	Explanation of the fragment for the tooltip

The note of a fragment of ruTS gives the number of words in the sentence, of syllables in the word, the length of the chain or the consonant of the alliteration.

Methods¶

to_html(legend=True, css=True) returns the HTML markup of the highlighted text: a div of the class <prefix>-highlight with the legend, its counts and the text, where a highlighted segment is a span of the classes <prefix>-hl and <prefix>-hl-<layer> with the notes in its title; <prefix> is css_prefix. Overlapping fragments of different layers give segments with several classes, in the order of drawing. Line breaks (\n, \r\n, \r) become character references, one per break, so the markup can go into Markdown.

Parameter	Type	Default	Description
`legend`	`bool`	`True`	Add the legend with the counts of the fragments
`css`	`bool`	`True`	Add the styles of the layers

The class method css() returns the styles to_html adds: the container, the legend and the text under the classes of css_prefix, the declarations of layer_styles for every layer, and a dark colour of the text on the layers with a background.

CodeResult

alliteration = highlight("Чуть слышно, бесшумно шуршат камыши.", layers="alliteration")
alliteration.to_html(legend=False, css=False)

'<div class="ruts-highlight"><div class="ruts-highlight-text">Чуть <span class="ruts-hl ruts-hl-alliteration" title="аллитерация на «ш»">слышно, бесшумно шуршат камыши</span>.</div></div>'