SEO style metrics¶

StyleStats(source, words_extractor=None, stopwords=None, top_n=10, cliches=None, nausea_stopwords=None)

The style metrics of a text: the SEO metrics of Advego and Text.ru and the markers of the officialese style. The services publish no formulas, so the common definitions are taken. The data source is a text or a Doc object of spaCy.

The SEO metrics are counted on the words of the word extractor, in lower case by default, or on the tokens of a Doc; an extractor passed explicitly is used on the text of a Doc. The markers are counted on the word forms in lower case (forms), whatever the extractor.

Raises:

SourceTypeError - a source that is neither a string nor a Doc, an extractor of another type, a list that is not a list of strings;
SourceError - a source without words;
ParameterError - a top_n that is not an integer of at least one.

The class extends the StyleStats of the anyTS core with the hooks of Russian of the metric cards; they tag a word by the first ruMorph analysis, without its neighbours. The words of a string are extracted by the Russian WordsExtractor without lemmatization, so the forms of one word count as different words, as in Advego; to compute by lemmas, pass a WordsExtractor with use_lexemes=True and lowercase=True. The words are compared with the stop words, the phrases and the keywords ignoring case and the letter ё (normalize_yo). The words of a Doc lose their stress marks, and the parts of a hyphenated word that spaCy splits are joined (во-первых is one word). The print_stats table and the error messages are in Russian (STYLE_STATS_DESC and CORE_MESSAGES of ruts.constants).

CodeResult

from ruts import StyleStats

text = "Ног нет, а хожу, рта нет, а скажу: когда спать, когда вставать, когда работу начинать"
StyleStats(text).print_stats()

                     Метрика                     | Значение
------------------------------------------------------------
Классическая тошнота                             |   1.73
Академическая тошнота (%)                        |  53.33
Водность (%)                                     |  33.33
Заспамленность (%)                               |  26.67
Естественность по Ципфу (%)                      |  33.33
Отглагольные существительные (% существительных) |   0.00
Производные предлоги (на 100 слов)               |   0.00
Вводные слова (на 100 слов)                      |   0.00
Штампы (на 100 слов)                             |   0.00

Parameters¶

Parameter	Type	Default	Description
`source`	`str` / `Doc`	-	Data source
`words_extractor`	`WordsExtractor`	`None`	Word extraction tool; used for a `Doc` too, on its text
`stopwords`	`list[str]`	`None`	Words of water; without them `is_water_word` decides
`top_n`	`int`	`10`	Number of the most frequent words of the academic nausea and the naturalness by Zipf's law (`NAUSEA_TOP_N`)
`cliches`	`list[str]`	`None`	Cliches; `cliches_list` of the class by default
`nausea_stopwords`	`list[str]`	`None`	Stopwords of the academic nausea; without them `is_stopword` decides

A set of words is taken in sorted order.

NAUSEA_TOP_N is a constant of anyts.constants.

Attributes¶

Attribute	Type	Description
`words`	`tuple[str]`	Words of the SEO metrics
`forms`	`tuple[str]`	Word forms in lower case, of the markers

The metrics are properties computed on each access.

Metrics¶

get_stats gives all the metrics. A metric is computed by the function of its card on words or forms with the hooks of the class.

Metric	Measures	Range	Higher means
`classic_nausea`	Repetition of the most frequent word	0 and above, grows with the text	One word pressed harder
`academic_nausea`	Share of the most frequent significant words	0-100 %	A narrower vocabulary
`water`	Share of the words without content	0-100 %	Less content per word
`spam`	Share of the repeated words	0-100 %	More repetition
`zipf_naturalness`	Agreement of the top frequencies with Zipf's law	0-100 %, `nan` without ranks to compare	Closer to a natural text
`verbal_nouns`	Share of the verbal nouns among the nouns	0-100 %, `nan` without nouns	A more nominal style
`compound_prepositions`	Compound prepositions per 100 words	0-100	More officialese
`parentheticals`	Parenthetical expressions per 100 words	0-100	More insertions
`cliches`	Cliches per 100 words	0-100	More cliches

The bands of the cards are the norms of Advego, Text.ru, pr-cy and megaindex, in Russian (STYLE_NORMS), which describe reads. They are guidelines of the services for website texts of several hundred words: on short texts nausea and spam are uninformative, and the norms do not apply to fiction. The markers of the officialese have no norms, and describe gives None for them.

Classic nausea¶

classic_nausea · Advego

calc_classic_nausea(text)

How much one word is pressed: the square root of the occurrences \(f_1\) of the most frequent word. The value is not corrected for the length of the text, so it grows with the text; no words give 0.

\[ \sqrt{f_1} \]

Parameter	Type	Default	Description
`text`	`list[str]`	-	List of words

Value	Reading
`[0, 5]`	the Advego norm
`(5, 7]`	at the upper bound of the Advego norm
`(7, ∞)`	above the Advego norm

Academic nausea¶

academic_nausea · nausea_words() · Advego

calc_academic_nausea(text, top_n=10, stopwords=None)

The share of the occurrences of the top_n most frequent significant words among all the words, in percent.

\[ 100 \cdot \frac{\sum_{w \in \textrm{top}_n} f_w}{\textrm{Number of words}} \]

Parameter	Type	Default	Description
`text`	`list[str]`	-	List of words
`top_n`	`int`	`10`	Number of the most frequent words
`stopwords`	`list[str]`	`None`	Stop words; without them `is_stopword` decides

Value	Reading
`[0, 5)`	below the Advego norm
`[5, 15]`	the Advego norm
`(15, 100]`	above the Advego norm

find_nausea_words(text, top_n=10, stopwords=None)

The top_n most frequent words without the stop words, those the academic nausea counts, with their occurrences: the most frequent first, words of equal frequency in the order of the text.

Parameter	Type	Default	Description
`text`	`list[str]`	-	List of words
`top_n`	`int`	`10`	Number of the most frequent words
`stopwords`	`list[str]`	`None`	Stop words; without them `is_stopword` decides

is_stopword(word)

Whether a word is insignificant for the academic nausea: conjunctions, particles, prepositions, pronouns, determiners and interjections (STOPWORD_POS: CCONJ, SCONJ, PART, ADP, PRON, DET, INTJ), predicatives (надо, можно, нет) and бы, demonstrative, interrogative and relative adverbs (там, тогда, где, почему) and the parenthetical words of is_parenthetical.

CodeResult

from ruts.style_stats import find_nausea_words

find_nausea_words(["кот", "и", "кот", "и", "пес"], top_n=1)

{'кот': 2}

Water content¶

water · Text.ru

calc_water(text, stopwords=None)

The share of the words without content, in percent: the words of stopwords or, without the list, the words of is_water_word. Advego's "water" is a different indicator; it is not implemented here.

Parameter	Type	Default	Description
`text`	`list[str]`	-	List of words
`stopwords`	`list[str]`	`None`	Water words

Value	Reading
`[0, 15)`	natural water content by Text.ru
`[15, 30]`	excessive water content by Text.ru
`(30, 100]`	high water content by Text.ru

is_water_word(word)

Whether a word is "water": conjunctions, particles (and бы), pronominal nouns and interjections (WATER_POS: CCONJ, SCONJ, PART, PRON, INTJ), его, её, их and the parenthetical words of is_parenthetical. Prepositions, determiners and predicatives, unlike in is_stopword, are not water.

Spam score¶

spam · Text.ru

calc_spam(text)

The share of the repeated words in percent: every occurrence of a word after the first.

\[ 100 \cdot \left(1 - \frac{\textrm{Number of lexemes}}{\textrm{Number of words}}\right) \]

Parameter	Type	Default	Description
`text`	`list[str]`	-	List of words

Value	Reading
`[0, 30)`	natural text by Text.ru
`[30, 60]`	SEO-optimized text by Text.ru
`(60, 100]`	spammed text by Text.ru

Naturalness by Zipf's law¶

zipf_naturalness · pr-cy, megaindex

calc_zipf_naturalness(text, top_n=10)

The agreement of the frequencies of the most frequent words with the ideal distribution of Zipf's law, \(f_r = f_1 / r\), where \(f_1\) is the frequency of the most frequent word and \(r\) the rank. The ranks run from 2 to \(R = \min(\textrm{top}_n, V, f_1)\), where \(V\) is the number of lexemes. Negative values are cut to 0. Without ranks to compare - all the words hapaxes, one lexeme or top_n below 2 - the value is nan.

\[ 100 \cdot \left(1 - \frac{1}{R - 1} \sum_{r=2}^{R} \frac{|r \cdot f_r - f_1|}{f_1}\right) \]

Parameter	Type	Default	Description
`text`	`list[str]`	-	List of words
`top_n`	`int`	`10`	Number of the most frequent words

Value	Reading
`[0, 50)`	below the pr-cy and megaindex norm
`[50, 100]`	the pr-cy and megaindex norm

Verbal nouns¶

verbal_nouns

calc_verbal_nouns(text)

The share of the verbal nouns among the nouns, in percent; nan for a text without nouns. A noun is verbal when its lemma ends with a suffix of VERBAL_NOUN_SUFFIXES (-ние, -нье, -тие, -тье, -ствие, -ция) or belongs to VERBAL_NOUN_LEMMAS (производство, руководство). The heuristic also catches non-verbal words with the same suffixes (здание). find_verbal_nouns(text) returns the bounds of the words found as slices of the list.

Parameter	Type	Default	Description
`text`	`list[str]`	-	List of words

Compound prepositions and cliches¶

compound_prepositions · cliches

calc_phrase_density(text, phrases)

The occurrences of the phrases of a list per 100 words. A phrase starting with an infinitive (after «не» too) is found in any form of its verb, participles and gerunds included («имело место», «доведено до сведения»); expand_phrases(text, phrases) gives these variants from the text with their phrases. StyleStats counts this way the compound prepositions of COMPOUND_PREPOSITIONS (в целях, в связи с) and the cliches cliches, by default of OFFICIALESE_CLICHES (на сегодняшний день, иметь место).

Parameter	Type	Default	Description
`text`	`list[str]`	-	List of words
`phrases`	`list[str]`	-	Phrases with words separated by spaces

Parenthetical words¶

parentheticals

calc_parentheticals(text)

The number of parenthetical words per 100 words: the phrases of PARENTHETICALS (таким образом, как правило) and, outside them, the single parenthetical words of is_parenthetical(word) - with the label Prnt (конечно, например) or with a lemma of PARENTHETICAL_WORDS (кстати, наверно). find_parentheticals(text) returns the bounds of the words and phrases found as slices of the list.

Parameter	Type	Default	Description
`text`	`list[str]`	-	List of words

Keyword density¶

keyword_density(*keywords) · Text.ru

calc_keyword_density(text, keywords)

The occurrences of every keyword per 100 words. A keyword of several words separated by spaces is a sequence of words whose occurrences may overlap; the keywords of a set come in sorted order.

Parameter	Type	Default	Description
`text`	`list[str]`	-	List of words
`keywords`	`list[str]`	-	Keywords or phrases

The functions check their arguments: words, stop words, keywords and phrases that are not a list of strings raise SourceTypeError, and a top_n that is not an integer of at least one ParameterError.

Methods¶

keyword_density(*keywords) gives the density of keywords and phrases in percent (calc_keyword_density with normalize of the class), nausea_words() the most frequent significant words of the academic nausea with their occurrences (find_nausea_words).

markers() returns the markers of the officialese found by metric - verbal_nouns, compound_prepositions, parentheticals, cliches - as the forms write them, in the order of the text.

CodeResult

from ruts import StyleStats

text = "В целях повышения качества в кратчайшие сроки, как правило, проводится проверка"
StyleStats(text).markers()
StyleStats(text, top_n=2).nausea_words()

{'verbal_nouns': ('повышения',),
 'compound_prepositions': ('в целях',),
 'parentheticals': ('как правило',),
 'cliches': ('в кратчайшие сроки',)}
{'целях': 1, 'повышения': 1}

The other methods - get_stats, to_dataframe, print_stats and describe - are those of the statistics interface of the anyTS core.

CodeResult

from ruts import StyleStats

text = "Ног нет, а хожу, рта нет, а скажу: когда спать, когда вставать, когда работу начинать"
ss = StyleStats(text)
ss.keyword_density("когда", "нет а")
ss.describe("water"), ss.describe("cliches")
ss.get_stats()

{'когда': 20.0, 'нет а': 13.333333333333334}
('высокая водность по Text.ru', None)
{'classic_nausea': 1.7320508075688772,
 'academic_nausea': 53.333333333333336,
 'water': 33.333333333333336,
 'spam': 26.666666666666668,
 'zipf_naturalness': 33.333333333333336,
 'verbal_nouns': 0.0,
 'compound_prepositions': 0.0,
 'parentheticals': 0.0,
 'cliches': 0.0}