Basic statistics¶

BasicStats(source, sents_extractor=None, words_extractor=None, normalize=False, complex_syl_factor=4, long_word_letter_factor=6)

The basic statistics of a text: the counts of its sentences, words, characters, syllables and punctuation marks and the distributions of its words. The data source is a text or a Doc object of spaCy. Letters are counted by str.isalpha, so digits, hyphens and marks inside a word are not letters.

The words and the sentences of a string come from the extractors. A Doc gives its tokens as words, without punctuation marks and symbols such as € or %, and its boundaries as sentences, or those of the sentence extractor when it has none or only partial ones (has_sents); an extractor given is used on its text instead. A blank line ends a sentence of a string and of a Doc without boundaries; a Doc with boundaries does so when its pipeline has start_sents_after_blank_lines (registered by a language library under its own name) before the parser, senter or sentencizer. A sentence without words, such as punctuation alone, is not counted.

Raises:

SourceTypeError - a source that is neither a string nor a Doc, an extractor of another type;
SourceError - a source without words;
ParameterError - a threshold that is not an integer of at least one, a normalize that is not a bool.

The class extends the BasicStats of the anyTS core for Russian: the syllables are counted by count_syllables, the default extractors are the Russian SentsExtractor and WordsExtractor, the punctuation marks are counted by the count_punctuations of ruTS, and a hyphenated word that spaCy splits stays one word (во-первых). The blank line component of ruTS is ruts_blank_lines: nlp.add_pipe("ruts_blank_lines", before="parser"); the recommended pipeline does not need it. Stress marks and soft hyphens count neither as characters nor as parts of words, and й and ё of two characters are one letter (ruts.utils.strip_marks): a text with stresses gives the same statistics as without them. The statistics are computed when the object is created; the print_stats table is in Russian.

CodeResult

from ruts import BasicStats

text = "Существуют три вида лжи: ложь, наглая ложь и статистика"
bs = BasicStats(text, normalize=True)
bs.print_stats()

            Статистика            | Значение
---------------------------------------------
Предложения                       |    1
Слова                             |    9
Уникальные слова                  |    8
Длинные слова                     |    3
Сложные слова                     |    2
Простые слова                     |    7
Односложные слова                 |    5
Многосложные слова                |    4
Символы                           |    55
Буквы                             |    45
Пробелы                           |    8
Слоги                             |    18
Знаки препинания                  |    2
Энтропия длин слов в буквах (бит) |   2.20
Энтропия длин слов в слогах (бит) |   1.66

Parameters¶

Parameter	Type	Default	Description
`source`	`str` / `Doc`	-	Data source (a string or a Doc object)
`sents_extractor`	`SentsExtractor`	`None`	Sentence extraction tool; used for a Doc too, on its text
`words_extractor`	`WordsExtractor`	`None`	Word extraction tool; used for a Doc too, on its text
`normalize`	`bool`	`False`	Compute normalized statistics
`complex_syl_factor`	`int`	`COMPLEX_SYL_FACTOR`	Minimum number of syllables in a complex word
`long_word_letter_factor`	`int`	`LONG_WORD_LETTER_FACTOR`	Minimum number of letters in a long word

The object keeps normalize, complex_syl_factor and long_word_letter_factor under their names.

In ruTS a complex word has four or more syllables and a long word six or more letters: COMPLEX_SYL_FACTOR = 4 and LONG_WORD_LETTER_FACTOR = 6 of ruts.constants.

Statistics¶

Statistic	Type	Description	Normalized
`c_letters`	`dict[int, int]`	Distribution of words by number of letters	-
`c_syllables`	`dict[int, int]`	Distribution of words by number of syllables	-
`n_sents`	`int`	Number of sentences containing words	-
`n_words`	`int`	Number of words	-
`n_unique_words`	`int`	Number of unique words, case ignored	`p_unique_words`
`n_long_words`	`int`	Number of long words, of at least `long_word_letter_factor` letters	`p_long_words`
`n_complex_words`	`int`	Number of complex words, of at least `complex_syl_factor` syllables	`p_complex_words`
`n_simple_words`	`int`	Number of simple words: with a syllable, below the complex ones	`p_simple_words`
`n_monosyllable_words`	`int`	Number of monosyllabic words	`p_monosyllable_words`
`n_polysyllable_words`	`int`	Number of polysyllabic words, of two syllables or more	`p_polysyllable_words`
`n_chars`	`int`	Number of characters without the line breaks	-
`n_letters`	`int`	Number of letters	`p_letters`
`n_spaces`	`int`	Number of spaces of Unicode (category Zs), such as the no-break space, and tabs	`p_spaces`
`n_syllables`	`int`	Number of syllables	-
`n_punctuations`	`int`	Number of punctuation marks	`p_punctuations`
`letters_entropy`	`float`	Shannon entropy in bits of the word lengths in letters, of `c_letters`: 0 when the words have one length; the estimate by the shares runs lower on a short text, by tenths of a bit on tens of words	-
`syllables_entropy`	`float`	Shannon entropy in bits of the word lengths in syllables, of `c_syllables`, with the same bias	-
`c_punctuations`	`dict[str, int]`	Distribution of punctuation marks by type (`count_punctuations`)	-

With normalize=True the object has the normalized statistics too, of the type float: the counts of the words divided by the number of words, those of the characters by the number of characters.

Methods¶

count_words_by_syllables(min_syllables) and count_words_by_letters(min_letters) return the number of words with at least the given number of syllables or letters; a minimum that is not an integer raises ParameterError.

This is how ReadabilityStats counts the complex words of SMOG (5 or more syllables) and the long words of LIX (7 or more letters).

get_stats(flat) returns a dictionary with the distributions c_letters and c_syllables, the statistics of stats_desc, the distribution c_punctuations and, with normalize=True, the normalized counts, as a copy: editing it does not change the object. With flat=True the distributions give a key for every value the text has, such as c_letters_3 and c_punctuations_comma.

print_stats() prints a table with the statistics of stats_desc, their descriptions and the headers from stats_headers.

The normalized statistics p_* are not in the table.

to_dataframe() returns the flat statistics, get_stats(flat=True), as a one-row DataFrame of pandas with a column for every key. A dictionary statistic gives columns only for the keys the text has, so in frames of several texts joined with pandas.concat a count a text lacks is NaN. Only these columns may be filled with 0: elsewhere NaN is an undefined value.

CodeResult

bs.count_words_by_syllables(5), bs.count_words_by_letters(7)
bs.get_stats()

(0, 2)
{'c_letters': {1: 1, 3: 2, 4: 3, 6: 1, 10: 2},
 'c_syllables': {1: 5, 2: 1, 3: 1, 4: 2},
 'n_sents': 1,
 'n_words': 9,
 'n_unique_words': 8,
 'n_long_words': 3,
 'n_complex_words': 2,
 'n_simple_words': 7,
 'n_monosyllable_words': 5,
 'n_polysyllable_words': 4,
 'n_chars': 55,
 'n_letters': 45,
 'n_spaces': 8,
 'n_syllables': 18,
 'n_punctuations': 2,
 'letters_entropy': 2.197159723424149,
 'syllables_entropy': 1.6577427265048887,
 'c_punctuations': {'comma': 1,
                    'period': 0,
                    'question': 0,
                    'exclamation': 0,
                    'ellipsis': 0,
                    'colon': 1,
                    'semicolon': 0,
                    'dash': 0,
                    'hyphen': 0,
                    'angle_quotes': 0,
                    'straight_quotes': 0,
                    'parentheses': 0,
                    'other': 0},
 'p_unique_words': 0.8888888888888888,
 'p_long_words': 0.3333333333333333,
 'p_complex_words': 0.2222222222222222,
 'p_simple_words': 0.7777777777777778,
 'p_monosyllable_words': 0.5555555555555556,
 'p_polysyllable_words': 0.4444444444444444,
 'p_letters': 0.8181818181818182,
 'p_spaces': 0.14545454545454545,
 'p_punctuations': 0.03636363636363636}

The describe method is that of the statistics interface of the anyTS core; the basic statistics have no scales, and it gives None for the statistics of get_stats.

Punctuation marks¶

count_punctuations(text, marks=PUNCTUATION_MARKS, dash_pattern=DASH_PATTERN)

Counts punctuation marks by type, the distribution of c_punctuations: commas, periods, question and exclamation marks, ellipses, colons, semicolons, dashes, hyphens, guillemets, quotation marks, parentheses and other marks. Dashes typed with hyphens (dash_pattern) are counted first, then ellipses - …, three or more periods, or two after ? and !, one mark whose periods are not counted as periods (Who?.. is a question and an ellipsis) - then each mark by its type in marks; any other character of the Unicode categories P and S is another mark (§, €, °).

Parameter	Type	Default	Description
`text`	`str`	-	Text string
`marks`	`dict[str, str]`	`PUNCTUATION_MARKS`	Type of every mark, among `PUNCTUATION_TYPES`
`dash_pattern`	`Pattern`	`DASH_PATTERN`	Dashes typed with hyphens

A mark of marks that is not a single character or has an unknown type raises ParameterError; a text that is not a string, marks that are not a mapping and a dash_pattern that is not a compiled regular expression raise SourceTypeError.

In ruTS the PUNCTUATION_MARKS and DASH_PATTERN of ruts.basic_stats are the marks of the core with the low quotation mark „ as a quotation mark and the pattern of dashes by which a hanging hyphen before и, или, либо or a comma stays a hyphen (двух- и трёхкомнатные, сорока-, пятидесятилетние), while a dialogue dash typed with a hyphen is a dash (- Ушли, - сказал он).

CodeResult

from ruts.basic_stats import count_punctuations

text = "Кот — «зверь»... Пёс, конечно, - друг; а кто-то (тот, что ещё жив) — нет!"
{kind: count for kind, count in count_punctuations(text).items() if count}

{'comma': 3,
 'exclamation': 1,
 'ellipsis': 1,
 'semicolon': 1,
 'dash': 3,
 'hyphen': 1,
 'angle_quotes': 2,
 'parentheses': 2}

Dashes typed with hyphens¶

dash_pattern(conjunctions=(), hanging_before_comma=False)

Builds the dash_pattern of count_punctuations. A hanging hyphen glued to a letter or a digit and carried on to a later word (pre- and post-war, 5- and 10-year) stays a hyphen before spaces of a line and one of the conjunctions, case ignored, and with hanging_before_comma before a comma (two-, three- and four-year). A language whose dialogue closes a line with a hyphen before a comma (-Come -he said-, and left) leaves hanging_before_comma off. Without the parameters the pattern is DASH_PATTERN.

Parameter	Type	Default	Description
`conjunctions`	`Collection[str]`	`()`	Conjunctions after a hanging hyphen
`hanging_before_comma`	`bool`	`False`	Take a hyphen between a letter and a comma for a hanging one

Conjunctions that are not a collection of strings raise SourceTypeError, and an empty conjunction ParameterError.

Without the parameters the pattern is the DASH_PATTERN of the core (anyts.basic_stats), while the DASH_PATTERN of ruts.basic_stats is dash_pattern(("и", "или", "либо"), hanging_before_comma=True).

CodeResult

from ruts.basic_stats import count_punctuations, dash_pattern

text = "Двух- и трёхкомнатные, сорока-, пятидесятилетние - сказал он"
counts = count_punctuations(text)
counts["dash"], counts["hyphen"]
counts = count_punctuations(text, dash_pattern=dash_pattern())
counts["dash"], counts["hyphen"]

(1, 2)
(3, 0)

Punctuation profile¶

punctuation_profile(text, n_words=None)

The frequencies of the punctuation marks by type (count_punctuations) per 1000 words and yo_share - the share of the letter ё among the letters е and ё, that is, whether the author writes ё. Without words or without the letters е and ё the values are nan. A text that is not a string raises SourceTypeError, a number of words that is not a non-negative integer ParameterError.

Parameter	Type	Default	Description
`text`	`str`	-	Text string
`n_words`	`int`	`None`	Number of words; without it the words are extracted by `WordsExtractor`

The profile is an editorial and stylometric feature: Chekhov has several times more ellipses and exclamations than Tolstoy (see corpus comparison). It depends on text formatting - quotation marks, dashes, the letter ё - and is best read separately from linguistic features.

CodeResult

from ruts.basic_stats import punctuation_profile

text = "Кот — «зверь»... Пёс, конечно, - друг; а кто-то (тот, что ещё жив) — нет!"
profile = punctuation_profile(text)
round(profile["dash"], 1), round(profile["yo_share"], 2)

(250.0, 0.33)