Basic statistics¶

BasicStats(source, sents_extractor=None, words_extractor=None, normalize=False, complex_syl_factor=3, long_word_letter_factor=7)

The basic statistics of a text: the counts of its sentences, words, characters, syllables and punctuation marks and the distributions of its words. The data source is a text or a Doc object of spaCy. Letters are counted by str.isalpha, so digits, hyphens and marks inside a word are not letters.

The words and the sentences of a string come from the extractors. A Doc gives its tokens as words, without punctuation marks and symbols such as € or %, and its boundaries as sentences, or those of the sentence extractor when it has none or only partial ones (has_sents); an extractor given is used on its text instead. A blank line ends a sentence of a string and of a Doc without boundaries; a Doc with boundaries does so when its pipeline has start_sents_after_blank_lines (registered by a language library under its own name) before the parser, senter or sentencizer. A sentence without words, such as punctuation alone, is not counted.

Raises:

SourceTypeError - a source that is neither a string nor a Doc, an extractor of another type;
SourceError - a source without words;
ParameterError - a threshold that is not an integer of at least one, a normalize that is not a bool.

The class extends the BasicStats of the anyTS core with the hooks of Spanish: the syllables of a word are counted by count_syllables, and the default extractors are the Spanish SentsExtractor and WordsExtractor. The ordinal indicators º and ª are letters (3.º is a one-letter word). The blank line component of esTS is called ests_blank_lines: nlp.add_pipe("ests_blank_lines", before="parser"). The statistics are computed when the object is created. A string with a character without a UTF-8 form, as a file read with the wrong encoding gives, raises SourceError from the Spanish tokenizer.

CodeResult

from ests import BasicStats

text = "Hay tres clases de mentiras: mentiras, malditas mentiras y estadísticas"
bs = BasicStats(text, normalize=True)
bs.print_stats()

                   Statistic                    |  Value
-----------------------------------------------------------
Sentences                                       |    1
Words                                           |    10
Unique words                                    |    8
Long words                                      |    5
Complex words                                   |    5
Simple words                                    |    5
Monosyllabic words                              |    4
Polysyllabic words                              |    6
Characters                                      |    71
Letters                                         |    60
Spaces                                          |    9
Syllables                                       |    23
Punctuation marks                               |    2
Entropy of the word lengths in letters (bits)   |   2.52
Entropy of the word lengths in syllables (bits) |   1.72

Parameters¶

Parameter	Type	Default	Description
`source`	`str` / `Doc`	-	Data source (a string or a Doc object)
`sents_extractor`	`SentsExtractor`	`None`	Sentence extraction tool; used for a Doc too, on its text
`words_extractor`	`WordsExtractor`	`None`	Word extraction tool; used for a Doc too, on its text
`normalize`	`bool`	`False`	Compute normalized statistics
`complex_syl_factor`	`int`	`COMPLEX_SYL_FACTOR`	Minimum number of syllables in a complex word
`long_word_letter_factor`	`int`	`LONG_WORD_LETTER_FACTOR`	Minimum number of letters in a long word

The object keeps normalize, complex_syl_factor and long_word_letter_factor under their names.

In esTS the thresholds are those of the core and of the readability formulas: a complex word has three or more syllables, as in SMOG, and a long word seven or more letters, as in LIX and RIX (COMPLEX_SYL_FACTOR = 3 and LONG_WORD_LETTER_FACTOR = 7 of anyts.constants).

Statistics¶

Statistic	Type	Description	Normalized
`c_letters`	`dict[int, int]`	Distribution of words by number of letters	-
`c_syllables`	`dict[int, int]`	Distribution of words by number of syllables	-
`n_sents`	`int`	Number of sentences containing words	-
`n_words`	`int`	Number of words	-
`n_unique_words`	`int`	Number of unique words, case ignored	`p_unique_words`
`n_long_words`	`int`	Number of long words, of at least `long_word_letter_factor` letters	`p_long_words`
`n_complex_words`	`int`	Number of complex words, of at least `complex_syl_factor` syllables	`p_complex_words`
`n_simple_words`	`int`	Number of simple words: with a syllable, below the complex ones	`p_simple_words`
`n_monosyllable_words`	`int`	Number of monosyllabic words	`p_monosyllable_words`
`n_polysyllable_words`	`int`	Number of polysyllabic words, of two syllables or more	`p_polysyllable_words`
`n_chars`	`int`	Number of characters without the line breaks	-
`n_letters`	`int`	Number of letters	`p_letters`
`n_spaces`	`int`	Number of spaces of Unicode (category Zs), such as the no-break space, and tabs	`p_spaces`
`n_syllables`	`int`	Number of syllables	-
`n_punctuations`	`int`	Number of punctuation marks	`p_punctuations`
`letters_entropy`	`float`	Shannon entropy in bits of the word lengths in letters, of `c_letters`: 0 when the words have one length; the estimate by the shares runs lower on a short text, by tenths of a bit on tens of words	-
`syllables_entropy`	`float`	Shannon entropy in bits of the word lengths in syllables, of `c_syllables`, with the same bias	-
`c_punctuations`	`dict[str, int]`	Distribution of punctuation marks by type (`count_punctuations`)	-

With normalize=True the object has the normalized statistics too, of the type float: the counts of the words divided by the number of words, those of the characters by the number of characters.

Methods¶

count_words_by_syllables(min_syllables) and count_words_by_letters(min_letters) return the number of words with at least the given number of syllables or letters; a minimum that is not an integer raises ParameterError.

The methods recount complex and long words with a threshold other than the one set when the object was created; ReadabilityStats counts the polysyllabic words of SOL and the long words of LIX and RIX by them, so these do not depend on the thresholds of the object.

get_stats(flat) returns a dictionary with the distributions c_letters and c_syllables, the statistics of stats_desc, the distribution c_punctuations and, with normalize=True, the normalized counts, as a copy: editing it does not change the object. With flat=True the distributions give a key for every value the text has, such as c_letters_3 and c_punctuations_comma.

print_stats() prints a table with the statistics of stats_desc, their descriptions and the headers from stats_headers.

The normalized statistics p_* are not in the table.

to_dataframe() returns the flat statistics, get_stats(flat=True), as a one-row DataFrame of pandas with a column for every key. A dictionary statistic gives columns only for the keys the text has, so in frames of several texts joined with pandas.concat a count a text lacks is NaN. Only these columns may be filled with 0: elsewhere NaN is an undefined value.

CodeResult

bs.count_words_by_syllables(4), bs.count_words_by_letters(8)
bs.get_stats()

(1, 5)
{'c_letters': {1: 1, 2: 1, 3: 1, 4: 1, 6: 1, 8: 4, 12: 1},
 'c_syllables': {1: 4, 2: 1, 3: 4, 5: 1},
 'n_sents': 1,
 'n_words': 10,
 'n_unique_words': 8,
 'n_long_words': 5,
 'n_complex_words': 5,
 'n_simple_words': 5,
 'n_monosyllable_words': 4,
 'n_polysyllable_words': 6,
 'n_chars': 71,
 'n_letters': 60,
 'n_spaces': 9,
 'n_syllables': 23,
 'n_punctuations': 2,
 'letters_entropy': 2.5219280948873624,
 'syllables_entropy': 1.7219280948873623,
 'c_punctuations': {'comma': 1,
                    'period': 0,
                    'question': 0,
                    'exclamation': 0,
                    'ellipsis': 0,
                    'colon': 1,
                    'semicolon': 0,
                    'dash': 0,
                    'hyphen': 0,
                    'angle_quotes': 0,
                    'straight_quotes': 0,
                    'parentheses': 0,
                    'other': 0},
 'p_unique_words': 0.8,
 'p_long_words': 0.5,
 'p_complex_words': 0.5,
 'p_simple_words': 0.5,
 'p_monosyllable_words': 0.4,
 'p_polysyllable_words': 0.6,
 'p_letters': 0.8450704225352113,
 'p_spaces': 0.1267605633802817,
 'p_punctuations': 0.028169014084507043}

The describe method is that of the statistics interface of the anyTS core; the basic statistics have no scales, so it gives None for the statistics of get_stats.

Punctuation marks¶

count_punctuations(text, marks=PUNCTUATION_MARKS, dash_pattern=DASH_PATTERN)

Counts punctuation marks by type, the distribution of c_punctuations: commas, periods, question and exclamation marks, ellipses, colons, semicolons, dashes, hyphens, guillemets, quotation marks, parentheses and other marks. Dashes typed with hyphens (dash_pattern) are counted first, then ellipses - …, three or more periods, or two after ? and !, one mark whose periods are not counted as periods (Who?.. is a question and an ellipsis) - then each mark by its type in marks; any other character of the Unicode categories P and S is another mark (§, €, °).

Parameter	Type	Default	Description
`text`	`str`	-	Text string
`marks`	`dict[str, str]`	`PUNCTUATION_MARKS`	Type of every mark, among `PUNCTUATION_TYPES`
`dash_pattern`	`Pattern`	`DASH_PATTERN`	Dashes typed with hyphens

A mark of marks that is not a single character or has an unknown type raises ParameterError; a text that is not a string, marks that are not a mapping and a dash_pattern that is not a compiled regular expression raise SourceTypeError.

esTS counts with the marks and the dash pattern of the core, PUNCTUATION_MARKS and DASH_PATTERN of anyts.basic_stats, which follow Spanish orthography: the inverted ¿ and ¡ are question and exclamation marks (¿Qué? carries two question marks), —, – and the horizontal bar ― are dashes, and so is the raya typed with hyphens - a run of two or more hyphens, a hyphen after whitespace, at the start of a line or after a closing mark, before a space or between a letter and an opening or a closing mark (--Hola --dijo Juan, - Se fueron - dijo, cuatro.-¿Cinco?); a hyphen inside a word, before a digit or at the end of a line inside a word is a hyphen (teórico-práctico, -5), «» are guillemets and "“”‘’ the quotation marks of the three levels of the orthography. The combining marks and the invisible format characters (Unicode M and Cf) are removed from the words but are not counted as marks.

CodeResult

from ests.basic_stats import count_punctuations

text = "El gato — «fiera»... El perro, claro, - amigo; y alguien (el que vive) — ¡no!"
{kind: count for kind, count in count_punctuations(text).items() if count}

{'comma': 2,
 'exclamation': 2,
 'ellipsis': 1,
 'semicolon': 1,
 'dash': 3,
 'angle_quotes': 2,
 'parentheses': 2}

Punctuation profile¶

punctuation_profile(text, n_words=None)

The frequencies of the punctuation marks by type (count_punctuations) per 1000 words and inverted_share - the share of the inverted marks among all question and exclamation marks: 0.5 when every question and exclamation opens with ¿ or ¡ as the orthography requires, lower when the writer drops them. Without words the frequencies are nan, and so is inverted_share without question and exclamation marks.

Parameter	Type	Default	Description
`text`	`str`	-	Text string
`n_words`	`int`	`None`	Number of words; without it the words are extracted by `WordsExtractor`

Raises:

SourceTypeError - a text that is not a string;
SourceError - without n_words, a text with a character without a UTF-8 form;
ParameterError - a number of words that is not a non-negative integer.

The profile depends on text formatting (typographic quotation marks and dashes, the inverted marks), so it is best read separately from linguistic features.

CodeResult

from ests.basic_stats import punctuation_profile

profile = punctuation_profile(text)
round(profile["dash"], 1), profile["inverted_share"]

(230.8, 0.5)