Russian Texts Statistics (ruTS)¶

ruTS

ruTS computes for Russian texts what usually takes several different tools: from basic statistics, readability and lexical diversity to morphology, syntax, cohesion, stylometry and corpus measures - by published formulas adapted to the Russian language.

The source is a string or a spaCy Doc; every statistic is available both as a class and as a spaCy pipeline component.

Try it without installing in the demo on Hugging Face Spaces: the readability grade, metrics, plots and highlighted fragments of a pasted text.

Features¶

Extraction

Tokenizers of words, sentences and character N-grams with filters, lemmatization and N-grams

Extraction
Syllables and stress

Syllabification of words, the number of syllables, the stressed syllable by the dictionary and the type of the ending

Syllables and stress
Basic statistics

Numbers of words, sentences, syllables, punctuation marks and their distributions

Basic statistics
Readability

Flesch-Kincaid test, SMOG index, LIX and other metrics with coefficients for Russian, formulas of other languages

Readability metrics
Lexical diversity

Type-Token Ratio, Measure of Textual Lexical Diversity, hapax index and other metrics

Lexical diversity metrics
Morphology

Morphological features in Universal Dependencies terms: part of speech, case, mood, transitivity and others

Morphological statistics
Style

SEO metrics: nausea, water content, spam score, naturalness by Zipf's law, keyword density

SEO style metrics
Phonetics

Shares of sound classes, consonant clusters, alliteration and assonance, syllables

Phonostatistics
Syntax

Statistics of the spaCy dependency tree: dependency distances, tree depth, clauses, participial clauses, passive voice, genitive chains, word order

Syntactic statistics
Cohesion

Word overlap between sentences and paragraphs, givenness, temporal cohesion, connectives

Cohesion statistics
Lexical sophistication

Word frequency by the Lyashevskaya-Sharoff dictionary, frequency bands, surprisal, lexical density

Lexical sophistication statistics
Verse

Stresses by the Koziev dictionary, meter, stress profile, rhyme schemes and stanzas

Verse statistics
Corpus measures

Keywords, collocations, word dispersion, KWIC concordance, stylometry measures (Burrows's Delta, Zeta, Kilgarriff's chi-square) and the comparison of corpora across all features at once

Corpus measures
Datasets

Soviet literature readers, Collected works of Stalin, Texts with grade labels, Lyashevskaya and Sharoff frequency dictionary, Russian poetry corpus, Russian classical literature, Koziev stress dictionary

Datasets
Visualization

Zipf's law, literature fingerprinting, word tree, text highlighting, corpus and stylometric plots, vocabulary growth and frequency spectrum, sentence lengths

Visualizations
Components

Components that put the statistics into spaCy pipelines

Components
Exceptions

One error hierarchy compatible with built-in Python exceptions

Exceptions and logging

Installation¶

Requires Python 3.11 or newer.

pip install ruts

More on dependencies and installing from the repository - on the Installation page.

Quick start¶

CodeResult

from ruts import BasicStats, DiversityStats, ReadabilityStats

text = "Существуют три вида лжи: ложь, наглая ложь и статистика"

BasicStats(text).get_stats()
ReadabilityStats(text).flesch_reading_easy
DiversityStats(text).ttr

{'c_letters': {1: 1, 3: 2, 4: 3, 6: 1, 10: 2},
 'c_syllables': {1: 5, 2: 1, 3: 1, 4: 2},
 'n_sents': 1,
 'n_words': 9,
 'n_unique_words': 8,
 'n_long_words': 3,
 'n_complex_words': 2,
 'n_simple_words': 7,
 'n_monosyllable_words': 5,
 'n_polysyllable_words': 4,
 'n_chars': 55,
 'n_letters': 45,
 'n_spaces': 8,
 'n_syllables': 18,
 'n_punctuations': 2,
 'letters_entropy': 2.197159723424149,
 'syllables_entropy': 1.6577427265048887,
 'c_punctuations': {'comma': 1,
                    'period': 0,
                    'question': 0,
                    'exclamation': 0,
                    'ellipsis': 0,
                    'colon': 1,
                    'semicolon': 0,
                    'dash': 0,
                    'hyphen': 0,
                    'angle_quotes': 0,
                    'straight_quotes': 0,
                    'parentheses': 0,
                    'other': 0}}
74.93500000000003
0.8888888888888888

Any statistic can be printed in a readable form:

CodeResult

BasicStats(text).print_stats()

            Статистика            | Значение
---------------------------------------------
Предложения                       |    1
Слова                             |    9
Уникальные слова                  |    8
Длинные слова                     |    3
Сложные слова                     |    2
Простые слова                     |    7
Односложные слова                 |    5
Многосложные слова                |    4
Символы                           |    55
Буквы                             |    45
Пробелы                           |    8
Слоги                             |    18
Знаки препинания                  |    2
Энтропия длин слов в буквах (бит) |   2.20
Энтропия длин слов в слогах (бит) |   1.66

A walkthrough of one short story with every tool of the library is in the notebook examples/01_text_walkthrough.ipynb (opens in Colab), the other notebooks are on the examples page.