Skip to content

Any Texts Statistics (anyTS)

anyTS is the language-independent core of the text statistics libraries ruTS (Russian) and esTS (Spanish). It holds the code that does not depend on a language, so that it is written, tested and documented once; a language library adds its tokenizer, lemmatizer, spaCy model and rules and re-exports the core under its own names.

Features

  • Document model

    Read a string, a spaCy Doc or a Document of anySeg as one document of tokens, words, sentences and paragraphs; a language library segments and annotates a string in its language

    Document model

  • Extraction

    Extract sentences, words and character N-grams with extractors whose tokenizer, lemmatizer and number pattern are hooks a language library overrides

    Extraction

  • Statistics interface

    Give the statistics of every class through one interface: a dictionary, nested or flat, a pandas frame, a printed table and the reading of a statistic by its scale

    Statistics interface

  • Basic statistics

    Count sentences, words, syllables, letters and punctuation marks by type, with the syllables of a language library

    Basic statistics

  • Readability

    Compute Flesch reading ease, Flesch-Kincaid, Coleman-Liau, ARI, SMOG, Gunning fog, LIX, RIX, Legibilidad µ, the consensus grade and the reading time, with the coefficients of a language library

    Readability metrics · Unadapted formulas

  • Lexical diversity

    Compute Type-Token Ratio and its variants, MATTR, MSTTR, MTLD, MA-MTLD, MTLD-W, HD-D, the indices of Simpson and Yule, entropy, the laws of Zipf and Heaps, over the whole text or over windows with confidence intervals

    Lexical diversity metrics

  • Lexical sophistication

    Measure a text by a frequency dictionary in the manner of TAALES: coverage, mean frequency, range and dispersion, surprisal and perplexity, frequency bands and lexical density

    Lexical sophistication

  • Style

    Compute the SEO metrics (nausea, water, spam, naturalness by Zipf's law, keyword density) and the markers of the officialese, with the lists and the morphology of a language library

    Style metrics

  • Cohesion

    Measure the cohesion between sentences in the manner of Coh-Metrix - the overlap of nouns, arguments and content words, adjacent and over all pairs, Dice, with the next two sentences and between paragraphs in the manner of TAACO, givenness, the repetition of the tense and the aspect - and find the connectors of a dictionary with the context rules of a language, with the words and the features of a language library

    Cohesion statistics

  • Syntax

    Compute the statistics of the dependency tree - dependency distances, tree depth, valency, coordination chains, clauses, modifiers of a noun, prepositional chains, participial clauses, the passive, negations, split predicates, the words before the root, the order of the subjects and the objects, the similarity of adjacent sentences and the T-units and phrases of L2SCA - with the clauses and the constructions of a language library, on tree helpers that read the labels shared by Universal Dependencies and ClearNLP and the verb form of a predicate by its auxiliaries

    Syntactic statistics

  • Morphology

    Count the parts of speech and the features of Universal Dependencies by value or word by word, with the descriptions of the values, the markers and the analysis of a language library

    Morphological statistics

  • Phonetics

    Compute classes of the sounds, consonant clusters, hiatuses, entropy of the CV patterns, alliteration and assonance by the repetition index, open syllables, with the transcription and the syllables of a language library

    Phonetic statistics

  • Corpus measures

    Compare corpora by keywords against a corpus or a frequency dictionary, collocations, the dispersion of a word, stylometry (Burrows's Delta and its variants, rolling Delta, Zeta, Kilgarriff's chi-square, the Mendenhall curve), the comparison of corpora feature by feature with effect sizes, a bootstrap by texts and the Holm correction, and a KWIC concordance

    Corpus measures

  • Visualization

    Plot Zipf's law, vocabulary growth, sentence lengths, literature fingerprinting, a word tree, corpus and stylometric measures, with the labels of the plots as a parameter, and highlight a text by the layers of a language library

    Visualization

  • Components

    Build components of spaCy pipelines that put the statistics into an extension of Doc

    Components

  • Datasets

    Download an archive with a checksum, extract it safely and filter the records

    Datasets

  • Exceptions

    Catch errors of one hierarchy whose classes are also built-in exceptions

    Exceptions

The reference of every function is a named section that the libraries include in their own documentation, so a formula is described in one place.

Installation

Requires Python 3.11 or newer.

pip install anyts

The dependencies are on the Installation page.

Quick start

from anyts import DiversityStats, SentsExtractor, WordsExtractor

text = "The cat sat on the mat. The dog sat on the log! The cat and the dog slept."
SentsExtractor().extract(text)

ds = DiversityStats(WordsExtractor(lowercase=True).extract(text))
ds.ttr, round(ds.entropy, 3)

('The cat sat on the mat.', 'The dog sat on the log!', 'The cat and the dog slept.')
(0.5, 2.864)

A language library subclasses the extractors and overrides their hooks:

class Words(WordsExtractor):
    def lemmatize(self, word):
        return {"sat": "sit", "slept": "sleep"}.get(word.lower(), word)


Words(lowercase=True, use_lexemes=True).extract("The cat sat. The dog slept.")

('the', 'cat', 'sit', 'the', 'dog', 'sleep')

The corpus measures take lists of words:

from anyts.corpus import keyness

cats = WordsExtractor(lowercase=True).extract(
    "The cat sat on the mat. The cat saw a bird. The cat slept."
)
dogs = WordsExtractor(lowercase=True).extract(
    "The dog sat on the log. The dog saw a cat. The dog barked."
)
[(keyword.word, round(keyword.log_ratio, 2)) for keyword in keyness(cats, dogs, min_freq=2)]

[('cat', 1.58)]