Skip to content

Spanish Texts Statistics (esTS)

esTS

¿Cómo esTáS, texto?

esTS computes statistics of Spanish texts: basic statistics, readability, lexical diversity, lexical sophistication, style, phonostatistics, morphology, syntax and cohesion - by published formulas with the coefficients and the scales of their authors, and by the parts of speech, the features and the dependencies of Universal Dependencies.

The library works both with raw strings and with Doc objects of spaCy; most statistics need no trained model (Installation).

Try it without installing in the demo on Hugging Face Spaces: paste a text and get its readability, the metrics, the plots and the highlighting of its fragments.

Features

  • Extraction

    Tokenizers of sentences, words and character N-grams that know the inverted marks, the dialogue dash and the abbreviations of Spanish, and the paragraphs of a text

    Extraction

  • Syllables and stress

    Division of a word into syllables and its stress by the orthographic rules, with no dictionary

    Syllables and stress

  • Basic statistics

    Numbers of sentences, words, letters, syllables and punctuation marks by type, their distributions and the entropies of word lengths

    Basic statistics

  • Readability

    Fernández Huerta, Szigriszt-Pazos with the INFLESZ scale, Gutiérrez de Polini, Crawford, Legibilidad µ, SOL, LIX and RIX with a consensus grade, the school stages of Spain and reading time, and formulas of other languages with their own coefficients

    Readability metrics

  • Lexical diversity

    Type-Token Ratio and its variations, MATTR, MSTTR, Measure of Textual Lexical Diversity and its moving average, HD-D, the indices of Simpson and Yule, entropy, Orlov's Z, the burstiness of the lexemes, the laws of Zipf and Heaps, over the whole text or over windows with confidence intervals

    Lexical diversity metrics

  • Morphology

    Parts of speech and fifteen grammatical features of Universal Dependencies with the markers of Spanish: the moods, the non-finite forms, the copulas ser and estar, the adverbs in -mente

    Morphological statistics

  • Syntax

    Statistics of the dependency tree (dependency distances, depth, clauses, coordination, the production units of L2SCA, prepositional chains, the order of the subject and the object, the similarity of adjacent sentences) with the constructions of the administrative style: the passive with ser and with se, the participial and the gerund clauses, the chains of de, the split predicates

    Syntactic statistics

  • Cohesion

    Overlap of nouns, arguments and content words between sentences in the manner of Coh-Metrix and with the next sentences and paragraphs in the manner of TAACO, givenness, temporal cohesion and the density of 273 Spanish discourse markers by class and by place, between and within sentences

    Cohesion statistics

  • Lexical sophistication

    How rare the words of a text are, in the manner of TAALES: frequency, range, dispersion and surprisal by a frequency dictionary of Google Books Ngram, the frequency bands top-1000 to 10000, lexical density

    Lexical sophistication statistics

  • Style

    SEO indicators of Advego and Text.ru (nausea, water content, spam score, naturalness by Zipf's law, keyword density) and the markers of the officialese by the Spanish-language guides to plain language: verbal nouns, compound prepositions, parenthetical expressions and clichés

    Style metrics

  • Phonetics

    Shares of the classes of sounds of a rule-based transcription, consonant clusters, hiatuses, open syllables, hardness and the indices of alliteration and assonance

    Phonostatistics

  • Verse

    Metrical syllables and the meter of a poem, the stress profile and the types of the endecasílabo, the rhyme in full and by assonance with its scheme, the strophes and the form of a poem

    Verse statistics

  • Corpus measures

    Keywords against a reference corpus or the frequency dictionary, collocations, the dispersion of a word over the parts of a text, a KWIC concordance, authorship by stylometry (Burrows's Delta and rolling Delta, Zeta and its markers in segments, the Mendenhall curve, the function words) and the comparison of corpora over 163 features of a text

    Corpus measures

  • Datasets

    A corpus of Spanish-language literature in the public domain (150 works by 33 authors in four genres), a collection of sonnets of the 15th-20th centuries with the metrical pattern and the rhyme of every line and a frequency dictionary of 83,785 lemmas by Google Books Ngram

    Datasets

  • Visualization

    Zipf's law, literature fingerprinting, word tree, corpus and stylometric plots, vocabulary growth and frequency spectrum, sentence lengths and the highlighting of a text by the fragments the statistics count

    Visualizers

  • Components

    Components of a spaCy pipeline, so that a text is annotated and measured in one pass and the statistics travel with the Doc

    spaCy components

  • Exceptions

    Errors of one hierarchy whose classes are also built-in Python exceptions

    Exceptions and logging

Installation

Requires Python 3.11 or newer.

pip install pyests

The distribution on PyPI is pyests, the package it installs is ests. The dependencies, the spaCy model and the datasets are on the Installation page.

Quick start

from ests import BasicStats, DiversityStats, ReadabilityStats

text = "Hay tres clases de mentiras: mentiras, malditas mentiras y estadísticas"

BasicStats(text).get_stats()
ReadabilityStats(text).flesch_reading_easy
DiversityStats(text).ttr

{'c_letters': {1: 1, 2: 1, 3: 1, 4: 1, 6: 1, 8: 4, 12: 1},
 'c_syllables': {1: 4, 2: 1, 3: 4, 5: 1},
 'n_sents': 1,
 'n_words': 10,
 'n_unique_words': 8,
 'n_long_words': 5,
 'n_complex_words': 5,
 'n_simple_words': 5,
 'n_monosyllable_words': 4,
 'n_polysyllable_words': 6,
 'n_chars': 71,
 'n_letters': 60,
 'n_spaces': 9,
 'n_syllables': 23,
 'n_punctuations': 2,
 'letters_entropy': 2.5219280948873624,
 'syllables_entropy': 1.7219280948873623,
 'c_punctuations': {'comma': 1,
                    'period': 0,
                    'question': 0,
                    'exclamation': 0,
                    'ellipsis': 0,
                    'colon': 1,
                    'semicolon': 0,
                    'dash': 0,
                    'hyphen': 0,
                    'angle_quotes': 0,
                    'straight_quotes': 0,
                    'parentheses': 0,
                    'other': 0}}
53.545000000000016
0.8

Any statistic can be printed in a readable form:

BasicStats(text).print_stats()

                   Statistic                    |  Value
-----------------------------------------------------------
Sentences                                       |    1
Words                                           |    10
Unique words                                    |    8
Long words                                      |    5
Complex words                                   |    5
Simple words                                    |    5
Monosyllabic words                              |    4
Polysyllabic words                              |    6
Characters                                      |    71
Letters                                         |    60
Spaces                                          |    9
Syllables                                       |    23
Punctuation marks                               |    2
Entropy of the word lengths in letters (bits)   |   2.52
Entropy of the word lengths in syllables (bits) |   1.72

A walkthrough of one short story with every tool of the library is in the notebook examples/01_text_walkthrough.ipynb, which opens in Colab; the notebooks on prose authors, sonnets and corpus measures are on the examples page.

Project structure
  • demo - the demo on Hugging Face Spaces (Gradio)
  • docs - project documentation
  • ests:
    • basic_stats.py - basic text statistics
    • cohesion_stats.py - cohesion statistics
    • components.py - spaCy components
    • constants.py - constants of the Spanish language and of the metrics
    • diversity_stats.py - lexical diversity metrics
    • exceptions.py - library exceptions
    • extractors.py - extractors of sentences, words and character N-grams with the Spanish tokenizers, the paragraphs of a text
    • lexical_stats.py - lexical sophistication statistics
    • morph_stats.py - morphological statistics
    • phon_stats.py - phonostatistics
    • readability_stats.py - readability metrics
    • unadapted_readability_stats.py - readability formulas of other languages
    • style_stats.py - style metrics
    • syllables.py - syllabification and stress
    • syntax_stats.py - syntactic statistics
    • utils.py - helper tools
    • verse_stats.py - verse statistics
    • corpus - corpus measures:
      • collocations.py - collocations and association measures
      • compare.py - comparison of corpora by the features of a text
      • dispersion.py - word dispersion over the parts of a text
      • keyness.py - keywords against a reference corpus or the frequency dictionary
      • kwic.py - KWIC concordance
      • stylometry.py - Burrows's Delta, Zeta, the function words and other stylometry measures
    • datasets - datasets:
      • freq_dict.py - the frequency dictionary by Google Books Ngram
      • spanish_literature.py - the corpus of Spanish-language literature
      • spanish_sonnets.py - the sonnets of the 15th-20th centuries
    • resources - embedded lexical resources (the discourse markers, the 10,000 most frequent lemmas)
    • visualizers - plots:
      • corpus.py - lexical dispersion, keyness chart, collocation network
      • fingerprinting.py - literature fingerprinting
      • highlight.py - text highlighting
      • sentences.py - sentence lengths
      • stylometry.py - dendrogram, principal components, scaling, distances within and between authors, rolling Delta, Zeta markers, Mendenhall curves
      • vocabulary.py - vocabulary growth and frequency spectrum
      • word_tree.py - word tree
      • zipf.py - Zipf's law
  • examples - notebooks with examples
  • scripts - scripts that build the archives of the datasets, fetch the anyTS pages of the documentation and check the outputs of the notebooks
  • tests - tests mirroring the package structure