Components¶

StatsComponent(nlp, name)

The base of the components of spaCy that put the statistics of a text into an extension of Doc: a component registers the extension when added to a pipeline and puts the statistics of every document into doc._.<name>. Writing components in general is described in the documentation of spaCy.

The components of esTS are subclasses of the StatsComponent of the anyTS core, one for every class of statistics.

Note

The examples below use the model es_core_news_sm, which is installed apart: python -m spacy download es_core_news_sm (see installation).

CodeResult

import ests
import spacy

nlp = spacy.load("es_core_news_sm")
nlp.add_pipe("ests_basic", name="basic")
doc = nlp("El gato duerme en la ventana. Los niños juegan en el parque.")
doc._.basic.n_words, doc._.basic.c_letters
nlp("?!")._.basic is None

(12, {2: 5, 3: 1, 4: 1, 5: 1, 6: 3, 7: 1})
True

Parameters¶

Parameter	Type	Default	Description
`nlp`	`Language`	-	Pipeline the component is added to
`name`	`str`	-	Name of the component in the pipeline and of the extension

spaCy sets both at nlp.add_pipe(factory, name=...); the other parameters of a component are passed in the config dictionary of the same call.

Names¶

The factories carry the prefix of the library: ests_basic, ests_readability, ests_unadapted_readability, ests_diversity, ests_morph, ests_syntax, ests_cohesion, ests_lexical, ests_style, ests_phon, ests_verse, and the component of sentence boundaries ests_blank_lines. They are declared as entry points of spacy_factories, so a pipeline saved with these components (nlp.to_disk(path), spacy package) loads with spacy.load(path) without importing the library. import ests does not load the components while spaCy is not loaded - the entry points find them, and ests.StyleStatsComponent loads them on first use; from the sources without an installed package, import ests.components comes before add_pipe.

The name of the pipe is the name of the extension: nlp.add_pipe(factory, name="basic") puts the statistics into doc._.basic. Without name both take the name of the factory. The extension follows the pipe when the pipeline renames it (nlp.rename_pipe). A name taken by an extension of another package raises ParameterError instead of replacing it: at add_pipe, or at the first document after rename_pipe, which renames the pipe before the component learns its name. A component taken from another pipeline (add_pipe(name, source=other)) is the same object in both, so under a new name it writes to that extension in the other pipeline too; a separate component comes from the factory, add_pipe(factory, name=...). The same component can be added twice under different names, with different parameters. A document with no words - an empty string, whitespace, punctuation alone - passes through a component untouched, its extension left at None.

In ests_lexical a document of numbers alone passes untouched as well, and in ests_verse a document without letters.

Serialization

A component keeps an object of statistics in doc._.<name>, and spaCy cannot serialize it: Doc.to_bytes(), DocBin(store_user_data=True) and nlp.pipe(..., n_process>1) fail with these components in the pipeline. To save a document, leave the user data out (doc.to_bytes(exclude=["user_data"])) or keep doc._.<name>.get_stats() separately; for multiprocessing compute the statistics in the main process after nlp.pipe, without the components.

Adding a component extends the tokenizer of its pipeline with the rules for the dashes of a dialogue glued to the words and for the abbreviations in every case (add_dash_rules), as in the string API: sí--dijo is two words and a dash, SR. one word. A tokenizer that is not the Tokenizer of spaCy is left as it is.

Components of the classes of statistics¶

Component	Factory	Class of statistics	Parameters
`BasicStatsComponent`	`ests_basic`	`BasicStats`	-
`ReadabilityStatsComponent`	`ests_readability`	`ReadabilityStats`	`preset`, `basic`
`UnadaptedReadabilityStatsComponent`	`ests_unadapted_readability`	`UnadaptedReadabilityStats`	`basic`
`DiversityStatsComponent`	`ests_diversity`	`DiversityStats`	`window_len`, `mtld_threshold`, `mtld_min_len`, `hdd_sample_size`, `log_base`
`MorphStatsComponent`	`ests_morph`	`MorphStats`	-
`SyntaxStatsComponent`	`ests_syntax`	`SyntaxStats`	-
`CohesionStatsComponent`	`ests_cohesion`	`CohesionStats`	`blank_lines`
`LexicalStatsComponent`	`ests_lexical`	`LexicalStats`	`data_dir`
`StyleStatsComponent`	`ests_style`	`StyleStats`	`stopwords`, `top_n`, `nausea_stopwords`, `cliches`
`PhonStatsComponent`	`ests_phon`	`PhonStats`	`window_len`
`VerseStatsComponent`	`ests_verse`	`VerseStats`	`seseo`

The basic, readability, diversity, phonetic and verse statistics and the formulas of other languages need no annotation of a model. In the pipeline of es_core_news_sm the parts of speech come from the morphologizer (or a tagger with an attribute_ruler; a tagger alone does not give them), the parse from the parser and the lemmas from the lemmatizer. A component whose annotation is missing, excluded components included, raises SourceError when the document goes through it; ests_lexical needs only the parts of speech, and ests_style raises when verbal_nouns or markers() is read.

Basic statistics¶

BasicStatsComponent(nlp, name)

BasicStatsComponent puts the basic statistics of a document into its extension.

Readability metrics¶

ReadabilityStatsComponent(nlp, name, preset=None, basic=None)

ReadabilityStatsComponent puts the readability metrics into the extension, computed from the document or, with basic, from the basic statistics another component put into it. A library sets basic_factory, the factory of its component of the basic statistics, which the error names when that extension holds none.

Parameter	Type	Default	Description
`preset`	`str`	`None`	Preset of the coefficients, one of the presets of the class; the default of the class if `None`
`basic`	`str`	`None`	Extension of a component of the basic statistics to reuse instead of computing them again

With preset left at None the config of a saved pipeline holds no preset, and the pipeline takes the default of the class when it is loaded; a library that wants its default written into the config gives it in the signature of its subclass.

Raises:

ParameterError - a stats_class that is not concrete, a preset that is not a string or not one of the class, an extension not named by a string;
SourceTypeError - a class without basic_stats_class and no basic;
SourceError - a document whose extension basic holds no basic statistics of the class.

The presets of esTS are general and classic, the preset of the class general. In esTS the class of statistics and its basic_stats_class are set, so the errors of stats_class and SourceTypeError do not occur. The component named by basic comes before this one in the pipeline; the message of SourceError names the component ests_basic (basic_factory) to add there.

CodeResult

nlp.add_pipe("ests_readability", name="readability")
config = {"preset": "classic", "basic": "basic"}
nlp.add_pipe("ests_readability", name="readability_classic", config=config)
doc = nlp("El gato duerme en la ventana. Los niños juegan en el parque.")
round(doc._.readability.flesch_reading_easy, 2)
round(doc._.readability_classic.flesch_reading_easy, 2)

102.19
105.72

Formulas of other languages¶

UnadaptedReadabilityStatsComponent(nlp, name, basic=None)

UnadaptedReadabilityStatsComponent puts the readability formulas of other languages into the extension, computed from the document or, with basic, from the basic statistics another component put into it. A library sets basic_factory, as for ReadabilityStatsComponent.

Parameter	Type	Default	Description
`basic`	`str`	`None`	Extension of a component of the basic statistics to reuse instead of computing them again

Raises:

ParameterError - a stats_class that is not concrete, an extension not named by a string;
SourceTypeError - a class without basic_stats_class and no basic;
SourceError - a document whose extension basic holds no basic statistics of the class.

The formulas are those of UnadaptedReadabilityStats, with their own coefficients and no calibration for Spanish. As for ReadabilityStatsComponent, the errors of stats_class and SourceTypeError do not occur in esTS, and the message of SourceError names the component ests_basic to add before it.

CodeResult

nlp.add_pipe("ests_unadapted_readability", name="unadapted", config={"basic": "basic"})
doc = nlp(
    "El gato estaba en la ventana y miraba a los pájaros que volaban "
    "sobre los tejados de las casas."
)
doc._.unadapted.gulpease, doc._.unadapted.forcast
doc._.unadapted.bs is doc._.basic

(64.78947368421052, 12.105263157894736)
True

Lexical diversity metrics¶

DiversityStatsComponent(nlp, name, window_len=50, mtld_threshold=0.72, mtld_min_len=10, hdd_sample_size=42, log_base=10)

DiversityStatsComponent puts the lexical diversity metrics into the extension, by a class of the library that reads a Doc. Its parameters are those of DiversityStats, with the same defaults.

Raises:

ParameterError - a wrong parameter (check_params of the diversity).

Parameter	Type	Default	Description
`window_len`	`int`	`50`	Window size for MATTR and segment size for MSTTR
`mtld_threshold`	`float`	`0.72`	TTR threshold for MTLD, MA-MTLD and MTLD-W
`mtld_min_len`	`int`	`10`	Minimum factor length for MTLD, MA-MTLD and MTLD-W
`hdd_sample_size`	`int`	`42`	Sample size for HD-D
`log_base`	`float`	`10`	Logarithm base for the Summer, Maas and Dugast metrics

CodeResult

config = {"window_len": 100, "log_base": 2.718281828459045}
nlp.add_pipe("ests_diversity", name="diversity", config=config)
doc = nlp("El gato duerme en la ventana. Los niños juegan en el parque.")
round(doc._.diversity.ttr, 3), round(doc._.diversity.sttr, 3)

(0.833, 0.916)

Morphological statistics¶

MorphStatsComponent(nlp, name)

MorphStatsComponent puts the morphological statistics into the extension; by default they need the parts of speech before it.

In esTS the pipeline needs a lemmatizer before the component as well; the factory requires token.pos and token.lemma, so nlp.analyze_pipes() names a pipeline without one of them.

Raises:

SourceError - a document without the parts of speech or the lemmas.

CodeResult

nlp.add_pipe("ests_morph", name="morph")
doc = nlp("El gato duerme en la ventana. Los niños juegan en el parque.")
doc._.morph.pos[:4]

('DET', 'NOUN', 'VERB', 'ADP')

Syntactic statistics¶

SyntaxStatsComponent(nlp, name)

SyntaxStatsComponent puts the syntactic statistics into the extension; they need a parser and the parts of speech before it.

In esTS they need a lemmatizer before it as well; the factory requires token.dep, token.head, token.pos and token.lemma, so nlp.analyze_pipes() names a pipeline without one of them.

Raises:

SourceError - a document without a dependency parse, the parts of speech or the lemmas.

CodeResult

nlp.add_pipe("ests_syntax", name="syntax")
doc = nlp("El gato duerme en la ventana. Los niños juegan en el parque.")
doc._.syntax.tree_depth, doc._.syntax.mean_dependency_distance

(2.0, 1.6)

Cohesion statistics¶

CohesionStatsComponent(nlp, name, blank_lines=False)

CohesionStatsComponent puts the cohesion statistics into the extension; by default they need the parts of speech and the lemmas before it. With blank_lines the paragraphs start after the blank lines alone, as in CohesionStats, for a text whose lines are wrapped inside its paragraphs.

Raises:

ParameterError - a blank_lines that is not a bool.

Parameter	Type	Default	Description
`blank_lines`	`bool`	`False`	Start the paragraphs after the blank lines alone, for a text whose lines are wrapped inside its paragraphs; after every line break by default

In the pipeline of es_core_news_sm the parts of speech and the lemmas come from the morphologizer and the lemmatizer; the factory requires token.pos and token.lemma, so nlp.analyze_pipes() names a pipeline without one of them, and a document without them raises SourceError. The texts of SpanishLiterature are wrapped inside their paragraphs and need blank_lines.

CodeResult

nlp.add_pipe("ests_cohesion", name="cohesion")
nlp.add_pipe("ests_cohesion", name="paragraphs", config={"blank_lines": True})
doc = nlp("El informe fue aprobado. Sin embargo, el informe\nno resuelve el problema.")
(
    doc._.cohesion.noun_overlap_adjacent,
    round(doc._.cohesion.connectors, 2),
    doc._.cohesion.n_paragraphs,
    doc._.paragraphs.n_paragraphs,
)

(1.0, 83.33, 2, 1)

Lexical sophistication statistics¶

LexicalStatsComponent(nlp, name, data_dir=None)

LexicalStatsComponent puts the lexical sophistication statistics into the extension, with the frequency dictionary of its attribute freq_dict: None by default, for the dictionary of the class, and set by a library in the constructor of its component. A document gets the statistics when it has a word of iter_doc_words (with join_hyphens) other than a number of the method is_number, anyts.utils.is_number by default; a library gives the numbers its class leaves out, so a document of numbers alone passes untouched.

In esTS freq_dict is the frequency dictionary FreqDict of the directory data_dir, and is_number takes the numbers of the Spanish NUMBER_PATTERN (2020, 1.500,50, 3.º). The pipeline needs a morphologizer before the component, and the factory requires token.pos; the lemmas of the model are not used. The frequency bands and the lexical density are computed without the dictionary, the statistics by the dictionary need it downloaded.

Parameter	Type	Default	Description
`data_dir`	`str`	`None`	Directory of the frequency dictionary; the default one if not given

Raises:

SourceError - a document without the parts of speech;
DatasetNotFoundError - the statistics by the dictionary and get_stats() when the dictionary is not downloaded;
ConfigValidationError of spaCy - a data_dir that is not a string, at add_pipe.

CodeResult

nlp.add_pipe("ests_lexical", name="lexical")
doc = nlp("El felinólogo examinaba al minino con parsimonia.")
round(doc._.lexical.coverage, 3), round(doc._.lexical.p_top1000, 3)
nlp("2020 1.500,50 3.º")._.lexical is None

(0.857, 0.429)
True

Style metrics¶

StyleStatsComponent(nlp, name, stopwords=None, top_n=10, nausea_stopwords=None, cliches=None)

StyleStatsComponent puts the style metrics into the extension.

Parameter	Type	Default	Description
`stopwords`	`list[str]`	`None`	Stopwords of the water content; those of the class if `None`
`top_n`	`int`	`10`	Number of the most frequent words of the academic nausea and of the naturalness by Zipf's law (`NAUSEA_TOP_N`)
`nausea_stopwords`	`list[str]`	`None`	Words left out of the most frequent words of the academic nausea; those of the class if `None`
`cliches`	`list[str]`	`None`	Cliches; those of the class (`cliches_list`) if `None`

Raises:

SourceTypeError - stopwords, nausea stopwords or cliches that are not a list of strings;
ParameterError - a top_n that is not an integer of at least one.

Without the lists the water content takes the stopwords of STOPWORDS and the one-word parenthetical expressions, the academic nausea leaves out the words of is_stopword, and the clichés come from OFFICIALESE_CLICHES; NAUSEA_TOP_N is a constant of anyts.constants. The verbal nouns need a morphologizer and a lemmatizer before the component, and without them reading verbal_nouns, markers() or get_stats() raises SourceError; the other metrics read the words alone.

CodeResult

nlp.add_pipe("ests_style", name="style")
doc = nlp("Se procedió a la revisión del expediente a la mayor brevedad.")
round(doc._.style.cliches, 2), round(doc._.style.verbal_nouns, 2)
nlp.add_pipe("ests_style", name="style_cliches", config={"cliches": ["se procedió a"]})
doc = nlp("Se procedió a la revisión del expediente a la mayor brevedad.")
round(doc._.style_cliches.cliches, 2)

(18.18, 33.33)
9.09

Phonostatistics¶

PhonStatsComponent(nlp, name, window_len=3)

PhonStatsComponent puts the phonetic statistics into the extension; window_len (PHON_WINDOW_LEN by default) is the window in words of the alliteration and the assonance.

Raises:

ParameterError - a window_len that is not an integer of at least 2.

CodeResult

nlp.add_pipe("ests_phon", name="phon", config={"window_len": 5})
doc = nlp("Tres tristes tigres tragaban trigo en un trigal")
round(doc._.phon.hardness, 3), round(doc._.phon.alliteration, 3)

(0.458, 0.975)

Verse statistics¶

VerseStatsComponent(nlp, name, seseo=False)

VerseStatsComponent puts the verse statistics into the extension. The statistics read the line breaks and blank lines of the document, so pass the text of a poem to nlp with its lines not joined; the component needs no annotation of a model. A document with no letter passes untouched, and a text with letters but no Spanish syllables gives empty statistics.

Parameter	Type	Default	Description
`seseo`	`bool`	`False`	Pronounce `c` and `z` before `e` and `i` as `s` in the rhyme

Raises:

ConfigValidationError of spaCy - a seseo that is not a boolean, at add_pipe.

CodeResult

nlp.add_pipe("ests_verse", name="verse")
doc = nlp(
    "Que por mayo era, por mayo,\ncuando hace la calor,\ncuando los trigos encañan\ny están los campos en flor,"
)
doc._.verse.meter, doc._.verse.c_clausulas

('octosílabo', {'aguda': 2, 'llana': 2})

Sentences after blank lines¶

start_sents_after_blank_lines(doc)

A component that starts a sentence at the first token after a blank line - a line break, optional whitespace and another line break - so that a heading, a list item or a paragraph without a final mark is a sentence of its own, as in the sentence extractor. It goes before the parser, senter or sentencizer, which keeps its starts and places the others. The whitespace before such a start stays with the sentence before it, and the marks that open a paragraph (a dash of a dialogue, ¡, «) stay with its first word. Alone, the component leaves the sentences within a paragraph unsplit and sets partial boundaries (has_sents), so the statistics take the sentences of such a document from its text. A parsed document, such as one passed through the pipeline again, passes untouched, since spaCy keeps its sentences. A language library registers the component under its prefix, Language.component("<prefix>_blank_lines", func=start_sents_after_blank_lines).

Parameter	Type	Default	Description
`doc`	`Doc`	-	Doc object

In esTS the component is registered as ests_blank_lines. Without it the es_core_news_sm model glues a paragraph without a final mark to the next sentence, and the statistics of a Doc differ from those of the same string. The pipeline of ests.utils.get_nlp, which parses the strings of the morphological, syntactic, cohesion and lexical statistics, has it before the parser.

CodeResult

nlp = spacy.load("es_core_news_sm")
nlp.add_pipe("ests_blank_lines", before="parser")
doc = nlp("Novedades\n\nImportación rápida. Menos errores")
[sent.text.strip() for sent in doc.sents]

['Novedades', 'Importación rápida.', 'Menos errores']

Everything in one pipeline¶

The eleven components can share one pipeline.

CodeResult

import ests
import spacy

nlp = spacy.load("es_core_news_sm")
nlp.add_pipe("ests_blank_lines", before="parser")
factories = (
    "basic",
    "readability",
    "unadapted_readability",
    "diversity",
    "morph",
    "syntax",
    "cohesion",
    "lexical",
    "style",
    "phon",
    "verse",
)
for factory in factories:
    nlp.add_pipe(f"ests_{factory}", name=factory)

doc = nlp("El gato duerme en la ventana. Los niños juegan en el parque.")
(
    doc._.basic.n_words,
    round(doc._.readability.flesch_reading_easy, 2),
    round(doc._.unadapted_readability.gulpease, 2),
    round(doc._.diversity.ttr, 3),
    doc._.morph.pos[:2],
    doc._.syntax.tree_depth,
    doc._.cohesion.p_given,
    round(doc._.lexical.p_top1000, 3),
    round(doc._.style.water, 2),
    round(doc._.phon.p_vowels, 3),
    doc._.verse.n_lines,
)

(12, 102.19, 99.83, 0.833, ('DET', 'NOUN'), 2.0, 0.0, 0.667, 50.0, 0.457, 1)