Morphological statistics¶

ests.morph_stats.MorphStats

Description¶

A module for computing the morphological statistics of a text. The data source can be either a text or a Doc object of the spaCy library.

Parts of speech and grammatical features are given in the terms of Universal Dependencies, as the Spanish models of spaCy annotate them. A text is parsed with es_core_news_sm or with the pipeline passed in nlp, without the entity recognizer, which nothing here reads; a Doc is taken as it is and must carry the parts of speech, which come from a morphologizer (or a tagger with an attribute_ruler), and the lemmas, which come from a lemmatizer - a Doc of spacy.blank("es") or of a pipeline with the lemmatizer excluded is not a valid source and raises SourceError. Words are taken from the tokens, punctuation marks and symbols are dropped.

A text longer than the max_length of the pipeline - a million characters by default, a long novel - raises SourceError instead of reaching spaCy: split it into parts, or raise max_length on a pipeline of your own and pass it in nlp.

Note

The statistics are computed when the MorphStats object is initialized.

Parameters¶

Parameter	Type	Default	Description
`source`	str/Doc	`-`	Data source (a string or a Doc object)
`nlp`	Language	`None`	Pipeline of spaCy that parses a string; without it the model `es_core_news_sm` is loaded

Attributes¶

Attribute	Type	Description
`words`	tuple[str]	Tuple of extracted words
`lemmas`	tuple[str]	Tuple of lemmas of the words
`tags`	tuple[str]	Tuple of feature strings in the CoNLL-U format
`pos`	tuple[str]	Tuple of the parts of speech
`case`	tuple[str]	Tuple of the values of case
`definite`	tuple[str]	Tuple of the values of definiteness
`degree`	tuple[str]	Tuple of the values of degree
`gender`	tuple[str]	Tuple of the values of gender
`mood`	tuple[str]	Tuple of the values of mood
`num_type`	tuple[str]	Tuple of the values of the numeral type
`number`	tuple[str]	Tuple of the values of number
`person`	tuple[str]	Tuple of the values of person
`polarity`	tuple[str]	Tuple of the values of polarity
`polite`	tuple[str]	Tuple of the values of politeness
`poss`	tuple[str]	Tuple of the values of the possessive
`pron_type`	tuple[str]	Tuple of the values of the pronoun type
`reflex`	tuple[str]	Tuple of the values of the reflexive
`tense`	tuple[str]	Tuple of the values of tense
`verb_form`	tuple[str]	Tuple of the values of the verb form

Every attribute has the length of words, and a word that the model gives the feature no value for holds None. The names of the attributes are the names of the statistics accepted by the methods; tags and lemmas are not statistics.

Note

A feature with several values keeps the form of CoNLL-U: the interrogative and relative qué, quién, cuál, which the models do not disambiguate, has pron_type equal to Int,Rel, and usted has case equal to Acc,Nom. print_stats describes such a value by the descriptions of its parts - "Interrogative or relative", "Accusative or nominative".

Features¶

The statistics count the fifteen features of the table below; the value Unknown in the printed tables and None in the attributes mean that the model gave the word no value of the feature.

They are a subset: the Spanish models annotate 23 features, and the ones left out are either marginal (AdvType, Foreign, NumForm, Number[psor], PrepCase, Typo) or live on punctuation (PunctSide, PunctType), which this module does not treat as words. Whatever the model annotates stays in tags, counted or not.

Statistic	Feature	Values
`pos`	Part of speech	NOUN, PROPN, ADJ, ADV, VERB, AUX, PRON, DET, NUM, ADP, CCONJ, SCONJ, PART, INTJ, SYM, X
`case`	Case	Nom, Acc, Dat, Com
`definite`	Definiteness	Def, Ind
`degree`	Degree	Cmp, Sup, Abs
`gender`	Gender	Masc, Fem
`mood`	Mood	Ind, Sub, Imp, Cnd
`num_type`	Numeral type	Card, Ord, Frac
`number`	Number	Sing, Plur
`person`	Person	1, 2, 3
`polarity`	Polarity	Neg
`polite`	Politeness	Form
`poss`	Possessive	Yes
`pron_type`	Pronoun type	Art, Prs, Dem, Ind, Int, Rel, Neg, Tot, Exc
`reflex`	Reflexive	Yes
`tense`	Tense	Pres, Past, Imp, Fut
`verb_form`	Verb form	Fin, Inf, Part, Ger

Warning

The statistics are as good as the annotation of the model. es_core_news_sm mis-analyses verbs with enclitic pronouns: dámelo, decírselo, vámonos, cuéntamelo come out as nouns or proper nouns and get invented lemmas (dámelir, siéntatir). The imperative suffers most of all: the models carry the value Mood=Imp in their label set - 21 of the 433 labels of the morphologizer - but in practice they tag the imperatives of tú as indicative, and Habla más despacio, Abre la ventana and Ven aquí all get Mood=Ind. Over 24 imperative sentences of tú, vosotros and usted, with and without enclitics, es_core_news_sm found the imperative twice, es_core_news_md three times and es_core_news_lg four, so p_imperative under-reports and p_indicative absorbs the orders.

Passing a bigger model in nlp does not change that. Measured against es_core_news_sm, both es_core_news_md (54 MB) and es_core_news_lg (631 MB) agree with it on every part of speech of modern prose and run at the same speed - the vectors they add weigh on memory, not on the tagger. What they bring is the rare and the old vocabulary: in the opening of the Quijote they read rocín, salpicón, lentejas and carnero as nouns, where the small model sees verbs and adjectives. On the enclitics they trade one improvement for one regression, Dime becoming a verb while Cantándole stops being a gerund.

Methods¶

get_stats¶

Returns a dictionary with the computed morphological statistics: for every statistic, how many words carry each of its values.

Parameters:

Parameter	Type	Default	Description
`args`	tuple[str]	`-`	Names of the selected statistics, by default all of them
`filter_none`	bool	`False`	Filter out the empty values

Example

Code:

# Import the library
from ests import MorphStats

# Prepare the data
text = "Si tuviera tiempo, leería el libro que me recomendaste ayer"

# Compute the statistics
ms = MorphStats(text)
ms.get_stats("pos", "mood", "tense", filter_none=True)

Result:

{'pos': {'SCONJ': 1, 'VERB': 3, 'NOUN': 2, 'DET': 1, 'PRON': 2, 'ADV': 1},
'mood': {'Sub': 1, 'Cnd': 1, 'Ind': 1},
'tense': {'Imp': 1, 'Pres': 1}}

get_markers¶

Returns a dictionary with the markers of Spanish computed from the features. Every marker is a share of its own base, so the markers of texts of different lengths can be compared directly:

Marker	Description
`p_indicative`	Indicative among the finite forms
`p_subjunctive`	Subjunctive among the finite forms
`p_conditional`	Conditional among the finite forms
`p_imperative`	Imperative among the finite forms
`p_infinitive`	Infinitive among the verb forms
`p_gerund`	Gerund among the verb forms
`p_participle`	Participle among the verb forms
`p_ser`	`ser` among the copulas `ser` and `estar`
`p_mente_adverbs`	Adverbs in `-mente` among the adverbs

The first four markers share the base of the finite forms and sum to one wherever the model leaves no finite form without a mood - five of its 433 labels carry VerbForm=Fin and no Mood, and each such form is missing from all four shares. The next three markers share the base of all the verb forms, counted on verbs and auxiliaries, so that the participles that the model annotates as adjectives (la casa pintada) stay out of the base, while the ones of the compound tenses and the passive (he leído, fue escrito) stay in.

The base of p_ser is the copular uses alone, read from the dependency of the token and from what it depends on: fue escrito and está cantando are the auxiliaries of the passive and of the progressive, and so is the es of es financiado, which the models tag as a copula in the present - none of the three is a choice between the two copulas, while es alta, está cansada and lo importante es que vengas are. The parse is what tells them apart, so for a Doc that carries none the marker is nan.

A marker whose base is empty - a text without verbs, without a copula, without adverbs - is nan.

Note

The subjunctive, the choice between ser and estar and the adverbs in -mente are the traits of Spanish that the readability formulas do not see: the subjunctive marks hypothesis and subordination, estar a state against the property of ser, and the adverbs in -mente a formal, written register.

Example

Code:

...

# Compute the markers
ms.get_markers()

Result:

{'p_indicative': 0.3333333333333333,
'p_subjunctive': 0.3333333333333333,
'p_conditional': 0.3333333333333333,
'p_imperative': 0.0,
'p_infinitive': 0.0,
'p_gerund': 0.0,
'p_participle': 0.0,
'p_ser': nan,
'p_mente_adverbs': 0.0}

explain_text¶

Returns a tuple of the words of the text with the values of the selected statistics.

Parameters:

Parameter	Type	Default	Description
`args`	tuple[str]	`-`	Names of the selected statistics, by default all of them
`filter_none`	bool	`False`	Filter out the empty values

Example

Code:

...

# Parse the text
ms.explain_text("pos", "mood", "tense", filter_none=True)

Result:

(('Si', {'pos': 'SCONJ'}),
('tuviera', {'pos': 'VERB', 'mood': 'Sub', 'tense': 'Imp'}),
('tiempo', {'pos': 'NOUN'}),
('leería', {'pos': 'VERB', 'mood': 'Cnd'}),
('el', {'pos': 'DET'}),
('libro', {'pos': 'NOUN'}),
('que', {'pos': 'PRON'}),
('me', {'pos': 'PRON'}),
('recomendaste', {'pos': 'VERB', 'mood': 'Ind', 'tense': 'Pres'}),
('ayer', {'pos': 'ADV'}))

print_stats¶

Prints a table of the values of every selected statistic, from the most frequent to the least frequent.

Parameters:

Parameter	Type	Default	Description
`args`	tuple[str]	`-`	Names of the selected statistics, by default all of them
`filter_none`	bool	`False`	Filter out the empty values

Example

Code:

...

# Print the table of computed statistics
ms.print_stats("pos", "mood")

Result:

-------------Part of speech-------------
Verb                          |    3
Noun                          |    2
Pronoun                       |    2
Subordinating conjunction     |    1
Determiner                    |    1
Adverb                        |    1

------------------Mood------------------
Unknown                       |    7
Subjunctive                   |    1
Conditional                   |    1
Indicative                    |    1

print_markers¶

Prints a table with the computed markers of Spanish.

Example

Code:

...

# Print the table of computed markers
ms.print_markers()

Result:

                     Marker                    |   Value
-----------------------------------------------------------
Indicative among the finite forms              |   0.33
Subjunctive among the finite forms             |   0.33
Conditional among the finite forms             |   0.33
Imperative among the finite forms              |   0.00
Infinitive among the verb forms                |   0.00
Gerund among the verb forms                    |   0.00
Participle among the verb forms                |   0.00
ser among the copulas ser and estar            |    nan
Adverbs in -mente among the adverbs            |   0.00