Morphological statistics¶

MorphStats(source, nlp=None)

The morphological statistics of a text: the part of speech and the features of Universal Dependencies of every word, counted by value or given word by word. By default the data source is a Doc object of spaCy with the parts of speech, read by token.pos_ and token.morph; a language library reads a string or a Doc without them in its own way.

Raises:

SourceTypeError - a source that is not a Doc, an analyze that does not give MorphWord objects;
SourceError - a source without parts of speech or without words.

The class extends the MorphStats of the anyTS core with the Spanish layer and the markers of Spanish; the statistics are computed when the object is created. In esTS the data source is a text or a Doc: a text is parsed with es_core_news_sm or with the pipeline passed in nlp, and a Doc must carry the parts of speech, which come from a morphologizer (or a tagger with an attribute_ruler), and the lemmas, which come from a lemmatizer. Words are taken from the tokens, punctuation marks and symbols are dropped. A text longer than the max_length of the pipeline - a million characters by default - has to be split into parts, or max_length raised on a pipeline of your own passed in nlp. The print_stats and print_markers tables take the descriptions of MORPHOLOGY_STATS_DESC and MORPHOLOGY_MARKERS_DESC of ests.constants.

Besides, in esTS a source that is neither a string nor a Doc and a pipeline in nlp that is not a spaCy Language raise SourceTypeError; a Doc without the lemmas (of a pipeline with the lemmatizer excluded), a string longer than the max_length of the pipeline or with a character that has no UTF-8 form raise SourceError; a string without nlp when the model es_core_news_sm is not installed raises DatasetNotFoundError.

CodeResult

from ests import MorphStats

text = "Si tuviera tiempo, leería el libro que me recomendaste ayer"
ms = MorphStats(text)
ms.print_stats("pos", "mood")

-------------Part of speech-------------
Verb                          |    3
Noun                          |    2
Pronoun                       |    2
Subordinating conjunction     |    1
Determiner                    |    1
Adverb                        |    1

------------------Mood------------------
Unknown                       |    7
Subjunctive                   |    1
Conditional                   |    1
Indicative                    |    1

Parameters¶

Parameter	Type	Default	Description
`source`	`str` / `Doc`	-	Data source
`nlp`	`Language`	`None`	Pipeline of spaCy that parses a string; without it the model `es_core_news_sm` is loaded

Attributes¶

Attribute	Type	Description
`words`	`tuple[str]`	Words of the text
`tags`	`tuple[str]`	Features of every word in the CoNLL-U format (`Case=Nom\|Number=Plur`), `_` for a word without them
`pos`	`tuple[str / None]`	Part of speech of every word
`animacy`, `aspect`, ..., `voice`	`tuple[str / None]`	Value of a statistic for every word, `None` for a word without the feature

In esTS the statistics are case, ..., verb_form of the table below; animacy, aspect and voice of the core are not among them. esTS also has the attribute lemmas (tuple[str]) - the lemmas of the words that the model gives.

CodeResult

ms.lemmas[:4]
ms.tags[1]

('si', 'tener', 'tiempo', 'leer')
'Mood=Sub|Number=Sing|Person=3|Tense=Imp|VerbForm=Fin'

Statistics¶

Every statistic is an attribute of the object, a tuple of the values of a feature by word (None for a word the model gives no value of the feature), and get_stats gives the distributions of their values. The values are read from token.pos_ and token.morph of the parse, that is, by context. The statistics are the part of speech and the fifteen features of MORPHOLOGY_FEATURES; the value Unknown in the printed tables means that the model gave the word no value of the feature. The other features the models annotate (AdvType, Foreign, NumForm, Number[psor], PrepCase, Typo, PunctSide, PunctType) are not counted but stay in tags.

Statistic	Feature	Values
`pos`	Part of speech	`NOUN`, `PROPN`, `ADJ`, `ADV`, `VERB`, `AUX`, `PRON`, `DET`, `NUM`, `ADP`, `CCONJ`, `SCONJ`, `PART`, `INTJ`, `SYM`, `X`
`case`	Case	`Nom`, `Acc`, `Dat`, `Com`
`definite`	Definiteness	`Def`, `Ind`
`degree`	Degree	`Cmp`, `Sup`, `Abs`
`gender`	Gender	`Masc`, `Fem`
`mood`	Mood	`Ind`, `Sub`, `Imp`, `Cnd`
`num_type`	Numeral type	`Card`, `Ord`, `Frac`
`number`	Number	`Sing`, `Plur`
`person`	Person	`1`, `2`, `3`
`polarity`	Polarity	`Neg`
`polite`	Politeness	`Form`
`poss`	Possessive	`Yes`
`pron_type`	Pronoun type	`Art`, `Prs`, `Dem`, `Ind`, `Int`, `Rel`, `Neg`, `Tot`, `Exc`
`reflex`	Reflexive	`Yes`
`tense`	Tense	`Pres`, `Past`, `Imp`, `Fut`
`verb_form`	Verb form	`Fin`, `Inf`, `Part`, `Ger`

A feature with several values keeps the form of CoNLL-U: qué, quién, cuál have pron_type equal to Int,Rel, and usted has case equal to Acc,Nom. print_stats names the values by MORPHOLOGY_STATS_DESC of ests.constants, and such a value by the names of its parts joined with a slash - "Interrogative / Relative", "Accusative / Nominative".

Warning

The statistics are as good as the annotation of the model. es_core_news_sm mis-analyses verbs with enclitic pronouns (dámelo, cuéntamelo come out as nouns with invented lemmas) and tags the imperatives of tú as indicative (Abre la ventana gets Mood=Ind), so p_imperative under-reports and p_indicative absorbs the orders: in the dialogue of the corpus of literature it finds about one imperative in ten. es_core_news_lg does little better (15%); the transformer model es_dep_news_trf passed in nlp finds 87% of them, and in drama p_imperative rises from 1.6% to 7.5% - at the cost of about 1 GB with PyTorch and five times the time on a CPU, and it needs the single line breaks of a hard-wrapped text joined first.

Markers of Spanish¶

get_markers() · print_markers()

The shares of the forms that mark the register of a Spanish text, each of its own base (MORPHOLOGY_MARKERS_DESC), so the markers of texts of different lengths can be compared directly; a marker whose base is empty - a text without verbs, without a copula, without adverbs - is nan. The subjunctive marks hypothesis and subordination, estar a state against the property of ser, and the adverbs in -mente a formal, written register.

Marker	Share	Base
`p_indicative`, `p_subjunctive`, `p_conditional`, `p_imperative`	of the indicative, subjunctive, conditional and imperative mood	finite forms
`p_infinitive`, `p_gerund`, `p_participle`	of infinitives, gerunds and participles	verb forms
`p_ser`	of `ser`	copulas `ser` and `estar`
`p_mente_adverbs`	of the adverbs in `-mente`	adverbs

The first four markers share the base of the finite forms and sum to one unless the model leaves a finite form without a mood or gives it several (Mood=Ind,Sub, which counts in the share of each; es_core_news_sm gives none). The next three share the base of all the verb forms of verbs and auxiliaries: a participle that the model annotates as an adjective (la casa pintada) is out of the base, the ones of the compound tenses and the passive (he leído, fue escrito) are in.

The base of p_ser is the copular uses alone, read from the parse: the auxiliaries of the passive and of the progressive (fue escrito, es financiado, está cantando) are left out, while es alta, está cansada and lo importante es que vengas count. For a Doc without a parse the marker is nan.

CodeResult

ms.get_markers()

{'p_indicative': 0.3333333333333333,
 'p_subjunctive': 0.3333333333333333,
 'p_conditional': 0.3333333333333333,
 'p_imperative': 0.0,
 'p_infinitive': 0.0,
 'p_gerund': 0.0,
 'p_participle': 0.0,
 'p_ser': nan,
 'p_mente_adverbs': 0.0}

Methods¶

get_stats(*stats, filter_none=False, flat=False) gives the number of the words by value of every statistic selected, all of them by default, in the order of the first occurrences; None counts the words without the feature, and filter_none leaves them out. With flat the keys are statistic_value, such as pos_NOUN, without the words that lack the feature, as in to_dataframe. An unknown statistic raises UnknownStatError, a filter_none or a flat that is not a bool - ParameterError.

explain_text(*stats, filter_none=False) gives every word with the values of the statistics selected; filter_none leaves out those a word lacks. An unknown statistic raises UnknownStatError.

print_stats(*stats, filter_none=False) prints a table for every statistic selected: the descriptions of its values from the most frequent, with the number of the words. A value of several joins the descriptions of its parts with a slash. An unknown statistic raises UnknownStatError.

The example at the top of the page prints two of the tables. to_dataframe and describe are those of the statistics interface of the anyTS core; the morphological statistics have no scales, so describe gives None.

CodeResult

ms.get_stats("pos", "mood", "tense", filter_none=True)
ms.explain_text("pos", "mood", "tense", filter_none=True)

{'pos': {'SCONJ': 1, 'VERB': 3, 'NOUN': 2, 'DET': 1, 'PRON': 2, 'ADV': 1},
 'mood': {'Sub': 1, 'Cnd': 1, 'Ind': 1},
 'tense': {'Imp': 1, 'Pres': 1}}
(('Si', {'pos': 'SCONJ'}),
 ('tuviera', {'pos': 'VERB', 'mood': 'Sub', 'tense': 'Imp'}),
 ('tiempo', {'pos': 'NOUN'}),
 ('leería', {'pos': 'VERB', 'mood': 'Cnd'}),
 ('el', {'pos': 'DET'}),
 ('libro', {'pos': 'NOUN'}),
 ('que', {'pos': 'PRON'}),
 ('me', {'pos': 'PRON'}),
 ('recomendaste', {'pos': 'VERB', 'mood': 'Ind', 'tense': 'Pres'}),
 ('ayer', {'pos': 'ADV'}))

get_markers() gives the markers of a language computed from the features - shares of grammatical forms, each of its own base - by the names of markers_desc, and print_markers() prints them with their descriptions. A library computes them; there are none by default.

In esTS the markers are the markers of Spanish.

CodeResult

ms.print_markers()

               Marker               |  Value
-----------------------------------------------
Indicative among the finite forms   |   0.33
Subjunctive among the finite forms  |   0.33
Conditional among the finite forms  |   0.33
Imperative among the finite forms   |   0.00
Infinitive among the verb forms     |   0.00
Gerund among the verb forms         |   0.00
Participle among the verb forms     |   0.00
ser among the copulas ser and estar |   nan
Adverbs in -mente among the adverbs |   0.00

MorphWord is a word of analyze: its text word, its part of speech pos (None when unknown) and its features features - the values of the features of Universal Dependencies by their names ({"Case": "Nom"}), a value of several joined by commas.

In esTS MorphWord is imported from ests.morph_stats too.