Syntactic statistics¶

SyntaxStats(source, nlp=None)

The syntactic statistics of a text by its dependency tree (Ivanov, Solnyshkina and Solovyev 2018): the measures of the tree and of the syntactic constructions. The data source is a text or a Doc object of spaCy, and it has to be parsed: a string is parsed with es_core_news_sm or with the pipeline passed in nlp, without the named entities, and a Doc must carry the dependencies, which come from a parser, the parts of speech, which come from a morphologizer (or a tagger with an attribute_ruler), and the lemmas, which come from a lemmatizer. Punctuation marks, symbols (%, €, +) and whitespace are no nodes of the tree, and the distances are counted in positions of words.

Raises:

SourceTypeError - a source that is neither a string nor a Doc, a pipeline in nlp that is not a spaCy Language;
SourceError - a source without words; a Doc without the dependencies, the parts of speech or the lemmas (a Doc of spacy.blank("es"), of a pipeline without a parser or with the morphologizer or the lemmatizer excluded); a string longer than the max_length of the pipeline or with a character without a UTF-8 form;
DatasetNotFoundError - a string without nlp when the model is not installed.

The class extends the SyntaxStats of the anyTS core with the Spanish layer. The relations and features are those of the Universal Dependencies annotation of the es_core_news_sm, es_core_news_md and es_core_news_lg models, and the statistics are as good as the parse. The clauses, the passive, the negations and the split predicates follow the rules of Spanish given in the cards of the metrics, and esTS adds its own metrics of the chains of de, the gerund clauses and the sentences with se. The constructions of the administrative style - the passive with ser and with se, the participial and gerund clauses, the chains of de, the split predicates - are the ones the Spanish-language guides to plain language (lenguaje claro) warn about. The print_stats table takes the descriptions of SYNTAX_STATS_DESC of ests.constants.

The max_length of the pipeline is a million characters by default: split a longer text into parts, or raise max_length on a pipeline of your own and pass it in nlp.

CodeResult

from ests import SyntaxStats

text = (
    "La revisión de las cuentas del organismo fue realizada por el comité. "
    "Se llevó a cabo la reforma, aprobada en 1998, sin que nadie hiciera "
    "mención de los problemas."
)
ss = SyntaxStats(text)
ss.print_stats()

                         Statistic                         |  Value
----------------------------------------------------------------------
Mean dependency distance                                   |   2.26
Standard deviation of the dependency distance              |   2.15
Mean of the longest dependencies of the sentences          |   9.00
Share of adjacent dependencies                             |   0.48
Depth of the dependency tree                               |   4.00
Leaves per sentence                                        |   8.00
Subtrees per sentence                                      |   6.50
Mean of the nodes per leaf of the sentences                |   1.80
Mean number of words before the root of a sentence         |   4.50
Valency of the finite verbs                                |   4.00
Coordination chains per sentence                           |   0.00
Mean length of a coordination chain                        |   nan
Clauses per sentence                                       |   1.50
Mean length of a clause (words)                            |   9.67
Subordinate clauses per sentence                           |   0.50
Share of sentences with a subordinate clause               |   0.50
Infinitive clauses per sentence                            |   0.00
T-units per sentence                                       |   1.00
Mean length of a T-unit (words)                            |  14.50
Clauses per T-unit                                         |   1.50
Share of T-units with a subordinate clause                 |   0.50
Verb phrases per T-unit                                    |   1.50
Coordinate phrases per clause                              |   0.00
Complex nominals per clause                                |   1.33
Modifiers per noun phrase                                  |   1.00
Prepositional chains per sentence                          |   0.50
Maximum length of a prepositional chain                    |    2
Chains of de per sentence                                  |   0.50
Maximum length of a chain of de                            |    2
Participial clauses per sentence                           |   0.50
Mean length of a participial clause (words)                |   3.00
Gerund clauses per sentence                                |   0.00
Mean length of a gerund clause (words)                     |   nan
Share of passive forms among the verbs                     |   0.67
Share of agentless forms among the passive ones            |   0.50
Passives with se per sentence                              |   0.50
Impersonal sentences with se per sentence                  |   0.00
Infinitives per sentence                                   |   0.00
Words of negation per sentence                             |   0.50
Split predicates per sentence                              |   1.00
Share of nominal subjects before their head                |   0.50
Share of nominal objects before their head                 |   0.00
Edit distance of the parts of speech of adjacent sentences |   0.47
Syntactic similarity of adjacent sentences                 |   0.45
Ratio of nouns to verbs                                    |   3.00

Parameters¶

Parameter	Type	Default	Description
`source`	`str` / `Doc`	-	Data source: a text or a `Doc` with the dependencies and the lemmas
`nlp`	`Language`	`None`	spaCy pipeline that parses a string; the model `es_core_news_sm` of `ests.utils.get_nlp` if not given

Attributes¶

Attribute	Type	Description
`sents`	`tuple[tuple[Token]]`	Words of every sentence with words, on a copy of the `Doc` without its tensor and user data
`n_sents`, `n_words`	`int`	Numbers of the sentences with words and of the words
`n_leaves`, `n_subtrees`	`int`	Numbers of the words without and with dependent words
`n_coordination_chains`	`int`	Number of the coordination chains
`n_clauses`, `n_subordinate_clauses`, `n_complex_sents`	`int`	Numbers of the clauses, of the subordinate clauses and of the sentences with one
`n_infinitive_clauses`	`int`	Number of the infinitive clauses
`n_t_units`, `n_complex_t_units`	`int`	Numbers of the T-units and of those with a subordinate clause
`n_l2sca_clauses`	`int`	Number of the clauses of L2SCA, without the predicates sharing a subject and the infinitive clauses
`n_verb_phrases`, `n_coordinate_phrases`, `n_complex_nominals`	`int`	Numbers of the verb phrases, of the coordinate phrases and of the complex nominals
`n_nouns`, `n_verbs`	`int`	Numbers of the nouns (`NOUN`, `PROPN`) and of the verb forms (`VERB`)
`n_prep_chains`	`int`	Number of the prepositional chains
`n_participle_clauses`	`int`	Number of the participial clauses
`n_passive`, `n_agentless_passive`	`int`	Numbers of the passive forms and of those with no agent
`n_infinitives`, `n_negations`	`int`	Numbers of the infinitives and of the words of negation
`n_split_predicates`, `split_predicates`	`int`, `tuple[str]`	Number of the split predicates and the predicates, the verb and the noun
`n_nominal_subjects`, `n_preverbal_subjects`	`int`	Numbers of the nominal subjects and of those before their head
`n_nominal_objects`, `n_preverbal_objects`	`int`	Numbers of the nominal direct objects and of those before their head
`c_children`	`dict[int, int]`	Number of the words by number of dependent words
`c_deps`	`dict[str, int]`	Number of the words by relation

The attributes of the constructions of esTS are n_de_chains, n_gerund_clauses, n_se_passives and n_impersonal_se (int): the numbers of the chains of de, of the gerund clauses, of the passives with se and of the impersonal sentences with se.

CodeResult

ss.n_clauses, ss.n_nouns, ss.n_verbs
ss.n_de_chains, ss.n_se_passives, ss.split_predicates
ss.c_children
ss.c_deps
ss.sents[0]

(3, 9, 3)
(1, 1, ('llevó cabo', 'hiciera mención'))
{0: 16, 1: 5, 2: 4, 3: 2, 4: 2}
{'ROOT': 2,
 'advcl': 1,
 'amod': 1,
 'aux': 1,
 'case': 6,
 'compound': 1,
 'det': 5,
 'expl:pass': 1,
 'mark': 2,
 'nmod': 3,
 'nsubj': 3,
 'obj': 2,
 'obl': 1}
(La, revisión, de, las, cuentas, del, organismo, fue, realizada, por, el, comité)

Metrics¶

Every metric is an attribute of the object, computed when it is created, and get_stats gives the metrics of the table. The measures of a sentence (the longest dependency, the depth, the nodes per leaf) are averaged over the sentences; the constructions are given per sentence.

Metric	Measures	Range	Higher means
`mean_dependency_distance`	Mean distance between a word and its head	1 and above, `nan` without dependencies	Harder to process
`std_dependency_distance`	Standard deviation of the dependency distance	0 and above, `nan` without dependencies	Less even distances
`max_dependency_distance`	Mean of the longest dependencies of the sentences that have dependencies	1 and above, `nan` without dependencies	Longer longest dependencies
`p_adjacent_dependencies`	Share of the dependencies of length 1	0-1, `nan` without dependencies	More links between neighbours
`tree_depth`	Mean depth of the tree of a sentence	0 and above	Deeper embedding
`leaves_per_sent`	Leaves per sentence	0 and above	Longer or flatter sentences
`subtrees_per_sent`	Subtrees per sentence	0 and above	More phrases in a sentence
`nodes_per_leaf`	Mean over the sentences of the words per leaf	1 and above, `nan` for a sentence without leaves	Longer branches
`words_before_root`	Mean number of the words before the root of a sentence	0 and above, `nan` without a root among the words	A later main predicate
`verb_valency`	Mean number of the dependents of a finite verb	0 and above, `nan` without finite verbs	More dependents of a verb
`coordination_chains_per_sent`	Coordination chains per sentence	0 and above	More coordination
`mean_coordination_chain_len`	Mean length of a coordination chain in words	2 and above, `nan` without chains	Longer enumerations
`clauses_per_sent`	Clauses per sentence	0 and above	More clauses in a sentence
`mean_clause_len`	Mean length of a clause in words	1 and above, `nan` without clauses	Longer clauses
`subordinate_clauses_per_sent`	Subordinate clauses per sentence	0 and above	More subordination
`p_complex_sents`	Share of the sentences with a subordinate clause	0-1	More complex sentences
`infinitive_clauses_per_sent`	Infinitive clauses per sentence	0 and above	More infinitive clauses
`t_units_per_sent`	T-units per sentence (T/S)	0 and above	More main clauses in a sentence
`mean_t_unit_len`	Mean length of a T-unit in words (MLT)	1 and above, `nan` without T-units	Longer T-units
`clauses_per_t_unit`	Clauses per T-unit (C/T)	1 and above, `nan` without T-units	More subordination
`p_complex_t_units`	Share of the T-units with a subordinate clause (CT/T)	0-1, `nan` without T-units	More complex T-units
`verb_phrases_per_t_unit`	Verb phrases per T-unit (VP/T)	0 and above, `nan` without T-units	More verb phrases
`coordinate_phrases_per_clause`	Coordinate phrases per clause (CP/C)	0 and above, `nan` without clauses	More phrasal coordination
`complex_nominals_per_clause`	Complex nominals per clause (CN/C)	0 and above, `nan` without clauses	Denser noun phrases
`modifiers_per_noun`	Mean number of the modifiers of a noun	0 and above, `nan` without nouns	Heavier noun phrases
`prep_chains_per_sent`	Chains of nested complements with a preposition per sentence	0 and above	More nested complements
`max_prep_chain_len`	Length of the longest prepositional chain	0, or 2 and above	Longer chains
`de_chains_per_sent`	Chains of `de` per sentence	0 and above	More chains
`max_de_chain_len`	Longest chain of `de` of the text	0 without chains, otherwise 2 and above	A longer chain
`participle_clauses_per_sent`	Participial clauses per sentence	0 and above	More participial clauses
`mean_participle_clause_len`	Mean length of a participial clause in words	2 and above, `nan` without participial clauses	Longer participial clauses
`gerund_clauses_per_sent`	Gerund clauses per sentence	0 and above	More gerund clauses
`mean_gerund_clause_len`	Mean length of a gerund clause in words	2 and above, `nan` without gerund clauses	Longer gerund clauses
`p_passive`	Share of the passive forms among the verb forms	0-1, `nan` without verb forms	More passive
`p_agentless_passive`	Share of the forms with no agent among the passive ones	0-1, `nan` without passive forms	More passive without an agent
`se_passives_per_sent`	Passives with `se` per sentence	0 and above	More passives with `se`
`impersonal_se_per_sent`	Impersonal sentences with `se` per sentence	0 and above	More impersonal sentences
`infinitives_per_sent`	Infinitives per sentence	0 and above	More infinitives
`negations_per_sent`	Words of negation per sentence	0 and above	More negation
`split_predicates_per_sent`	Split predicates per sentence	0 and above	More split predicates
`p_preverbal_subjects`	Share of the nominal subjects before their head	0-1, `nan` without nominal subjects	Fewer inverted subjects
`p_preverbal_objects`	Share of the nominal direct objects before their head	0-1, `nan` without nominal objects	More fronted objects
`pos_edit_distance`	Mean edit distance of the parts of speech of adjacent sentences over the longer one	0-1, `nan` for one sentence	Less alike sentences
`syntax_similarity`	Mean similarity of the dependency triples of adjacent sentences	0-1, `nan` for one sentence	More alike sentences
`noun_verb_ratio`	Ratio of the nouns to the verb forms	0 and above, `nan` without verb forms	A more nominal style

The functions of the cards are imported from ests.syntax_stats; those of the measures of the tree that esTS does not re-export are on the anyTS page.

Dependency distance¶

mean_dependency_distance · std_dependency_distance · max_dependency_distance · p_adjacent_dependencies

The dependency distance (Liu 2008) is the distance between a word and its head in positions of words, punctuation left out; the head of a sentence has none. mean_dependency_distance and std_dependency_distance are the mean and the standard deviation of the distances of the text, max_dependency_distance the mean of the longest distance of every sentence that has one, and p_adjacent_dependencies the share of the distances of 1. One-word sentences enter none of the metrics. The distances are those of calc_dependency_distances of anyts.syntax_stats.

Tree depth¶

tree_depth

The depth of the dependency tree of a sentence is the longest path from its head to a leaf, in relations between words, 0 for a sentence of one word; the metric is its mean over the sentences. The depth is that of calc_tree_depth of anyts.syntax_stats.

Leaves and subtrees¶

leaves_per_sent · subtrees_per_sent · nodes_per_leaf

A leaf is a word without dependent words (count_children), a subtree a word with them. The nodes per leaf are the words of a sentence over its leaves, averaged over the \(S\) sentences.

\[ \frac{1}{S} \sum_{s=1}^{S} \frac{\textrm{Number of words of } s}{\textrm{Number of leaves of } s} \]

The distribution of the words by number of dependents is the attribute c_children.

Words before the root¶

words_before_root · Coh-Metrix

The left embeddedness of Coh-Metrix (SYNLE): the number of the words of a sentence before its root, the main predicate, averaged over the sentences whose root is a word. The root of a sentence with a copula is its nominal part (she is a doctor**), as Universal Dependencies has it. In a language with a free word order the measure reads the order as much as the complexity.

Valency of the finite verbs¶

verb_valency

calc_valency(token, join_hyphens=False)

The valency of a token: the number of its dependent words, the relations cc, conj, parataxis and punct left out (anyts.constants.VALENCY_IGNORED_DEPS); with join_hyphens=True a hyphenated dependent counts once, so came in Some-one came has one dependent and not two.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

A finite verb is a word tagged VERB with VerbForm=Fin (is_finite_verb of anyts.utils). The Spanish models tag the participle of a compound tense or of the passive with ser (ha llegado, fue construida) as the verb and its auxiliary as AUX, so such predicates are left out.

Coordination chains¶

coordination_chains_per_sent · mean_coordination_chain_len

A coordination chain is a group of words linked by conj: manzanas, peras y ciruelas is one chain of three. A nested coordination and an enumeration split between several heads are one chain too; the length of a chain is the number of its words. The chains are those of calc_coordination_chains of anyts.syntax_stats.

Clauses¶

clauses_per_sent · mean_clause_len

is_clause_head(token)

Checks whether a word heads a clause: the head of the sentence or, by the form of its predicate (predicate_form), a word with ccomp, advcl, acl (a relative clause included), csubj or parataxis (anyts.constants.CLAUSE_DEPS). A finite predicate heads its clause (dijo que había llegado, estaba cantando, los regalos que te van a dar, sé que os habéis posado), also under xcomp, which the models give to some finite complements; a predicate in the form of a participle or a gerund heads no clause of that kind - such predicates are counted apart (llegando tarde, habiendo llegado tarde) - and neither does one in the form of an infinitive under acl (el deseo de irse, el miedo de ser robado). Under xcomp an infinitive heads a clause by is_infinitive_clause, and a predicative adjective (parece cansado) heads none. A parenthetical (parataxis) and a coordinated predicate (conj of the head of a clause) count only when they can be a predicate (is_predicate), so that parentheticals such as por ejemplo are no clauses; an infinitive the models attach as conj or csubj to the verb of its periphrasis (hay que salir, llega a convertirse) makes one predicate with it (is_infinitive_periphrasis). It is the is_clause_head of the anyTS core with the Spanish rules of is_infinitive_periphrasis and predicate_form.

is_infinitive_clause(token)

Checks whether an infinitive under xcomp heads a clause of its own: a predicate in the form of an infinitive that completes its head (quiere salir, le hizo reír, quiere ser médico, quiere ser elegido, cree haber ganado), unless the two make one predicate (is_infinitive_periphrasis: puede salir, parece dormir) or the word it completes heads no clause (is_clause_head): an infinitive in a gerund or participial phrase belongs to that phrase (intentando salir, destinada a disculpar).

is_infinitive_periphrasis(token)

Checks whether an infinitive makes one predicate with its head: an infinitive under xcomp of a verb of RAISING_VERBS, parecer (parece dormir), or a periphrasis. In a periphrasis the head is a verb of INFINITIVE_PERIPHRASES (NGLE, ch. 28) and the infinitive is linked to it by one of the words of that verb - a mark, case or cc before the infinitive, or the word right before it, the old spelling á read as a. The verbs are poder, soler and deber with no word (puede salir, suele salir), deber de, tener que, haber que, haber de, ir a, venir a, pasar a, volver a, tornar a, empezar a and por, comenzar a and por, principiar a, poner a, echar a, romper a, llegar a, alcanzar a, acabar de and por, terminar de and por, dejar de, cesar de, parar de, estar para and por. The lemma of the head is that of the model, or of ests.utils.lemmatize when the model gives none of these verbs (it can read a verb opening a sentence as a proper noun: Solía salir).

Parameter	Type	Default	Description
`token`	`Token`	-	Token

The mean length of a clause is the number of the words over the number of the clauses.

Subordinate clauses¶

subordinate_clauses_per_sent · p_complex_sents

is_subordinate_clause_head(token)

Checks whether a word heads a subordinate clause: a clause head (is_clause_head) whose predicate is no infinitive, with ccomp, advcl, acl or csubj (anyts.constants.SUBORDINATE_CLAUSE_DEPS) or under xcomp, or a coordinated predicate of a subordinate clause, by the relation of the first word up its chain of conjuncts. A clause under conj or parataxis with a subordinating conjunction (is_subordinating_mark) is subordinate too, with its conjuncts that have a subject of their own: the models attach a clause after y, pero or sino as a conjunct of the clause before it (sino porque le había cobrado afición, quiero salir y que vengas). An infinitive clause (quiere salir, salió para ayudar) is counted apart, among the infinitive clauses.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

p_complex_sents is the share of the sentences with at least one subordinate clause.

is_subordinating_mark(token)

Checks whether a word with the relation mark is a subordinating conjunction: one of SUBORDINATING_MARKS with the words of its fixed expression (que, si, como, cuando, porque, aunque, ya que, puesto que), but not que after sino, which coordinates (no lo dijo, sino que lo calló). The models parse para que, sin que and hasta que with a mark que of their own. Prepositions are left out: a and de before an infinitive mostly make a periphrasis (iba a preparar, ha de vivir). It is the hook is_subordinating_mark of the SyntaxStats of the anyTS core.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

CodeResult

from ests.syntax_stats import is_clause_head, is_subordinate_clause_head
from ests.utils import get_nlp

nlp = get_nlp()
doc = nlp("Él quiere salir, pero tiene que quedarse porque parece cansado.")
[token.text for token in doc if is_clause_head(token)]
[token.text for token in doc if is_subordinate_clause_head(token)]
doc = nlp("Se sentaba a sus pies, no por servicio, sino porque le había cobrado afición.")
[token.text for token in doc if is_subordinate_clause_head(token)]

['quiere', 'salir', 'tiene', 'parece']
['parece']
['cobrado']

Infinitive clauses¶

infinitive_clauses_per_sent

is_infinitive_clause_head(token)

Checks whether a word heads an infinitive clause: a clause head (is_clause_head) whose own predicate is an infinitive under xcomp, ccomp, advcl or csubj (quiere salir, le pidió salir, salió para ayudar, es difícil decirlo), an infinitive under conj or parataxis with a subordinating conjunction, or an infinitive conjunct of one; a finite conjunct of an infinitive (quiero salir y que vengas) is none. These clauses count among the clauses, but not among the subordinate ones: grammars take an infinitive complement for a part of a simple sentence, and the published correlations of the subordination were obtained for finite subordinate clauses. It is the is_infinitive_clause_head of the anyTS core over is_clause_head.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

CodeResult

from ests.syntax_stats import is_clause_head, is_infinitive_clause_head
from ests.utils import get_nlp

doc = get_nlp()("Quiere salir y volver, pero intentando salir se cayó.")
[token.text for token in doc if is_clause_head(token)]
[token.text for token in doc if is_infinitive_clause_head(token)]

['Quiere', 'salir', 'volver', 'cayó']
['salir', 'volver']

Production units of L2SCA¶

t_units_per_sent · mean_t_unit_len · clauses_per_t_unit · p_complex_t_units · verb_phrases_per_t_unit · coordinate_phrases_per_clause · complex_nominals_per_clause · Lu 2010

The indices of the L2 Syntactic Complexity Analyzer (L2SCA) that the other metrics do not give, on Universal Dependencies. MLC and C/S are mean_clause_len and clauses_per_sent; MLS and DC/C come from the counts, n_words / n_sents and n_subordinate_clauses / n_clauses.

A T-unit is a main clause with the clauses that depend on it: the head of a sentence and every clause head joined to it through conj or parataxis (anyts.constants.MAIN_CLAUSE_DEPS), so I came, and he left holds two; a conjunct of a subordinate clause is none. A conjunct without a subject of its own (I came and left) is a predicate sharing the T-unit of the clause it is coordinated to, as in L2SCA, unless its person, number or gender - those of its auxiliary or copula, its own without one - differ from those of that clause: in a language that drops its subjects the ending names another one (llegamos al puerto y nos esperabas). An existential there (expl of ClearNLP) is a subject. A conjunct or a parataxis with a subordinating mark (is_subordinating_mark) heads a subordinate or an infinitive clause, which is no T-unit, nor is a clause under it. A finite conjunct with a subject of its own of an infinitive clause, a predicate a model attaches to the infinitive, joins the clause of the first word above the infinitives that is none (he came to help and she stayed: two T-units). A T-unit is complex with a subordinate clause (is_subordinate_clause_head) whose nearest T-unit above is it.
A verb phrase is a predicate (is_predicate) that is no auxiliary or copula: a verb, an auxiliary used as a predicate (there is trouble), a word with a subject or a copula; an infinitive that makes one predicate with its head (is_infinitive_periphrasis), and its infinitive conjunct, is part of the verb phrase of the head. A participle or another non-finite form under amod or acl, or coordinated to one, without an auxiliary, a copula or a subject of its own is an attribute of its noun (the house built by workers), as in L2SCA, not a verb phrase.
A coordinate phrase is a coordination with a conjunct that is no clause head (the man and the woman, is big and nice) or a predicate without a subject of its own (came and left), counted once by its first conjunct; a coordination of clauses with their own subjects counts as clauses.
The clauses of C/T, CP/C and CN/C are those of L2SCA (n_l2sca_clauses): the clauses of is_clause_head without the predicates sharing a subject (came and left) and the infinitive clauses, as L2SCA counts finite clauses with a subject. C/S and MLC keep the clauses of the other metrics, so C/S runs higher and MLC lower than in L2SCA.
A complex nominal is a noun with a modifier of anyts.constants.COMPLEX_NOMINAL_DEPS (an adjective, a complement, a relative or participial clause, an apposition, a possessive), or a nominal clause, a clause head of anyts.constants.NOMINAL_CLAUSE_DEPS or a conjunct of one.

\[ \textrm{MLT} = \frac{\textrm{Words}}{\textrm{T-units}}, \quad \textrm{C/T} = \frac{\textrm{Clauses of L2SCA}}{\textrm{T-units}}, \quad \textrm{CT/T} = \frac{\textrm{Complex T-units}}{\textrm{T-units}}, \quad \textrm{CN/C} = \frac{\textrm{Complex nominals}}{\textrm{Clauses of L2SCA}} \]

The clauses and the subordinate clauses are those of the Spanish rules of the cards above, and a periphrasis is one verb phrase (puede venir). The models attach the infinitive of some periphrases as a conjunct of their verb (tiene que venir): it stays inside the verb phrase of the verb, but the pair counts as a coordinate phrase too.

Modifiers of a noun¶

modifiers_per_noun

count_noun_modifiers(token, join_hyphens=False)

Counts the modifiers of a noun phrase: the dependents with the relations amod, det, nmod, nummod, acl and their subtypes (anyts.constants.NOUN_MODIFIER_DEPS); conj and appos are no modifiers.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

The nouns are the words tagged NOUN or PROPN: la nueva casa de mi padre gives casa three modifiers, la, nueva and padre.

Prepositional chains¶

prep_chains_per_sent · max_prep_chain_len · Profiling-UD

A prepositional chain is two or more nested complements with a preposition, each hanging on the previous one: on the mat of the house of the mayor (length 3). The links are the words of the hook is_prep_complement, and the chains and their lengths are those of calc_modifier_chains; a chain may start at a complement of a verb (obl) or at a prepositional predicate with a copula (was in the house of the mayor), but not at a conjunct of a complement (in Madrid and in the centre of the city), and a predicate of a relative clause (the house that was by the river) is no link of the chain of its noun. The chains read Universal Dependencies alone: ClearNLP heads a phrase by its preposition (prep, pobj), so its parses give no chains.

The chains of de of esTS (de_chains_per_sent) are counted apart: their links are complements of de alone (la revisión de las cuentas del organismo), the marker of the nominal style of Spanish, and those of a prepositional chain complements with any preposition (sobre la mesa en la sala).

Chains of de¶

de_chains_per_sent · max_de_chain_len

calc_de_chains(tokens)

The lengths of the chains of de in the order of the words: the chains of calc_modifier_chains whose modifiers are complements with de; el aumento de la eficiencia del uso de los recursos is a chain of length 3, and a single complement (el uso del agua) is no chain.

is_de_modifier(token)

Checks whether a word is a complement introduced by de: a word with the relation nmod whose preposition (case) is de or its contraction del (DE_PREPOSITIONS): el uso del agua, but not la vuelta al campo.

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens (`calc_de_chains`)
`token`	`Token`	-	Token (`is_de_modifier`)

max_de_chain_len is the longest chain of the text, 0 without chains.

CodeResult

from ests.syntax_stats import calc_de_chains
from ests.utils import get_nlp

calc_de_chains(get_nlp()("El aumento de la eficiencia del uso de los recursos fue notable."))

[3]

Participial clauses¶

participle_clauses_per_sent · mean_participle_clause_len

The clauses headed by the words of the hook is_participle_clause; the length of a clause is the number of the words in the subtree of its head (subtree_len).

is_participle_clause(token)

Checks whether a word heads a participial clause: a participle (is_participle) that is not the head of the sentence, carries no auxiliary or copula (has_auxiliary) and has at least one dependent (calc_valency). la casa, construida por los obreros, se vendió has one, while ha pintado and fue construida are predicates. SyntaxStats counts the participial clauses by the same rule.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

CodeResult

from ests.syntax_stats import is_participle_clause, subtree_len
from ests.utils import get_nlp

doc = get_nlp()("La casa, construida por los obreros en 1900, fue vendida.")
[(token.text, subtree_len(token)) for token in doc if is_participle_clause(token)]

[('construida', 6)]

Gerund clauses¶

gerund_clauses_per_sent · mean_gerund_clause_len

is_gerund_clause(token)

Checks whether a word heads a gerund clause: a predicate in the form of a gerund (predicate_form), simple or compound (llegando tarde, habiendo llegado tarde, siendo elegido presidente), that is not the head of the sentence and has at least one dependent besides its auxiliaries and copulas (calc_valency). The periphrases are left out: one with a finite auxiliary (está cantando) is a finite predicate, and a gerund the models attach as xcomp or advcl to a verb of GERUND_PERIPHRASIS_VERBS - quedar, llevar, seguir, venir, ir, andar, continuar, acabar (sigue trabajando, acabó reconociendo) - heads no clause. The length of a clause is the number of the words in the subtree of the gerund (subtree_len).

Parameter	Type	Default	Description
`token`	`Token`	-	Token

CodeResult

from ests.syntax_stats import is_gerund_clause
from ests.utils import get_nlp

doc = get_nlp()("Llegando tarde, encontró la puerta cerrada y siguió trabajando.")
[token.text for token in doc if is_gerund_clause(token)]

['Llegando']

Passive¶

p_passive · p_agentless_passive

is_passive(token)

Checks whether a word is a passive verb form: a verb (VERB) that is a participle with the auxiliary ser (PASSIVE_AUX) as aux or cop - la casa fue construida, also when the models read the auxiliary as a copula, el proyecto es financiado - or that carries the se of the passive (expl:pass, SE_PASSIVE_DEP: se construyó la casa). A participle that modifies a noun (la casa construida por los obreros) is a participial clause instead.

is_agentless(token)

Checks whether a passive form has no agent: none of its dependents is one of is_agent - la casa fue construida, but not la casa fue construida por los obreros. A word that is no passive form (is_passive) gives False.

is_agent(token)

Checks whether a word is the agent of a passive: a complement whose preposition (case) is por (AGENT_PREPOSITION), whatever its relation, since the models have no obl:agent.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

The verb forms of p_passive are the words tagged VERB.

CodeResult

from ests.syntax_stats import is_agentless, is_passive
from ests.utils import get_nlp

doc = get_nlp()(
    "La casa fue construida por los obreros y el proyecto es financiado por el Estado. "
    "Se construyó la casa."
)
[(token.text, is_agentless(token)) for token in doc if is_passive(token)]

[('construida', False), ('financiado', False), ('construyó', True)]

Sentences with se¶

se_passives_per_sent · impersonal_se_per_sent

se_passives_per_sent counts the se of the passive (expl:pass, SE_PASSIVE_DEP: se construyó la casa) per sentence, impersonal_se_per_sent the se of an impersonal sentence (expl:impers, SE_IMPERSONAL_DEP: se vive bien, aquí se trabaja mucho), as the model labels them. A verb with the se of the passive is a passive form of p_passive too.

Warning

es_core_news_sm tells the three se of Spanish apart poorly - it rarely marks an impersonal se and reads se fue dando un portazo as a passive - so read se_passives_per_sent and impersonal_se_per_sent together rather than apart.

CodeResult

ss = SyntaxStats("Se construyó la casa en un año. Aquí se trabaja mucho.")
ss.se_passives_per_sent, ss.impersonal_se_per_sent

(0.5, 0.5)

Infinitives, negations and split predicates¶

infinitives_per_sent · negations_per_sent · split_predicates_per_sent

The infinitives (is_infinitive of anyts.utils), the words of negation of the hook is_negation and the split predicates of the hook find_split_predicates, per sentence.

is_negation(token)

Checks whether a word is a word of negation: a word with Polarity=Neg, which the models give to no alone, or one of NEGATION_WORDS in any case - no, nunca, jamás, nada, nadie, ninguno with its forms, tampoco; a conjunction (ni of ni... ni) is none. Every such word counts, so the negative concord of Spanish, where one negation is written twice (no vino nadie), gives two. SyntaxStats counts the negations by the same rule.

find_split_predicates(tokens)

is_light_verb(token)

is_split_predicate_noun(token, verb)

The split predicates, pairs of a verb and a noun in the order of the words, replace a verb with a light verb and a noun: hacer una revisión instead of revisar. A light verb (is_light_verb) is a VERB with a lemma of LIGHT_VERBS - hacer, dar, tomar, tener, poner, llevar, prestar, efectuar, realizar, proceder, proporcionar, ejercer. Its nominal part (is_split_predicate_noun) is a NOUN whose lemma is derived from a verb, by the rule of the verbal nouns (revisión, decisión, uso), or one of the nouns of SPLIT_PREDICATE_NOUNS - cabo, cargo, caso, comienzo, cuenta, efecto, fin, gala, hincapié, lugar, manifiesto, marcha, parte - with a verb verb of its fixed expression (llevar a cabo, tomar parte, poner de manifiesto, tener lugar), so poner en primer lugar la seguridad is none. A verb takes one nominal part at most, preferred by relation in the order compound, obj, nsubj (only with the se of the passive: se tomó la decisión), iobj, nmod, obl. A complement with a preposition is the nominal part only in a fixed expression (poner de manifiesto) or with a verb of PREPOSITIONAL_LIGHT_VERBS, proceder (se procedió a la votación); the agent of a passive never is. The pairs found are in the attribute split_predicates.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens (`find_split_predicates`)
`verb`	`Token`	-	Verb the noun depends on (`is_split_predicate_noun`)

CodeResult

from ests.syntax_stats import find_split_predicates, is_negation
from ests.utils import get_nlp

doc = get_nlp()(
    "Se procedió a la votación y el comité tomó parte en la revisión. "
    "No vino nadie, ni Juan ni María."
)
find_split_predicates(doc)
[token.text for token in doc if is_negation(token)]

[(procedió, votación), (tomó, parte)]
['No', 'nadie']

Order of the subjects and the objects¶

p_preverbal_subjects · p_preverbal_objects · Profiling-UD

The order of the subjects and of the direct objects in the manner of Profiling-UD: the shares of those before their head, the predicate. Only the nominal ones (NOUN, PROPN) count: the place of a pronoun, such as a clitic object, is set by the grammar. A subject is a word of nsubj (anyts.constants.NOMINAL_SUBJECT_DEPS), an object one of obj (anyts.constants.OBJECT_DEPS), with the relations of ClearNLP and the subtypes.

\[ \frac{\textrm{Nominal subjects before their head}}{\textrm{Nominal subjects}}, \quad \frac{\textrm{Nominal objects before their head}}{\textrm{Nominal objects}} \]

Spanish puts the subject after the verb more often than English does (llegó el tren, me gusta el café). A fronted object comes back as a clitic (el libro lo compré ayer), and the models take the noun for an indirect object, so the share of the nominal objects before their head stays near 0.

Similarity of adjacent sentences¶

pos_edit_distance · syntax_similarity · Coh-Metrix

How alike the structures of two adjacent sentences are, averaged over the \(S - 1\) pairs. pos_edit_distance is the edit distance of the parts of speech \(P_s\) of the words of the sentences (SYNMEDpos of Coh-Metrix) over the length \(n\) of the longer sentence of the pair, calc_edit_distance. syntax_similarity is the similarity of the multisets of the dependency triples \(T\) of the two sentences (dependency_triples, calc_multiset_similarity), in the manner of SYNSTRUT of Coh-Metrix, which compares the nodes of phrase-structure trees. The edit distance reads only the order of the parts of speech: a sentence with its object first and one with its subject first are alike when that order is the same (a distance of 0). The triples read no order of the words at all: any order of the words of one tree gives a similarity of 1. The order of the subject and the object is read by p_preverbal_subjects and p_preverbal_objects. The similarity of the triples grows with the length of the sentences.

\[ \frac{1}{S - 1} \sum_{s=1}^{S-1} \frac{\textrm{lev}(P_s, P_{s+1})}{\max(n_s, n_{s+1})}, \quad \frac{1}{S - 1} \sum_{s=1}^{S-1} \frac{|T_s \cap T_{s+1}|}{|T_s \cup T_{s+1}|} \]

Ratio of the nouns to the verbs¶

noun_verb_ratio

The number of the nouns n_nouns to the number of the verb forms n_verbs.

\[ \frac{\textrm{Number of nouns}}{\textrm{Number of verb forms}} \]

Methods¶

The methods get_stats, to_dataframe, print_stats and describe are those of the statistics interface of the anyTS core; the syntactic statistics have no scales, so describe gives None for the statistics of get_stats.

CodeResult

ss = SyntaxStats(text)
ss.get_stats()
ss.describe("tree_depth"), ss.describe("de_chains_per_sent")

{'mean_dependency_distance': 2.259259259259259,
 'std_dependency_distance': 2.1532506449997975,
 'max_dependency_distance': 9.0,
 'p_adjacent_dependencies': 0.48148148148148145,
 'tree_depth': 4.0,
 'leaves_per_sent': 8.0,
 'subtrees_per_sent': 6.5,
 'nodes_per_leaf': 1.8015873015873014,
 'words_before_root': 4.5,
 'verb_valency': 4.0,
 'coordination_chains_per_sent': 0.0,
 'mean_coordination_chain_len': nan,
 'clauses_per_sent': 1.5,
 'mean_clause_len': 9.666666666666666,
 'subordinate_clauses_per_sent': 0.5,
 'p_complex_sents': 0.5,
 'infinitive_clauses_per_sent': 0.0,
 't_units_per_sent': 1.0,
 'mean_t_unit_len': 14.5,
 'clauses_per_t_unit': 1.5,
 'p_complex_t_units': 0.5,
 'verb_phrases_per_t_unit': 1.5,
 'coordinate_phrases_per_clause': 0.0,
 'complex_nominals_per_clause': 1.3333333333333333,
 'modifiers_per_noun': 1.0,
 'prep_chains_per_sent': 0.5,
 'max_prep_chain_len': 2,
 'de_chains_per_sent': 0.5,
 'max_de_chain_len': 2,
 'participle_clauses_per_sent': 0.5,
 'mean_participle_clause_len': 3.0,
 'gerund_clauses_per_sent': 0.0,
 'mean_gerund_clause_len': nan,
 'p_passive': 0.6666666666666666,
 'p_agentless_passive': 0.5,
 'se_passives_per_sent': 0.5,
 'impersonal_se_per_sent': 0.0,
 'infinitives_per_sent': 0.0,
 'negations_per_sent': 0.5,
 'split_predicates_per_sent': 1.0,
 'p_preverbal_subjects': 0.5,
 'p_preverbal_objects': 0.0,
 'pos_edit_distance': 0.47058823529411764,
 'syntax_similarity': 0.45,
 'noun_verb_ratio': 3.0}
(None, None)

The functions work with spaCy tokens: the calc_* and find_* functions and dependency_triples take a sequence of tokens (Doc, Span or a list), the is_*, has_* and count_* functions a single token, and calc_edit_distance and calc_multiset_similarity lists of the items of sentences; the words of a sequence are the tokens of is_word of anyts.utils. The functions of the dependency tree come from the core, those of the Spanish constructions are esTS's own.

Words of a sequence¶

get_words(tokens, join_hyphens=False)

The words of a sequence of tokens in their order; with join_hyphens=True a word the tokenizer split at its hyphens is one word, given by its part that holds its relation, as with the dependents of get_children (governor in He is vice-governor), or by its first part when a hyphen holds it.

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

Base relation¶

base_dep(token)

The relation of a token without its subtype, which Universal Dependencies separates by a colon: nsubj:pass gives nsubj, expl:pass gives expl.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

Dependent words¶

get_children(token, join_hyphens=False)

The dependent words of a token, punctuation and whitespace left out; with join_hyphens=True, the dependents of the whole hyphenated word the token belongs to.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

subtree_len(token, join_hyphens=False)

The number of words in the subtree of a token: the token itself and all of its direct and indirect dependents; the arguments are those of get_children.

Verb forms¶

predicate_form(token)

The verb form of a predicate by the forms (VerbForm) of its auxiliaries and copulas (aux, cop): a finite auxiliary makes the predicate finite (había llegado, estaba cantando); otherwise the first auxiliary decides (de haber matado is an infinitive, siendo elegido a gerund), and without auxiliaries the word itself. A participle auxiliary counts as finite: sido and estado follow a form of haber, which the models may attach to another word (hubieras sido albañil). An auxiliary the model leaves without a form is read by its ending when it is a verb of AUXILIARY_VERBS (habéis llegado) - an infinitive in -r and a gerund in -ndo, enclitics after them (ENCLITICS), a finite form otherwise - and skipped when it is not (the old spelling á). It is the predicate_form of the anyTS core with the Spanish reading of an auxiliary without a form. The form is 'Fin', 'Inf', 'Ger' or 'Part', an empty string for a word with no verb form.

is_participle(token)

is_gerund(token)

has_auxiliary(token)

is_participle checks whether a token is a participle (VerbForm=Part) whatever its part of speech, since the models tag a participle that modifies a noun as an adjective (la casa pintada); is_gerund whether it is a gerund (VerbForm=Ger); has_auxiliary whether it carries an auxiliary or a copula (aux, cop).

Parameter	Type	Default	Description
`token`	`Token`	-	Token

CodeResult

from ests.syntax_stats import is_participle, predicate_form
from ests.utils import get_nlp

doc = get_nlp()(
    "Dijo que había llegado tarde. Lo acusan de haber matado al rey siendo elegido presidente."
)
[(token.text, predicate_form(token)) for token in doc if is_participle(token)]

[('llegado', 'Fin'), ('matado', 'Inf'), ('elegido', 'Ger')]

Predicate of a clause¶

is_predicate(token, join_hyphens=False)

Checks whether a word can be the predicate of its clause: a verb (VERB, AUX), a word with a subject of its own (nsubj, csubj and their subtypes, anyts.constants.SUBJECT_DEPS) or with a copula (cop), so the nominal predicate of he is tall and of a language that drops the subject passes and a parenthetical word does not.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

A value that is not a Token raises SourceTypeError.

Chains of modifiers¶

calc_modifier_chains(tokens, is_modifier, join_hyphens=False)

The lengths of the chains of nested modifiers, in the order of the words. A chain is two or more modifiers each hanging on the previous one, such as the complements of the increase of the efficiency of the use of resources (length 3); the check is_modifier of a language says what a modifier is (a complement with a preposition, a noun in the genitive). A chain starts at a modifier whose head is none, and its length is the number of modifiers in its longest branch. The links are those of get_children: punctuation is no link, and with join_hyphens a hyphenated word is one link.

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens
`is_modifier`	`Callable[[Token], bool]`	-	Whether a word is a modifier of the chain
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

A check that is not callable raises SourceTypeError.

Complements with a preposition¶

is_prep_complement(token, join_hyphens=False)

Checks whether a word is a complement with a preposition: a word with the relation nmod or obl or a subtype of them (anyts.constants.PREP_COMPLEMENT_DEPS), or a predicate with a copula or a subject of its own out of a relative clause (acl), such as a nominal predicate of a language that drops the copula of the present tense, that has a dependent of the relation case that is an adposition (ADP) or begins one of several words (fixed: in front of). A noun in a case without a preposition and a possessive marker are none.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

A value that is not a Token raises SourceTypeError.

Dependency triples¶

dependency_triples(tokens, join_hyphens=False)

The dependency triples of the words of a sentence (get_words), in their order: the part of speech of the head of a word - the nearest word above it through punctuation - its relation without a subtype (base_dep) and its own part of speech, such as ("NOUN", "amod", "ADJ") for a red house. The head of the sentence, and a word with no word above it, give an empty head and the relation root, whatever its label.

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

Tokens that are not a Doc, a Span or a list of Token objects raise SourceTypeError.

Edit distance and multiset similarity¶

calc_edit_distance(sents)

The mean edit distance of the items of adjacent sentences: the Levenshtein distance of every two - the fewest insertions, deletions and substitutions that turn one into the other - over the number of the items of the longer one, 0 for the same sequences and 1 for ones with no item in common; nan for fewer than two sentences.

Parameter	Type	Default	Description
`sents`	`list[list]`	-	Items of every sentence, such as its parts of speech

Sentences that are not a list of lists, or items that cannot be hashed, raise SourceTypeError.

calc_multiset_similarity(sents)

The mean similarity of the multisets of the items of adjacent sentences: the weighted Jaccard similarity of every two - the items they share, each as many times as the sentence with fewer of them has it, over the items of both, each as many times as the sentence with more of them has it - 1 for the same multisets and for two empty ones, 0 for disjoint ones; nan for fewer than two sentences. A mapping is read by its keys, as a set.

Parameter	Type	Default	Description
`sents`	`list[list]`	-	Items of every sentence, such as its dependency triples

Sentences that are not a list of collections, or items that cannot be hashed, raise SourceTypeError.