Syntactic statistics¶

SyntaxStats(source)

The syntactic statistics of a text by its dependency tree (Ivanov, Solnyshkina and Solovyev 2018): the measures of the tree and of the syntactic constructions. The data source is a Doc object of spaCy with a dependency parse and parts of speech. Punctuation, symbols and whitespace are no nodes of the tree, and the distances are counted in positions of words. The units of L2SCA, the prepositional chains and the similarity of sentences are computed on first use, so a subclass whose stats_desc does not name them does not pay for them.

Raises:

SourceTypeError - a source that is not a Doc;
SourceError - a source without a dependency parse (relations and sentence boundaries), without parts of speech, without lemmas when the class requires them, or without words.

The class extends the SyntaxStats of the anyTS core with the Russian layer. The Universal Dependencies relations come from the dependency parse of the spaCy model (ru_core_news_sm, ru_core_news_md, ru_core_news_lg), the parts of speech and features from the annotation of the pipeline: ruMorph in the recommended pipeline, otherwise the model; the quality of the statistics is bounded by the quality of the parse. The clauses, participial clauses, negations and split predicates follow the rules of Russian given in the cards of the metrics. The syntactic markers of officialese are the passive, participial clauses, genitive chains, split predicates and the ratio of nouns to verbs. The print_stats table and the error messages are in Russian (SYNTAX_STATS_DESC and CORE_MESSAGES of ruts.constants).

The cards of the metrics name the original features of the paper of Ivanov, Solnyshkina and Solovyev and their correlation with the grade of a school textbook.

CodeResult

from ruts import SyntaxStats, load_pipeline

nlp = load_pipeline()
text = "Дом, построенный рабочими в прошлом году, был продан. Хлопнув дверью, он ушел и сказал, что не придет."
ss = SyntaxStats(nlp(text))
ss.print_stats()

                       Статистика                        | Значение
--------------------------------------------------------------------
Средняя длина зависимости                                |   2.07
Стандартное отклонение длины зависимости                 |   1.61
Средний максимум длины зависимости в предложении         |   5.00
Доля смежных связей                                      |   0.53
Средняя глубина дерева зависимостей предложения          |   3.50
Листьев на предложение                                   |   4.50
Поддеревьев на предложение                               |   4.00
Среднее число узлов на лист в предложении                |   1.90
Среднее число слов до вершины предложения                |   5.00
Валентность финитных глаголов                            |   1.67
Сочинительных цепочек на предложение                     |   0.50
Средняя длина сочинительной цепочки                      |   2.00
Клауз на предложение                                     |   2.00
Средняя длина клаузы (слов)                              |   4.25
Придаточных клауз на предложение                         |   0.50
Доля предложений с придаточными                          |   0.50
Инфинитивных клауз на предложение                        |   0.00
T-единиц на предложение                                  |   1.00
Средняя длина T-единицы (слов)                           |   8.50
Клауз на T-единицу                                       |   1.50
Доля T-единиц с придаточными                             |   0.50
Глагольных групп на T-единицу                            |   2.50
Сочинённых групп на клаузу                               |   0.33
Сложных именных групп на клаузу                          |   1.00
Модификаторов на именную группу                          |   0.50
Предложных цепочек на предложение                        |   0.00
Максимальная длина предложной цепочки                    |    0
Цепочек родительных падежей на предложение               |   0.00
Максимальная длина цепочки родительных падежей           |    0
Именных цепочек на предложение                           |   0.00
Максимальная длина именной цепочки                       |    0
Причастных оборотов на предложение                       |   0.50
Средняя длина причастного оборота (слов)                 |   5.00
Деепричастных оборотов на предложение                    |   0.50
Средняя длина деепричастного оборота (слов)              |   2.00
Доля пассивных форм среди глаголов                       |   0.33
Доля безагентного пассива                                |   0.50
Инфинитивов на предложение                               |   0.00
Отрицаний на предложение                                 |   0.50
Расщеплённых сказуемых на предложение                    |   0.00
Доля именных подлежащих перед вершиной                   |   1.00
Доля именных дополнений перед вершиной                   |   0.00
Редакционное расстояние частей речи соседних предложений |   0.78
Синтаксическая схожесть соседних предложений             |   0.06
Отношение существительных к глаголам                     |   0.67

Parameters¶

Parameter	Type	Default	Description
`source`	`Doc`	-	Data source: a `Doc` with a dependency parse

Attributes¶

Attribute	Type	Description
`sents`	`tuple[tuple[Token]]`	Words of every sentence with words, on a copy of the `Doc` without its tensor and user data
`n_sents`, `n_words`	`int`	Numbers of the sentences with words and of the words
`n_leaves`, `n_subtrees`	`int`	Numbers of the words without and with dependent words
`n_coordination_chains`	`int`	Number of the coordination chains
`n_clauses`, `n_subordinate_clauses`, `n_complex_sents`	`int`	Numbers of the clauses, of the subordinate clauses and of the sentences with one
`n_infinitive_clauses`	`int`	Number of the infinitive clauses
`n_t_units`, `n_complex_t_units`	`int`	Numbers of the T-units and of those with a subordinate clause
`n_l2sca_clauses`	`int`	Number of the clauses of L2SCA, without the predicates sharing a subject and the infinitive clauses
`n_verb_phrases`, `n_coordinate_phrases`, `n_complex_nominals`	`int`	Numbers of the verb phrases, of the coordinate phrases and of the complex nominals
`n_nouns`, `n_verbs`	`int`	Numbers of the nouns (`NOUN`, `PROPN`) and of the verb forms (`VERB`)
`n_prep_chains`	`int`	Number of the prepositional chains
`n_participle_clauses`	`int`	Number of the participial clauses
`n_passive`, `n_agentless_passive`	`int`	Numbers of the passive forms and of those with no agent
`n_infinitives`, `n_negations`	`int`	Numbers of the infinitives and of the words of negation
`n_split_predicates`, `split_predicates`	`int`, `tuple[str]`	Number of the split predicates and the predicates, the verb and the noun
`n_nominal_subjects`, `n_preverbal_subjects`	`int`	Numbers of the nominal subjects and of those before their head
`n_nominal_objects`, `n_preverbal_objects`	`int`	Numbers of the nominal direct objects and of those before their head
`c_children`	`dict[int, int]`	Number of the words by number of dependent words
`c_deps`	`dict[str, int]`	Number of the words by relation

The attributes of the constructions of ruTS are n_genitive_chains, n_nominal_chains and n_converb_clauses (int): the numbers of genitive chains, of nominal chains and of adverbial participle clauses.

CodeResult

ss.n_clauses, ss.n_nouns, ss.n_verbs
ss.c_children
ss.c_deps
ss.sents[0]

(4, 4, 6)
{0: 9, 1: 2, 2: 5, 3: 1}
{'ROOT': 2,
 'acl': 1,
 'advcl': 1,
 'advmod': 1,
 'amod': 1,
 'aux:pass': 1,
 'case': 1,
 'cc': 1,
 'ccomp': 1,
 'conj': 1,
 'mark': 1,
 'nsubj': 1,
 'nsubj:pass': 1,
 'obj': 1,
 'obl': 1,
 'obl:agent': 1}
(Дом, построенный, рабочими, в, прошлом, году, был, продан)

Metrics¶

Every metric is an attribute of the object, computed when it is created - the units of L2SCA, the prepositional chains and the similarity of sentences on first use - and get_stats gives them all. The measures of a sentence (the longest dependency, the depth, the nodes per leaf) are averaged over the sentences; the constructions are given per sentence.

Metric	Measures	Range	Higher means
`mean_dependency_distance`	Mean distance between a word and its head	1 and above, `nan` without dependencies	Harder to process
`std_dependency_distance`	Standard deviation of the dependency distance	0 and above, `nan` without dependencies	Less even distances
`max_dependency_distance`	Mean of the longest dependencies of the sentences that have dependencies	1 and above, `nan` without dependencies	Longer longest dependencies
`p_adjacent_dependencies`	Share of the dependencies of length 1	0-1, `nan` without dependencies	More links between neighbours
`tree_depth`	Mean depth of the tree of a sentence	0 and above	Deeper embedding
`leaves_per_sent`	Leaves per sentence	0 and above	Longer or flatter sentences
`subtrees_per_sent`	Subtrees per sentence	0 and above	More phrases in a sentence
`nodes_per_leaf`	Mean over the sentences of the words per leaf	1 and above, `nan` for a sentence without leaves	Longer branches
`words_before_root`	Mean number of the words before the root of a sentence	0 and above, `nan` without a root among the words	A later main predicate
`verb_valency`	Mean number of the dependents of a finite verb	0 and above, `nan` without finite verbs	More dependents of a verb
`coordination_chains_per_sent`	Coordination chains per sentence	0 and above	More coordination
`mean_coordination_chain_len`	Mean length of a coordination chain in words	2 and above, `nan` without chains	Longer enumerations
`clauses_per_sent`	Clauses per sentence	0 and above	More clauses in a sentence
`mean_clause_len`	Mean length of a clause in words	1 and above, `nan` without clauses	Longer clauses
`subordinate_clauses_per_sent`	Subordinate clauses per sentence	0 and above	More subordination
`p_complex_sents`	Share of the sentences with a subordinate clause	0-1	More complex sentences
`infinitive_clauses_per_sent`	Infinitive clauses per sentence (wants to leave, came to help), which are not subordinate	0 and above	More infinitive clauses
`t_units_per_sent`	T-units per sentence (T/S)	0 and above	More main clauses in a sentence
`mean_t_unit_len`	Mean length of a T-unit in words (MLT)	1 and above, `nan` without T-units	Longer T-units
`clauses_per_t_unit`	Clauses per T-unit (C/T)	1 and above, `nan` without T-units	More subordination
`p_complex_t_units`	Share of the T-units with a subordinate clause (CT/T)	0-1, `nan` without T-units	More complex T-units
`verb_phrases_per_t_unit`	Verb phrases per T-unit (VP/T)	0 and above, `nan` without T-units	More verb phrases
`coordinate_phrases_per_clause`	Coordinate phrases per clause (CP/C)	0 and above, `nan` without clauses	More phrasal coordination
`complex_nominals_per_clause`	Complex nominals per clause (CN/C)	0 and above, `nan` without clauses	Denser noun phrases
`modifiers_per_noun`	Mean number of the modifiers of a noun	0 and above, `nan` without nouns	Heavier noun phrases
`prep_chains_per_sent`	Chains of nested complements with a preposition per sentence	0 and above	More nested complements
`max_prep_chain_len`	Length of the longest prepositional chain	0, or 2 and above	Longer chains
`participle_clauses_per_sent`	Participial clauses per sentence	0 and above	More participial clauses
`mean_participle_clause_len`	Mean length of a participial clause in words	1 and above, `nan` without participial clauses	Longer participial clauses
`p_passive`	Share of the passive forms among the verb forms	0-1, `nan` without verb forms	More passive
`p_agentless_passive`	Share of the forms with no agent among the passive ones	0-1, `nan` without passive forms	More passive without an agent
`infinitives_per_sent`	Infinitives per sentence	0 and above	More infinitives
`negations_per_sent`	Words of negation per sentence	0 and above	More negation
`split_predicates_per_sent`	Split predicates per sentence	0 and above	More split predicates
`p_preverbal_subjects`	Share of the nominal subjects before their head	0-1, `nan` without nominal subjects	Fewer inverted subjects
`p_preverbal_objects`	Share of the nominal direct objects before their head	0-1, `nan` without nominal objects	More fronted objects
`pos_edit_distance`	Mean edit distance of the parts of speech of adjacent sentences over the longer one	0-1, `nan` for one sentence	Less alike sentences
`syntax_similarity`	Mean similarity of the dependency triples of adjacent sentences	0-1, `nan` for one sentence	More alike sentences
`noun_verb_ratio`	Ratio of the nouns to the verb forms	0 and above, `nan` without verb forms	A more nominal style

The metrics of the constructions of ruTS:

Metric	Measures	Range	Higher means
`genitive_chains_per_sent`	Genitive chains per sentence	0 and above	More chains
`max_genitive_chain_len`	Longest genitive chain of the text	0 without chains, otherwise 2 and above	A longer chain
`nominal_chains_per_sent`	Nominal chains (genitive and prepositional links) per sentence	0 and above	More chains
`max_nominal_chain_len`	Longest nominal chain of the text	0 without chains, otherwise 2 and above	A longer chain
`converb_clauses_per_sent`	Adverbial participle clauses per sentence	0 and above	More adverbial participle clauses
`mean_converb_clause_len`	Mean length of an adverbial participle clause in words	2 and above, `nan` without adverbial participle clauses	Longer adverbial participle clauses

Dependency distance¶

mean_dependency_distance · std_dependency_distance · max_dependency_distance · p_adjacent_dependencies

calc_dependency_distances(tokens, join_hyphens=False)

The dependency distances (Liu 2008): the distance between a word and its head in positions of words, punctuation left out, in the order of the words. The head of a sentence and a word whose head lies outside the sequence are skipped.

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one position

The mean dependency distance grows with sentence length, and the longest one is the best of the tree features at the level of a single sentence.

CodeResult

from ruts import load_pipeline
from ruts.syntax_stats import calc_dependency_distances

nlp = load_pipeline()
doc = nlp("Повышение эффективности использования ресурсов предприятия обсуждалось на совещании.")
calc_dependency_distances(doc)

[5, 1, 1, 1, 1, 1, 2]

Tree depth¶

tree_depth

calc_tree_depth(tokens, join_hyphens=False)

The depth of the dependency tree: the longest path from the head of a sentence to a leaf, in relations between words; the maximum for several sentences, 0 for a sentence of one word. With join_hyphens a hyphenated word is one word, and the words that hang on any of its parts are one level below it.

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

The depth is LONGEST_PATH of the paper of Ivanov et al. (r = 0.84).

Leaves and subtrees¶

leaves_per_sent · subtrees_per_sent · nodes_per_leaf

A leaf is a word without dependent words (count_children), a subtree a word with them. The nodes per leaf are the words of a sentence over its leaves, averaged over the \(S\) sentences.

\[ \frac{1}{S} \sum_{s=1}^{S} \frac{\textrm{Number of words of } s}{\textrm{Number of leaves of } s} \]

The features of the paper: LEAVES_NUMBER (r = 0.84), PATH_NUMBER (r = 0.87) and AVERAGE_PATH (r = 0.84).

Words before the root¶

words_before_root · Coh-Metrix

The left embeddedness of Coh-Metrix (SYNLE): the number of the words of a sentence before its root, the main predicate, averaged over the sentences whose root is a word. The root of a sentence with a copula is its nominal part (she is a doctor), as Universal Dependencies has it. In a language with a free word order the measure reads the order as much as the complexity.

Valency of the finite verbs¶

verb_valency

calc_valency(token, join_hyphens=False)

The valency of a token: the number of its dependent words, the relations cc, conj, parataxis and punct left out (anyts.constants.VALENCY_IGNORED_DEPS); with join_hyphens=True a hyphenated dependent counts once, so came in Some-one came has one dependent and not two.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

A finite verb is one of is_finite_verb; the valency is VERBS_DEP of the paper (r = 0.43).

Coordination chains¶

coordination_chains_per_sent · mean_coordination_chain_len

calc_coordination_chains(tokens, join_hyphens=False)

The lengths of the coordination chains, in the order of their first words. A chain is a group of words linked by conj, whichever word each conjunct hangs on: Universal Dependencies attaches every conjunct to the first one (pears → apples, plums → apples), ClearNLP to the previous one (pears → apples, plums → pears); both give one chain of three for apples, pears and plums. A nested coordination (cats and dogs, or birds) and an enumeration split between several heads are one chain too. The length of a chain is the number of its words; a conjunct whose head lies outside the sequence is not linked to it. With join_hyphens a hyphenated word is one conjunct, linked by the conj of any of its parts (member and vice-governor is a chain of two).

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

The features of the paper: SOCHIN_NUMBER (r = 0.93), the number of chains per sentence, and AVERAGE_SOCHIN_LENGTH (r = 0.87), their mean length.

Clauses¶

clauses_per_sent · mean_clause_len

is_clause_head(token)

Checks whether a word heads a clause: the head of the sentence or a word with ccomp, advcl, acl, csubj, parataxis and their subtypes (anyts.constants.CLAUSE_DEPS), except full participles and adverbial participles, whose clauses are counted apart, and infinitives under a noun («желание уйти»). Under xcomp only an infinitive of is_infinitive_clause heads a clause. An infinitive subject (csubj) of a modal word («нужно уйти») heads no clause of its own. A parenthetical construction (parataxis) and a coordinated predicate (a conj relation from the clause head) form their own clause only if they can be a predicate (is_predicate): «Он сказал: „Уходи“» is a clause, while parenthetical words («например»), which spaCy models also attach as parataxis, are not.

is_infinitive_clause(token)

Checks whether an infinitive under xcomp heads a clause of its own («хочет уйти»): it does not when it makes one predicate with its head (is_infinitive_periphrasis) or stands in a participial or adverbial participle clause («решивший уйти», «желая уйти»).

is_infinitive_periphrasis(token)

Checks whether an infinitive makes one predicate with its head: a modal or phasal verb or a modal word of INFINITIVE_PERIPHRASES («может уйти», «начал читать», «нужно уйти»), the short form of a modal adjective of MODAL_SHORT_ADJECTIVES («готов уйти») or the hortative «давай», «давайте» of HORTATIVE_WORDS without a negation («давайте играть», but not «не давайте детям играть»).

Parameter	Type	Default	Description
`token`	`Token`	-	Token

Subordinate clauses¶

subordinate_clauses_per_sent · p_complex_sents

is_subordinate_clause_head(token)

Checks whether a word heads a subordinate clause: a clause head (is_clause_head) whose predicate is no infinitive under ccomp, advcl, acl, csubj and their subtypes, a coordinated predicate of a subordinate clause, or a clause under conj or parataxis with a subordinating conjunction (is_subordinating_mark: «ушел, ибо устал») with its conjoined predicates that have a subject of their own, but not those sharing its subject.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

is_subordinating_mark(token)

Checks whether a conjunction (relation mark) of a conjunct or a parataxis is a subordinating one: a conjunction of SUBORDINATING_MARKS with the words of a compound conjunction («потому что») or «как» in «после того как», «с тех пор как». Not subordinating are a conjunction after «не то» and «всё равно» («не то что»), a conjunction of a predicate with a correlative «то» after it that is not of a nested «если..., то» («когда пришел, то...»), and «что», «чем» used as a pronoun or a particle: «что ж», «что за» and at the start of a replica that ends with a question or an exclamation («А что вы сделаете?»).

Parameter	Type	Default	Description
`token`	`Token`	-	Token

The features of the paper: PODCHIN_RATE (r = 0.64), the subordinate clauses per sentence, and PODCHIN_NUMBER (r = 0.62), the share of the sentences with at least one subordinate clause.

Infinitive clauses¶

infinitive_clauses_per_sent

is_infinitive_clause_head(token)

Checks whether a word heads an infinitive clause: an infinitive under xcomp that heads a clause of its own (is_infinitive_clause: «хочет уйти»), an infinitive clause head under ccomp, advcl, acl or csubj («пришел, чтобы помочь»), or an infinitive coordinated with it.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

CodeResult

from ruts import load_pipeline
from ruts.syntax_stats import is_clause_head, is_infinitive_clause_head, is_subordinate_clause_head

nlp = load_pipeline()
doc = nlp("Он хочет уйти, но может остаться, если его попросят.")
[token.text for token in doc if is_clause_head(token)]
[token.text for token in doc if is_subordinate_clause_head(token)]
[token.text for token in doc if is_infinitive_clause_head(token)]

['хочет', 'уйти', 'может', 'попросят']
['попросят']
['уйти']

Production units of L2SCA¶

t_units_per_sent · mean_t_unit_len · clauses_per_t_unit · p_complex_t_units · verb_phrases_per_t_unit · coordinate_phrases_per_clause · complex_nominals_per_clause · Lu 2010

The indices of the L2 Syntactic Complexity Analyzer (L2SCA) that the other metrics do not give, on Universal Dependencies. MLC and C/S are mean_clause_len and clauses_per_sent; MLS and DC/C come from the counts, n_words / n_sents and n_subordinate_clauses / n_clauses.

A T-unit is a main clause with the clauses that depend on it: the head of a sentence and every clause head joined to it through conj or parataxis (anyts.constants.MAIN_CLAUSE_DEPS), so I came, and he left holds two; a conjunct of a subordinate clause is none. A conjunct without a subject of its own (I came and left) is a predicate sharing the T-unit of the clause it is coordinated to, as in L2SCA, unless its person, number or gender - those of its auxiliary or copula, its own without one - differ from those of that clause: in a language that drops its subjects the ending names another one (llegamos al puerto y nos esperabas). An existential there (expl of ClearNLP) is a subject. A conjunct or a parataxis with a subordinating mark (is_subordinating_mark) heads a subordinate or an infinitive clause, which is no T-unit, nor is a clause under it. A finite conjunct with a subject of its own of an infinitive clause, a predicate a model attaches to the infinitive, joins the clause of the first word above the infinitives that is none (he came to help and she stayed: two T-units). A T-unit is complex with a subordinate clause (is_subordinate_clause_head) whose nearest T-unit above is it.
A verb phrase is a predicate (is_predicate) that is no auxiliary or copula: a verb, an auxiliary used as a predicate (there is trouble), a word with a subject or a copula; an infinitive that makes one predicate with its head (is_infinitive_periphrasis), and its infinitive conjunct, is part of the verb phrase of the head. A participle or another non-finite form under amod or acl, or coordinated to one, without an auxiliary, a copula or a subject of its own is an attribute of its noun (the house built by workers), as in L2SCA, not a verb phrase.
A coordinate phrase is a coordination with a conjunct that is no clause head (the man and the woman, is big and nice) or a predicate without a subject of its own (came and left), counted once by its first conjunct; a coordination of clauses with their own subjects counts as clauses.
The clauses of C/T, CP/C and CN/C are those of L2SCA (n_l2sca_clauses): the clauses of is_clause_head without the predicates sharing a subject (came and left) and the infinitive clauses, as L2SCA counts finite clauses with a subject. The head of a sentence heads a clause and a T-unit even without a predicate (Yes., a heading, the name of a speaker), as a fragment does in L2SCA. C/S and MLC keep the clauses of the other metrics, so C/S runs higher and MLC lower than in L2SCA.
A complex nominal is a noun with a modifier of anyts.constants.COMPLEX_NOMINAL_DEPS (an adjective, a complement, a relative or participial clause, an apposition, a possessive), or a nominal clause, a clause head of anyts.constants.NOMINAL_CLAUSE_DEPS or a conjunct of one.

\[ \textrm{MLT} = \frac{\textrm{Words}}{\textrm{T-units}}, \quad \textrm{C/T} = \frac{\textrm{Clauses of L2SCA}}{\textrm{T-units}}, \quad \textrm{CT/T} = \frac{\textrm{Complex T-units}}{\textrm{T-units}}, \quad \textrm{CN/C} = \frac{\textrm{Complex nominals}}{\textrm{Clauses of L2SCA}} \]

Modifiers of a noun¶

modifiers_per_noun

count_noun_modifiers(token, join_hyphens=False)

Counts the modifiers of a noun phrase: the dependents with the relations amod, det, nmod, nummod, acl and their subtypes (anyts.constants.NOUN_MODIFIER_DEPS); conj and appos are no modifiers.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

The number of modifiers is NOUNS_DEP of the paper (r = 0.88).

Prepositional chains¶

prep_chains_per_sent · max_prep_chain_len · Profiling-UD

A prepositional chain is two or more nested complements with a preposition, each hanging on the previous one: on the mat of the house of the mayor (length 3). The links are the words of the hook is_prep_complement, and the chains and their lengths are those of calc_modifier_chains; a chain may start at a complement of a verb (obl) or at a prepositional predicate with a copula (was in the house of the mayor), but not at a conjunct of a complement (in Madrid and in the centre of the city), and a predicate of a relative clause (the house that was by the river) is no link of the chain of its noun. The chains read Universal Dependencies alone: ClearNLP heads a phrase by its preposition (prep, pobj), so its parses give no chains.

ruTS counts the prepositionless genitive chains apart (genitive_chains_per_sent).

Genitive chains¶

genitive_chains_per_sent · max_genitive_chain_len

calc_genitive_chains(tokens)

The lengths of the genitive chains in the order of the words: the chains of calc_modifier_chains whose modifiers are prepositionless genitive modifiers; «повышение эффективности использования ресурсов» is a chain of length 3.

is_genitive_modifier(token)

Checks whether a word is a prepositionless genitive modifier: a word with the nmod relation in the genitive case without a dependent preposition (case), «дом отца», but not «дом у дороги».

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens (`calc_genitive_chains`)
`token`	`Token`	-	Token (`is_genitive_modifier`)

CodeResult

from ruts import load_pipeline
from ruts.syntax_stats import calc_genitive_chains

nlp = load_pipeline()
calc_genitive_chains(
    nlp("Повышение эффективности использования ресурсов предприятия обсуждалось на совещании.")
)

[4]

Nominal chains¶

nominal_chains_per_sent · max_nominal_chain_len

calc_nominal_chains(tokens)

The lengths of the nominal chains in the order of the words: the chains of calc_modifier_chains whose links are genitive modifiers and complements with a preposition in any order; «рост цен на нефть» is a chain of length 2. A prepositional phrase is a link itself, so with a verb the same phrase gives a longer chain than as a subject: «сидел во втором ряду кресел» has length 2, «второй ряд кресел пустовал» no chain. Genitive chains and prepositional chains are special cases of a nominal one; on TextsByGrade the nominal chains go with the grade of a text more closely (ρ = 0.76 against 0.65 and 0.63).

is_nominal_chain_link(token)

Checks whether a word is a link of a nominal chain: a prepositionless genitive modifier (is_genitive_modifier) or a complement with a preposition (is_prep_complement).

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens (`calc_nominal_chains`)
`token`	`Token`	-	Token (`is_nominal_chain_link`)

CodeResult

from ruts import load_pipeline
from ruts.syntax_stats import calc_nominal_chains

nlp = load_pipeline()
calc_nominal_chains(
    nlp(
        "Рост цен на нефть замедлился, и о повышении эффективности работы отдела больше не говорили."
    )
)

[2, 4]

Participial clauses¶

participle_clauses_per_sent · mean_participle_clause_len

The clauses headed by the words of the hook is_participle_clause; the length of a clause is the number of the words in the subtree of its head (subtree_len).

is_participle_clause(token)

Checks whether a word heads a participial clause: a full participle (is_participle) with at least one dependent word, the coordination and parenthetical relations left out. The features of the paper are PRICH_RATE (r = 0.91) and PRICH_V.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

Adverbial participle clauses¶

converb_clauses_per_sent · mean_converb_clause_len

is_converb_clause(token)

Checks whether a word heads an adverbial participle clause: an adverbial participle (is_converb) with at least one dependent word, the coordination and parenthetical relations left out. The length of a clause is the number of the words in the subtree of the adverbial participle (subtree_len). The features of the paper are DEEPRICH_RATE (r = 0.44) and DEEPRICH_V.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

Passive¶

p_passive · p_agentless_passive

is_passive(token, join_hyphens=False)

Checks whether a word is a passive verb form: a verb with Voice=Pass or a dependent nsubj:pass, csubj:pass, aux:pass, or nsubjpass, csubjpass, auxpass of ClearNLP (PASSIVE_DEPS).

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

A value that is not a Token raises SourceTypeError.

has_agent(token, join_hyphens=False)

Checks whether a word has an agent: a dependent obl:agent, or agent of ClearNLP (AGENT_DEPS). The arguments and the error are those of is_passive.

is_agentless(token, join_hyphens=False)

Checks whether a word is a passive form (is_passive) without an agent (has_agent), with the same arguments and error.

In Russian the passive is a passive participle or the reflexive passive («обсуждалось»), and an agentless form is «дом построен», but not «дом построен рабочими».

Infinitives, negations and split predicates¶

infinitives_per_sent · negations_per_sent · split_predicates_per_sent

The infinitives (is_infinitive of anyts.utils), the words of negation of the hook is_negation and the split predicates of the hook find_split_predicates, per sentence.

is_negation(token)

Checks whether a word is a negative particle: a particle (PART) with the feature Polarity=Neg or the particle «не», «ни» (NEGATION_PARTICLES); the conjunction «ни» of «ни… ни» is no negation. The negations are NEG of the paper (r = 0.70).

find_split_predicates(tokens)

is_light_verb(token)

is_reflexive(token)

is_split_predicate_noun(token, verb=None)

The split predicates are pairs of a verb and a noun in the order of the words: a light verb (is_light_verb) with a lemma of LIGHT_VERBS (осуществлять, проводить, оказывать and reflexive passives such as проводиться) with a nominal part (is_split_predicate_noun), a noun with a verbal lemma (ruts.utils.is_verbal_noun) or a lemma of SPLIT_PREDICATE_NOUNS (роль, работа, помощь) with a verb verb of its own phrase (оказать помощь, but not получить работу). At most one nominal part is taken per verb in the order of preference obj, nsubj:pass, nsubj (only for a reflexive or passive verb, is_reflexive), iobj, nmod, obl; prepositional adjuncts and the passive agent (obl:agent, instrumental obl) are not counted.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens (`find_split_predicates`)
`verb`	`Token`	`None`	Light verb (`is_split_predicate_noun`); without it the phrase is not checked

CodeResult

from ruts import load_pipeline
from ruts.syntax_stats import find_split_predicates, is_negation

nlp = load_pipeline()
doc = nlp("Проверка проводится ежегодно, но решение не принято.")
find_split_predicates(doc)
[token.text for token in doc if is_negation(token)]

[(проводится, Проверка), (принято, решение)]
['не']

Order of the subjects and the objects¶

p_preverbal_subjects · p_preverbal_objects · Profiling-UD

The order of the subjects and of the direct objects in the manner of Profiling-UD: the shares of those before their head, the predicate. Only the nominal ones (NOUN, PROPN) count: the place of a pronoun, such as a clitic object, is set by the grammar. A subject is a word of nsubj (anyts.constants.NOMINAL_SUBJECT_DEPS), an object one of obj (anyts.constants.OBJECT_DEPS), with the relations of ClearNLP and the subtypes.

\[ \frac{\textrm{Nominal subjects before their head}}{\textrm{Nominal subjects}}, \quad \frac{\textrm{Nominal objects before their head}}{\textrm{Nominal objects}} \]

Similarity of adjacent sentences¶

pos_edit_distance · syntax_similarity · Coh-Metrix

How alike the structures of two adjacent sentences are, averaged over the \(S - 1\) pairs. pos_edit_distance is the edit distance of the parts of speech \(P_s\) of the words of the sentences (SYNMEDpos of Coh-Metrix) over the length \(n\) of the longer sentence of the pair, calc_edit_distance. syntax_similarity is the similarity of the multisets of the dependency triples \(T\) of the two sentences (dependency_triples, calc_multiset_similarity), in the manner of SYNSTRUT of Coh-Metrix, which compares the nodes of phrase-structure trees. The edit distance reads only the order of the parts of speech: a sentence with its object first and one with its subject first are alike when that order is the same (a distance of 0). The triples read no order of the words at all: any order of the words of one tree gives a similarity of 1. The order of the subject and the object is read by p_preverbal_subjects and p_preverbal_objects. The similarity of the triples grows with the length of the sentences.

\[ \frac{1}{S - 1} \sum_{s=1}^{S-1} \frac{\textrm{lev}(P_s, P_{s+1})}{\max(n_s, n_{s+1})}, \quad \frac{1}{S - 1} \sum_{s=1}^{S-1} \frac{|T_s \cap T_{s+1}|}{|T_s \cup T_{s+1}|} \]

Ratio of the nouns to the verbs¶

noun_verb_ratio

The number of the nouns n_nouns to the number of the verb forms n_verbs.

\[ \frac{\textrm{Number of nouns}}{\textrm{Number of verb forms}} \]

Methods¶

The methods get_stats, to_dataframe, print_stats and describe are those of the statistics interface of the anyTS core; describe gives None for the statistics of get_stats: the syntactic statistics have no scales.

CodeResult

ss.get_stats()

{'mean_dependency_distance': 2.066666666666667,
 'std_dependency_distance': 1.6110727964792764,
 'max_dependency_distance': 5.0,
 'p_adjacent_dependencies': 0.5333333333333333,
 'tree_depth': 3.5,
 'leaves_per_sent': 4.5,
 'subtrees_per_sent': 4.0,
 'nodes_per_leaf': 1.9,
 'words_before_root': 5.0,
 'verb_valency': 1.6666666666666667,
 'coordination_chains_per_sent': 0.5,
 'mean_coordination_chain_len': 2.0,
 'clauses_per_sent': 2.0,
 'mean_clause_len': 4.25,
 'subordinate_clauses_per_sent': 0.5,
 'p_complex_sents': 0.5,
 'infinitive_clauses_per_sent': 0.0,
 't_units_per_sent': 1.0,
 'mean_t_unit_len': 8.5,
 'clauses_per_t_unit': 1.5,
 'p_complex_t_units': 0.5,
 'verb_phrases_per_t_unit': 2.5,
 'coordinate_phrases_per_clause': 0.3333333333333333,
 'complex_nominals_per_clause': 1.0,
 'modifiers_per_noun': 0.5,
 'prep_chains_per_sent': 0.0,
 'max_prep_chain_len': 0,
 'genitive_chains_per_sent': 0.0,
 'max_genitive_chain_len': 0,
 'nominal_chains_per_sent': 0.0,
 'max_nominal_chain_len': 0,
 'participle_clauses_per_sent': 0.5,
 'mean_participle_clause_len': 5.0,
 'converb_clauses_per_sent': 0.5,
 'mean_converb_clause_len': 2.0,
 'p_passive': 0.3333333333333333,
 'p_agentless_passive': 0.5,
 'infinitives_per_sent': 0.0,
 'negations_per_sent': 0.5,
 'split_predicates_per_sent': 0.0,
 'p_preverbal_subjects': 1.0,
 'p_preverbal_objects': 0.0,
 'pos_edit_distance': 0.7777777777777778,
 'syntax_similarity': 0.0625,
 'noun_verb_ratio': 0.6666666666666666}

The words of a sequence of tokens in the functions below are the tokens of is_word.

Words of a sequence¶

get_words(tokens, join_hyphens=False)

The words of a sequence of tokens in their order; with join_hyphens=True a word the tokenizer split at its hyphens is one word, given by its part that holds its relation, as with the dependents of get_children (governor in He is vice-governor), or by its first part when a hyphen holds it.

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

Hyphenated words¶

A tokenizer may split a word at its hyphens (well-known into well, -, known), and a parser then hangs every part on its own head. With join_hyphens=True the functions of the tree take a word whose parts anyts.utils.joins_previous joins for one word with its hyphens. A word holding the head of the sentence has no head; the head of another one is that of its part hanging outside it, the nearest to the root (a word rather than a hyphen when equal). The dependents of a word are the words whose head hangs on one of its tokens, each given by that part (or by its first part when it hangs by a hyphen); its subtree is the word and the words that hang on it, directly or through punctuation. So every word has one dependency and is the dependent of one word.

SyntaxStats and the functions of the Russian constructions always join hyphenated words, and the functions of the core with the parameter join_hyphens give the values of SyntaxStats with join_hyphens=True.

Head of a sentence¶

is_root(token)

Checks whether a token is the head of its sentence: its head is the token itself.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

Base relation¶

base_dep(token)

The relation of a token without its subtype, which Universal Dependencies separates by a colon: nsubj:pass gives nsubj, expl:pass gives expl.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

Dependent words¶

get_children(token, join_hyphens=False)

The dependent words of a token, punctuation and whitespace left out; with join_hyphens=True, the dependents of the whole hyphenated word the token belongs to.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

count_children(token, join_hyphens=False)

The number of the dependent words of a token; the arguments are those of get_children.

subtree_len(token, join_hyphens=False)

The number of words in the subtree of a token: the token itself and all of its direct and indirect dependents; the arguments are those of get_children.

Verb forms¶

is_finite_verb(token)

Checks whether a token is a finite verb: the part of speech VERB with VerbForm=Fin, so a finite auxiliary (AUX) is not one.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

is_infinitive(token)

Checks whether a token is an infinitive, VerbForm=Inf whatever its part of speech.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

is_participle(token)

is_converb(token)

is_participle checks whether a token is a full participle (VerbForm=Part without Variant=Short and StyleVariant=Short), is_converb whether it is an adverbial participle (VerbForm=Conv); the argument is the same token.

Predicate of a clause¶

is_predicate(token, join_hyphens=False)

Checks whether a word can be the predicate of its clause: a verb (VERB, AUX), a word with a subject of its own (nsubj, csubj and their subtypes, anyts.constants.SUBJECT_DEPS) or with a copula (cop), so the nominal predicate of he is tall and of a language that drops the subject passes and a parenthetical word does not.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

A value that is not a Token raises SourceTypeError.

Chains of modifiers¶

calc_modifier_chains(tokens, is_modifier, join_hyphens=False)

The lengths of the chains of nested modifiers, in the order of the words. A chain is two or more modifiers each hanging on the previous one, such as the complements of the increase of the efficiency of the use of resources (length 3); the check is_modifier of a language says what a modifier is (a complement with a preposition, a noun in the genitive). A chain starts at a modifier whose head is none, and its length is the number of modifiers in its longest branch. The links are those of get_children: punctuation is no link, and with join_hyphens a hyphenated word is one link.

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens
`is_modifier`	`Callable[[Token], bool]`	-	Whether a word is a modifier of the chain
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

A check that is not callable raises SourceTypeError.

Complements with a preposition¶

is_prep_complement(token, join_hyphens=False)

Checks whether a word is a complement with a preposition: a word with the relation nmod or obl or a subtype of them (anyts.constants.PREP_COMPLEMENT_DEPS), or a predicate with a copula or a subject of its own out of a relative clause (acl), such as a nominal predicate of a language that drops the copula of the present tense, that has a dependent of the relation case that is an adposition (ADP) or begins one of several words (fixed: in front of). A noun in a case without a preposition and a possessive marker are none.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

A value that is not a Token raises SourceTypeError.

Dependency triples¶

dependency_triples(tokens, join_hyphens=False)

The dependency triples of the words of a sentence (get_words), in their order: the part of speech of the head of a word - the nearest word above it through punctuation - its relation without a subtype (base_dep) and its own part of speech, such as ("NOUN", "amod", "ADJ") for a red house. The head of the sentence, and a word with no word above it, give an empty head and the relation root, whatever its label.

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

Tokens that are not a Doc, a Span or a list of Token objects raise SourceTypeError.

Edit distance and multiset similarity¶

calc_edit_distance(sents)

The mean edit distance of the items of adjacent sentences: the Levenshtein distance of every two - the fewest insertions, deletions and substitutions that turn one into the other - over the number of the items of the longer one, 0 for the same sequences and 1 for ones with no item in common; nan for fewer than two sentences.

Parameter	Type	Default	Description
`sents`	`list[list]`	-	Items of every sentence, such as its parts of speech

Sentences that are not a list of lists, or items that cannot be hashed, raise SourceTypeError.

calc_multiset_similarity(sents)

The mean similarity of the multisets of the items of adjacent sentences: the weighted Jaccard similarity of every two - the items they share, each as many times as the sentence with fewer of them has it, over the items of both, each as many times as the sentence with more of them has it - 1 for the same multisets and for two empty ones, 0 for disjoint ones; nan for fewer than two sentences. A mapping is read by its keys, as a set.

Parameter	Type	Default	Description
`sents`	`list[list]`	-	Items of every sentence, such as its dependency triples

Sentences that are not a list of collections, or items that cannot be hashed, raise SourceTypeError.

Helper functions¶

is_word(token)

Checks whether a token is a word: not whitespace and not a token of is_punctuation, so symbols like %, €, + and invisible characters are no words either. The words of iter_doc_tokens and the nodes of the dependency tree of the syntactic statistics are such tokens.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

has_feature(token, field, value)

Checks whether a token carries a morphological feature with a given value: has_feature(token, "VerbForm", "Fin").

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`field`	`str`	-	Name of the feature
`value`	`str`	-	Value of the feature

get_lemma(token)

The lemma of a token in lower case by the ruMorph analysis with the part of speech of the token, not token.lemma_.

Parameter	Type	Default	Description
`token`	`Token`	-	Token