Syntactic statistics¶

SyntaxStats(source)

The syntactic statistics of a text by its dependency tree (Ivanov, Solnyshkina and Solovyev 2018): the measures of the tree and of the syntactic constructions. The data source is a Doc object of spaCy with a dependency parse. Punctuation, symbols and whitespace are no nodes of the tree, and the distances are counted in positions of words.

Raises:

SourceTypeError - a source that is not a Doc;
SourceError - a source without a dependency parse (relations and sentence boundaries), without lemmas when the class requires them, or without words.

CodeResult

import spacy
from spacy.tokens import Doc

from anyts.syntax_stats import SyntaxStats

# She said that the house was not built by him.
doc = Doc(
    spacy.blank("en").vocab,
    words="She said that the house was not built by him .".split(),
    heads=[1, 1, 7, 4, 7, 7, 7, 1, 9, 7, 1],
    deps="nsubj ROOT mark det nsubj:pass aux:pass advmod ccomp case obl:agent punct".split(),
    pos="PRON VERB SCONJ DET NOUN AUX PART VERB ADP PRON PUNCT".split(),
    morphs="_ VerbForm=Fin _ _ _ VerbForm=Fin Polarity=Neg VerbForm=Part|Voice=Pass _ _ _".split(),
)
stats = SyntaxStats(doc)
stats.print_stats()

                    Statistic                     |  Value
-------------------------------------------------------------
Mean dependency distance                          |   2.44
Standard deviation of the dependency distance     |   1.77
Mean of the longest dependencies of the sentences |   6.00
Share of adjacent dependencies                    |   0.44
Depth of the dependency tree                      |   3.00
Leaves per sentence                               |   6.00
Subtrees per sentence                             |   4.00
Mean of the nodes per leaf of the sentences       |   1.67
Valency of the finite verbs                       |   2.00
Coordination chains per sentence                  |   0.00
Mean length of a coordination chain               |   nan
Clauses per sentence                              |   2.00
Mean length of a clause (words)                   |   5.00
Subordinate clauses per sentence                  |   1.00
Share of sentences with a subordinate clause      |   1.00
Infinitive clauses under a verb per sentence      |   0.00
Modifiers per noun phrase                         |   1.00
Participial clauses per sentence                  |   0.00
Mean length of a participial clause (words)       |   nan
Share of passive forms among the verbs            |   0.50
Share of agentless forms among the passive ones   |   0.00
Infinitives per sentence                          |   0.00
Words of negation per sentence                    |   1.00
Split predicates per sentence                     |   0.00
Ratio of nouns to verbs                           |   0.50

Parameters¶

Parameter	Type	Default	Description
`source`	`Doc`	-	Data source: a `Doc` with a dependency parse, or what `parse` takes

Attributes¶

Attribute	Type	Description
`sents`	`tuple[tuple[Token]]`	Words of every sentence with words
`n_sents`, `n_words`	`int`	Numbers of the sentences with words and of the words
`n_leaves`, `n_subtrees`	`int`	Numbers of the words without and with dependent words
`n_coordination_chains`	`int`	Number of the coordination chains
`n_clauses`, `n_subordinate_clauses`, `n_complex_sents`	`int`	Numbers of the clauses, of the subordinate clauses and of the sentences with one
`n_infinitive_clauses`	`int`	Number of the infinitive clauses under a verb
`n_nouns`, `n_verbs`	`int`	Numbers of the nouns (`NOUN`, `PROPN`) and of the verb forms (`VERB`)
`n_participle_clauses`	`int`	Number of the participial clauses
`n_passive`, `n_agentless_passive`	`int`	Numbers of the passive forms and of those with no agent
`n_infinitives`, `n_negations`	`int`	Numbers of the infinitives and of the words of negation
`n_split_predicates`, `split_predicates`	`int`, `tuple[str]`	Number of the split predicates and the predicates, the verb and the noun
`c_children`	`dict[int, int]`	Number of the words by number of dependent words
`c_deps`	`dict[str, int]`	Number of the words by relation

Metrics¶

Every metric is an attribute of the object, computed when it is created, and get_stats gives them all. The measures of a sentence (the longest dependency, the depth, the nodes per leaf) are averaged over the sentences; the constructions are given per sentence.

Metric	Measures	Range	Higher means
`mean_dependency_distance`	Mean distance between a word and its head	1 and above, `nan` without dependencies	Harder to process
`std_dependency_distance`	Standard deviation of the dependency distance	0 and above, `nan` without dependencies	Less even distances
`max_dependency_distance`	Mean of the longest dependencies of the sentences that have dependencies	1 and above, `nan` without dependencies	Longer longest dependencies
`p_adjacent_dependencies`	Share of the dependencies of length 1	0-1, `nan` without dependencies	More links between neighbours
`tree_depth`	Mean depth of the tree of a sentence	0 and above	Deeper embedding
`leaves_per_sent`	Leaves per sentence	0 and above	Longer or flatter sentences
`subtrees_per_sent`	Subtrees per sentence	0 and above	More phrases in a sentence
`nodes_per_leaf`	Mean over the sentences of the words per leaf	1 and above, `nan` for a sentence without leaves	Longer branches
`verb_valency`	Mean number of the dependents of a finite verb	0 and above, `nan` without finite verbs	More dependents of a verb
`coordination_chains_per_sent`	Coordination chains per sentence	0 and above	More coordination
`mean_coordination_chain_len`	Mean length of a coordination chain in words	2 and above, `nan` without chains	Longer enumerations
`clauses_per_sent`	Clauses per sentence	0 and above	More clauses in a sentence
`mean_clause_len`	Mean length of a clause in words	1 and above, `nan` without clauses	Longer clauses
`subordinate_clauses_per_sent`	Subordinate clauses per sentence	0 and above	More subordination
`p_complex_sents`	Share of the sentences with a subordinate clause	0-1	More complex sentences
`infinitive_clauses_per_sent`	Infinitive clauses under a verb per sentence (wants to leave), which are not subordinate	0 and above	More infinitive complements
`modifiers_per_noun`	Mean number of the modifiers of a noun	0 and above, `nan` without nouns	Heavier noun phrases
`participle_clauses_per_sent`	Participial clauses per sentence	0 and above	More participial clauses
`mean_participle_clause_len`	Mean length of a participial clause in words	1 and above, `nan` without participial clauses	Longer participial clauses
`p_passive`	Share of the passive forms among the verb forms	0-1, `nan` without verb forms	More passive
`p_agentless_passive`	Share of the forms with no agent among the passive ones	0-1, `nan` without passive forms	More passive without an agent
`infinitives_per_sent`	Infinitives per sentence	0 and above	More infinitives
`negations_per_sent`	Words of negation per sentence	0 and above	More negation
`split_predicates_per_sent`	Split predicates per sentence	0 and above	More split predicates
`noun_verb_ratio`	Ratio of the nouns to the verb forms	0 and above, `nan` without verb forms	A more nominal style

Dependency distance¶

mean_dependency_distance · std_dependency_distance · max_dependency_distance · p_adjacent_dependencies

calc_dependency_distances(tokens, join_hyphens=False)

The dependency distances (Liu 2008): the distance between a word and its head in positions of words, punctuation left out, in the order of the words. The head of a sentence and a word whose head lies outside the sequence are skipped.

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one position

Tree depth¶

tree_depth

calc_tree_depth(tokens, join_hyphens=False)

The depth of the dependency tree: the longest path from the head of a sentence to a leaf, in relations between words; the maximum for several sentences, 0 for a sentence of one word. With join_hyphens a hyphenated word is one word, and the words that hang on any of its parts are one level below it.

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

Leaves and subtrees¶

leaves_per_sent · subtrees_per_sent · nodes_per_leaf

A leaf is a word without dependent words (count_children), a subtree a word with them. The nodes per leaf are the words of a sentence over its leaves, averaged over the \(S\) sentences.

\[ \frac{1}{S} \sum_{s=1}^{S} \frac{\textrm{Number of words of } s}{\textrm{Number of leaves of } s} \]

Valency of the finite verbs¶

verb_valency

calc_valency(token, join_hyphens=False)

The valency of a token: the number of its dependent words, the relations cc, conj, parataxis and punct left out (anyts.constants.VALENCY_IGNORED_DEPS); with join_hyphens=True a hyphenated dependent counts once, so came in Some-one came has one dependent and not two.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

Coordination chains¶

coordination_chains_per_sent · mean_coordination_chain_len

calc_coordination_chains(tokens, join_hyphens=False)

The lengths of the coordination chains, in the order of their first words. A chain is a group of words linked by conj, whichever word each conjunct hangs on: Universal Dependencies attaches every conjunct to the first one (pears → apples, plums → apples), ClearNLP to the previous one (pears → apples, plums → pears); both give one chain of three for apples, pears and plums. A nested coordination (cats and dogs, or birds) and an enumeration split between several heads are one chain too. The length of a chain is the number of its words; a conjunct whose head lies outside the sequence is not linked to it. With join_hyphens a hyphenated word is one conjunct, linked by the conj of any of its parts (member and vice-governor is a chain of two).

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

Clauses¶

clauses_per_sent · mean_clause_len

is_clause_head(token, join_hyphens=False, is_infinitive_periphrasis=None)

Checks whether a word heads a clause: the head of the sentence or a word with ccomp, advcl, acl, csubj or parataxis (anyts.constants.CLAUSE_DEPS, subtypes included) whose predicate (predicate_form, so has been built is finite) is not a participle, a converb or a gerund (NONFINITE_CLAUSE_FORMS, counted apart), nor an infinitive under acl; under xcomp, a finite predicate or an infinitive clause (is_infinitive_clause). A parataxis and a conjunct of a clause head count only as predicates (is_predicate); an infinitive conjunct and an infinitive subject (csubj) are no clause by the check is_infinitive_periphrasis, since a model may attach the infinitive of a periphrasis to its verb as conj and to a modal word as csubj; a loop of conjuncts is no clause.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word
`is_infinitive_periphrasis`	`Callable[[Token], bool]`	`None`	Whether an infinitive makes one predicate with its head; none by default

A check of a periphrasis that is not callable raises SourceTypeError.

is_infinitive_clause(token, is_infinitive_periphrasis=None, is_clause_head=is_clause_head)

Checks whether an infinitive complement heads a clause of its own: an infinitive (predicate_form, so to be chosen is one) under xcomp (wants to leave), unless the language check is_infinitive_periphrasis says it makes one predicate with its head (can leave, begins to leave), or the word it completes, the first one up its chain of non-finite complements under xcomp (kept trying to leave, seemed able to leave), heads no clause by the check is_clause_head: an infinitive in a participial phrase or under an infinitive under acl (trying to leave, the wish to try to leave) belongs to that phrase. A ClearNLP parse hangs an infinitive after a predicative adjective on an acomp or oprd (is ready to leave), which heads no clause, so that infinitive is none.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`is_infinitive_periphrasis`	`Callable[[Token], bool]`	`None`	Whether an infinitive makes one predicate with its head; none by default
`is_clause_head`	`Callable[[Token], bool]`	`is_clause_head`	Whether a word heads a clause

A check that is not callable raises SourceTypeError.

Subordinate clauses¶

subordinate_clauses_per_sent · p_complex_sents

is_subordinate_clause_head(token, is_clause_head=is_clause_head)

Checks whether a word heads a subordinate clause: a clause head by the check is_clause_head with ccomp, advcl, acl or csubj (SUBORDINATE_CLAUSE_DEPS), or under xcomp with a finite predicate (a complement a model attaches as xcomp), or a conjunct of one, by the relation of the first word up its chain of conjuncts. An infinitive under xcomp heads an infinitive clause, counted apart.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`is_clause_head`	`Callable[[Token], bool]`	`is_clause_head`	Whether a word heads a clause

A check that is not callable raises SourceTypeError.

Infinitive clauses¶

infinitive_clauses_per_sent

is_infinitive_clause_head(token, is_clause_head=is_clause_head)

Checks whether a word heads an infinitive clause under a verb: a clause head by the check is_clause_head whose predicate is an infinitive under xcomp (wants to leave, asked him to leave), or a conjunct of one. Such a clause is counted apart from the subordinate ones, as grammars of many languages take it for a part of a simple sentence.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`is_clause_head`	`Callable[[Token], bool]`	`is_clause_head`	Whether a word heads a clause

A check that is not callable raises SourceTypeError.

Modifiers of a noun¶

modifiers_per_noun

count_noun_modifiers(token, join_hyphens=False)

Counts the modifiers of a noun phrase: the dependents with the relations amod, det, nmod, nummod, acl and their subtypes (anyts.constants.NOUN_MODIFIER_DEPS); conj and appos are no modifiers.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

Participial clauses¶

participle_clauses_per_sent · mean_participle_clause_len

The clauses headed by the words of the hook is_participle_clause; the length of a clause is the number of the words in the subtree of its head (subtree_len).

Passive¶

p_passive · p_agentless_passive

is_passive(token, join_hyphens=False)

Checks whether a word is a passive verb form: a verb with Voice=Pass or a dependent nsubj:pass, csubj:pass, aux:pass, or nsubjpass, csubjpass, auxpass of ClearNLP (PASSIVE_DEPS).

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

has_agent(token, join_hyphens=False)

Checks whether a word has an agent: a dependent obl:agent, or agent of ClearNLP (AGENT_DEPS). The arguments are those of is_passive.

is_agentless(token, join_hyphens=False)

Checks whether a word is a passive form (is_passive) without an agent (has_agent), with the same arguments.

Infinitives, negations and split predicates¶

infinitives_per_sent · negations_per_sent · split_predicates_per_sent

The infinitives (is_infinitive of anyts.utils), the words of negation of the hook is_negation and the split predicates of the hook find_split_predicates, per sentence.

Ratio of the nouns to the verbs¶

noun_verb_ratio

The number of the nouns n_nouns to the number of the verb forms n_verbs.

\[ \frac{\textrm{Number of nouns}}{\textrm{Number of verb forms}} \]

Methods¶

The methods get_stats, to_dataframe, print_stats and describe are those of the statistics interface.

The functions of the tree take a spaCy Doc, a Span or a list of tokens; its words are the tokens of anyts.utils.is_word. They read the relations that Universal Dependencies and the ClearNLP labels of the English models share: the head of a token, cc, conj, parataxis, punct and aux, the subjects nsubj and csubj, the modifiers of a noun amod, det, nmod, nummod and acl, the clause relations ccomp, advcl and xcomp, and the root; the copula cop of predicate_form exists in Universal Dependencies alone. The passive and its agent (is_passive, has_agent) read the relations of both schemes (PASSIVE_DEPS, AGENT_DEPS). Everything else is up to a language library.

Words of a sequence¶

get_words(tokens, join_hyphens=False)

The words of a sequence of tokens in their order; with join_hyphens=True a word the tokenizer split at its hyphens is one word, given by its part that holds its relation, as with the dependents of get_children (governor in He is vice-governor), or by its first part when a hyphen holds it.

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

Hyphenated words¶

A tokenizer may split a word at its hyphens (well-known into well, -, known), and a parser then hangs every part on its own head. With join_hyphens=True the functions of the tree take a word whose parts anyts.utils.joins_previous joins for one word with its hyphens. A word holding the head of the sentence has no head; the head of another one is that of its part hanging outside it, the nearest to the root (a word rather than a hyphen when equal). The dependents of a word are the words whose head hangs on one of its tokens, each given by that part (or by its first part when it hangs by a hyphen); its subtree is the word and the words that hang on it, directly or through punctuation. So every word has one dependency and is the dependent of one word.

Head of a sentence¶

is_root(token)

Checks whether a token is the head of its sentence: its head is the token itself.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

Base relation¶

base_dep(token)

The relation of a token without its subtype, which Universal Dependencies separates by a colon: nsubj:pass gives nsubj, expl:pass gives expl.

Parameter	Type	Default	Description
`token`	`Token`	-	Token

Dependent words¶

get_children(token, join_hyphens=False)

The dependent words of a token, punctuation and whitespace left out; with join_hyphens=True, the dependents of the whole hyphenated word the token belongs to.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

count_children(token, join_hyphens=False)

The number of the dependent words of a token; the arguments are those of get_children.

subtree_len(token, join_hyphens=False)

The number of words in the subtree of a token: the token itself and all of its direct and indirect dependents; the arguments are those of get_children.

Verb form of a predicate¶

predicate_form(token, auxiliary_form=None)

The verb form of a predicate by the forms (VerbForm) of its auxiliaries and copulas (aux and cop with their subtypes, anyts.constants.AUXILIARY_DEPS), as Universal Dependencies annotates them. A finite auxiliary makes the predicate finite wherever it stands (had arrived, the German gekommen sein wird); otherwise the first auxiliary decides (to have gone is an infinitive, being chosen a gerund), and without auxiliaries the word itself. A past participle auxiliary counts as finite, since it follows a finite one the parser may attach to another word (had been chosen); a present participle (having eaten) does not. A ClearNLP parse has no copula relation: its copula heads the clause with its own form, and its passive auxiliary auxpass is not read. An auxiliary the model leaves without a form is read by the language hook auxiliary_form, which gives its form or an empty string to skip it; without the hook it is skipped. A word with no verb form gives an empty string.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`auxiliary_form`	`Callable[[Token], str]`	`None`	Verb form of an auxiliary without `VerbForm`

A hook that is not callable raises SourceTypeError.

CodeResult

import spacy
from spacy.tokens import Doc

from anyts.syntax_stats import predicate_form

vocab = spacy.blank("en").vocab
doc = Doc(
    vocab,
    words=["She", "had", "been", "chosen"],
    heads=[3, 3, 3, 3],
    deps=["nsubj:pass", "aux", "aux:pass", "ROOT"],
    morphs=["Case=Nom", "VerbForm=Fin", "VerbForm=Part", "VerbForm=Part"],
)
predicate_form(doc[3])
# An auxiliary the model left without a form, read by the hook
doc = Doc(
    vocab,
    words=["to", "have", "gone"],
    heads=[2, 2, 2],
    deps=["mark", "aux", "ROOT"],
    morphs=["", "", "VerbForm=Part"],
)
predicate_form(doc[2]), predicate_form(doc[2], lambda aux: "Inf" if aux.lower_ == "have" else "")

'Fin'
('Part', 'Inf')

Predicate of a clause¶

is_predicate(token, join_hyphens=False)

Checks whether a word can be the predicate of its clause: a verb (VERB, AUX) or a word with a subject of its own (nsubj, csubj and their subtypes, anyts.constants.SUBJECT_DEPS), so the nominal predicate of he is tall passes and a parenthetical word does not.

Parameter	Type	Default	Description
`token`	`Token`	-	Token
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

Chains of modifiers¶

calc_modifier_chains(tokens, is_modifier, join_hyphens=False)

The lengths of the chains of nested modifiers, in the order of the words. A chain is two or more modifiers each hanging on the previous one, such as the complements of the increase of the efficiency of the use of resources (length 3); the check is_modifier of a language says what a modifier is (a complement with a preposition, a noun in the genitive). A chain starts at a modifier whose head is none, and its length is the number of modifiers in its longest branch. The links are those of get_children: punctuation is no link, and with join_hyphens a hyphenated word is one link.

Parameter	Type	Default	Description
`tokens`	`Doc` / `Span` / `list[Token]`	-	Sequence of tokens
`is_modifier`	`Callable[[Token], bool]`	-	Whether a word is a modifier of the chain
`join_hyphens`	`bool`	`False`	Take a hyphenated word for one word

A check that is not callable raises SourceTypeError.

CodeResult

import spacy
from spacy.tokens import Doc

from anyts.syntax_stats import calc_modifier_chains

doc = Doc(
    spacy.blank("en").vocab,
    words=["the", "use", "of", "the", "water", "of", "the", "river"],
    heads=[1, 1, 4, 4, 1, 7, 7, 4],
    deps=["det", "ROOT", "case", "det", "nmod", "case", "det", "nmod"],
)
calc_modifier_chains(doc, lambda token: token.dep_ == "nmod")

[2]

For library developers¶

Language hooks of a subclass

A language library subclasses SyntaxStats. The defaults read Universal Dependencies with its morphological features and the relations it shares with ClearNLP; the passive and its agent read the relations of both:

Hook	Kind	Description
`parse(source)`	method	`Doc` of a source; by default the source itself, which must be a `Doc`; a library may parse a string
`is_clause_head(token)`	method	Whether a word heads a clause; by default `is_clause_head` of the module with `join_hyphens` of the class and the hook `is_infinitive_periphrasis`
`is_infinitive_periphrasis(token)`	method	Whether an infinitive under `xcomp`, an infinitive conjunct or an infinitive subject (`csubj`) of a modal word makes one predicate with its head, a modal or phasal verb (can leave, begins to leave); none by default
`is_subordinate_clause_head(token)`	method	Whether a clause head heads a subordinate clause, asked only of the words of `is_clause_head`; by default `is_subordinate_clause_head` of the module over the hook `is_clause_head`
`is_infinitive_clause_head(token)`	method	Whether a clause head heads an infinitive clause under a verb (wants to leave), counted apart from the subordinate ones; by default `is_infinitive_clause_head` of the module over the hook `is_clause_head`
`is_participle_clause(token)`	method	Whether a word heads a participial clause; by default a participle that is not the head of the sentence, has no auxiliary or copula and has a dependent (`calc_valency`)
`is_passive(token)`	method	Whether a word is a passive verb form; by default `is_passive` of the module with `join_hyphens` of the class
`is_agentless(token)`	method	Whether a passive form has no agent; by default the hook `is_passive` without an agent (`has_agent` of the module)
`is_negation(token)`	method	Whether a word is a word of negation; by default `Polarity=Neg` or a word of `negation_words`, a conjunction excluded
`find_split_predicates(words)`	method	Pairs of a light verb and the noun that carries its meaning (`to make a decision`); none by default
`negation_words`	class attribute	Words of negation besides `Polarity=Neg`, in lower case; none by default
`join_hyphens`	class attribute	Take a word the tokenizer split at its hyphens for one word, in the tree and in the default hooks
`requires_lemmas`	class attribute	Whether the source must carry lemmas, for hooks that read them
`stats_desc`, `stats_headers`	class attributes	Statistics of `get_stats` with their descriptions (`SYNTAX_STATS_DESC` by default), headers of the columns for `print_stats`
`messages`	class attribute	Messages of the errors by key; `MESSAGES` by default

The subclass adds the statistics of the constructions of its language (chains of complements by calc_modifier_chains, its own clauses) in its constructor from sents, with its own stats_desc. A negation_words that is not a collection of strings raises SourceTypeError when the class is defined, and so does a parse that does not return a Doc when the statistics are computed.