Cohesion statistics¶

CohesionStats(source, sents_extractor=None, words_extractor=None, connectors=None, blank_lines=False)

The cohesion statistics of a text in the manner of Coh-Metrix and TAACO: the overlap of the sentences, the givenness, the temporal cohesion and the connectors of a dictionary. The sentences are compared by lemmas. The data source is a text or a Doc object of spaCy.

Raises:

SourceTypeError - a source that is neither a string nor a Doc, an extractor of another type, connectors that are not a dictionary of phrases and pairs of strings;
SourceError - a source without words;
ParameterError - an unknown class or kind of a connector, a blank_lines that is not a bool.

The class extends the CohesionStats of the anyTS core with the Russian layer. A string and a Doc without sentence boundaries are split into sentences by the Russian SentsExtractor; the words of a string come from the Russian WordsExtractor or from the words_extractor given, and the parts of a hyphenated word that spaCy splits are joined (во-первых is one word). The features of the words of a string come from the ruMorph analysis in the context of the sentence, those of a Doc of the recommended pipeline from its ruMorph annotation, and with a plain model the part of speech, the tense and the aspect come from token.pos_ and token.morph, with the lemma by ruMorph. In a Doc without annotation the sentences whose words are those of the string are analyzed in context, and the other words, as with a word extractor given, get the first ruMorph analysis. The connectors are found by RussianConnectorMatcher over the dictionary of load_connectors. The print_stats table and the error messages are in Russian (COHESION_STATS_DESC and CORE_MESSAGES of ruts.constants).

The words are classified by the Universal Dependencies parts of speech and features - of the ruMorph analysis for a string or of the annotation of a Doc:

Notion	String (ruMorph)	`Doc`
Nouns	`NOUN`, `PROPN`	`NOUN`, `PROPN`
Pronouns	`PRON`, `DET`, lemmas of `DEMONSTRATIVE_LEMMAS`	`PRON`, `DET`, lemmas of `DEMONSTRATIVE_LEMMAS`
Arguments	`NOUN`, `PROPN`, `PRON`, the possessive его, её, их with the lemma он, она, они	the same
Content words	`CONTENT_UD_POS` and the copula быть (`AUX` with a verb form) without demonstratives, pronominal (`PronType` Dem, Int, Rel: там, где, почему) and parenthetical (the `Prnt` label and `PARENTHETICAL_WORDS`: кстати, наверно) words, predicatives and numbers in digits	the same by the features of the token
Demonstratives	lemmas of `DEMONSTRATIVE_LEMMAS`: этот, это, тот, то (a pronoun only), такой, таковой, столько, сей, оный	the same lemmas
Tense and aspect	features `Tense` and `Aspect`	features `Tense` and `Aspect`

CodeResult

from ruts import CohesionStats

text = "Кот сидел на окне. Он смотрел на птиц. Птицы улетели, и кот уснул. Завтра он снова будет сидеть на этом окне."
cs = CohesionStats(text)
cs.print_stats()

                       Статистика                        | Значение
--------------------------------------------------------------------
Повтор существительных в соседних предложениях           |   0.33
Повтор существительных во всех парах предложений         |   0.50
Повтор аргументов в соседних предложениях                |   0.33
Повтор аргументов во всех парах предложений              |   0.67
Повтор знаменательных слов в соседних предложениях       |   0.33
Повтор знаменательных слов во всех парах предложений     |   0.50
Доля общих знаменательных слов в соседних предложениях   |   0.11
Доля общих знаменательных слов во всех парах предложений |   0.19
Повтор существительных в двух следующих предложениях     |   0.67
Повтор аргументов в двух следующих предложениях          |   0.75
Повтор знаменательных слов в двух следующих предложениях |   0.40
Повтор существительных в соседних абзацах                |   nan
Повтор аргументов в соседних абзацах                     |   nan
Повтор знаменательных слов в соседних абзацах            |   nan
Доля местоимений                                         |   0.14
Отношение местоимений к существительным                  |   0.50
Доля указательных местоимений                            |   0.05
Доля знаменательных слов, встречавшихся ранее            |   0.29
Повтор времени в соседних предложениях                   |   0.67
Повтор вида в соседних предложениях                      |   0.33
Темпоральная связность                                   |   0.50
Коннекторов на 1000 слов                                 |  47.62
Межпредложенческих коннекторов на 1000 слов              |   0.00
Внутрипредложенческих коннекторов на 1000 слов           |  47.62
Причинных коннекторов на 1000 слов                       |   0.00
Противительных коннекторов на 1000 слов                  |   0.00
Уступительных коннекторов на 1000 слов                   |   0.00
Временных коннекторов на 1000 слов                       |   0.00
Аддитивных коннекторов на 1000 слов                      |  47.62
Условных коннекторов на 1000 слов                        |   0.00
Переформулирующих коннекторов на 1000 слов               |   0.00
Первичных коннекторов на 1000 слов                       |  47.62
Вторичных коннекторов на 1000 слов                       |   0.00

Parameters¶

Parameter	Type	Default	Description
`source`	`str` / `Doc`	-	Data source
`sents_extractor`	`SentsExtractor`	`None`	Sentence extraction tool; given, it splits a `Doc` by its text too, whatever its boundaries
`words_extractor`	`WordsExtractor`	`None`	Word extraction tool; given, it is applied to the text of a `Doc` too
`connectors`	`dict[str, tuple[str, str]]`	`None`	Class and kind by connector; the dictionary of `load_connectors` by default
`blank_lines`	`bool`	`False`	Start the paragraphs after the blank lines alone, for a text whose lines are wrapped inside its paragraphs; a paragraph is a line by default

Attributes¶

Attribute	Type	Description
`words`, `lemmas`	`tuple[tuple[str]]`	Words and lemmas of every sentence with words
`n_sents`, `n_words`	`int`	Numbers of the sentences with words and of the words
`blank_lines`	`bool`	Whether the paragraphs start after the blank lines alone
`n_paragraphs`	`int`	Number of the paragraphs, the lines of the text with content words (the parts between blank lines with `blank_lines`)
`n_nouns`, `n_pronouns`, `n_demonstratives`, `n_content_words`	`int`	Numbers of the words by kind
`n_given`	`int`	Number of the content words whose lemma was used before
`n_connectors`, `connector_spans`, `c_connectors`	`int`, `tuple[Connector]`, `dict[str, int]`	Number of the connectors, their occurrences and their number by connector
`n_inter_connectors`	`int`	Number of the inter-sentential connectors

The extractors given are kept in the attributes sents_extractor and words_extractor, None without them.

CodeResult

cs.lemmas[1]
cs.n_nouns, cs.n_pronouns, cs.n_content_words, cs.n_given
cs.connector_spans
cs.c_connectors

('он', 'смотреть', 'на', 'птица')
(6, 3, 14, 4)
(Connector(sent=2, start=2, end=3, text='и', cls='additive', kind='primary'),)
{'и': 1}

Metrics¶

Every metric is an attribute of the object, computed when it is created; get_stats gives those of stats_desc, all but mood_repetition by default. The functions of the cards take a sentence as the collection of its elements - lemmas of nouns, arguments or content words, values of a feature of its verbs - and see no language.

Metric	Measures	Range	Higher means
`noun_overlap_adjacent`	Share of the adjacent pairs of sentences sharing a noun	0-1, `nan` for one sentence	Neighbouring sentences tied tighter
`noun_overlap_all`	Share of all the pairs of sentences sharing a noun	0-1, `nan` for one sentence	The whole text tied tighter
`argument_overlap_adjacent`	Share of the adjacent pairs sharing a noun or a pronoun	0-1, `nan` for one sentence	Neighbouring sentences tied tighter
`argument_overlap_all`	Share of all the pairs sharing a noun or a pronoun	0-1, `nan` for one sentence	The whole text tied tighter
`content_overlap_adjacent`	Share of the adjacent pairs sharing a content word	0-1, `nan` for one sentence	Neighbouring sentences tied tighter
`content_overlap_all`	Share of all the pairs sharing a content word	0-1, `nan` for one sentence	The whole text tied tighter
`content_overlap_prop_adjacent`	Mean share of the shared content words (Dice) in the adjacent pairs	0-1, `nan` for one sentence	More words shared by neighbours
`content_overlap_prop_all`	Mean share of the shared content words (Dice) in all the pairs	0-1, `nan` for one sentence	More words shared across the text
`noun_overlap_2_sent`	Share of the nouns of a sentence repeated in one of the next two	0-1, `nan` for fewer than three sentences or without nouns in the compared ones	More nouns carried on
`argument_overlap_2_sent`	Share of the arguments of a sentence repeated in one of the next two	0-1, `nan` for fewer than three sentences or without arguments in the compared ones	More arguments carried on
`content_overlap_2_sent`	Share of the content words of a sentence repeated in one of the next two	0-1, `nan` for fewer than three sentences or without content words in the compared ones	More content words carried on
`noun_overlap_para`	Share of the nouns of a paragraph repeated in the next one	0-1, `nan` for one paragraph or without nouns in the compared ones	More nouns carried to the next paragraph
`argument_overlap_para`	Share of the arguments of a paragraph repeated in the next one	0-1, `nan` for one paragraph or without arguments in the compared ones	More arguments carried to the next paragraph
`content_overlap_para`	Share of the content words of a paragraph repeated in the next one	0-1, `nan` for one paragraph or without content words in the compared ones	More content words carried to the next paragraph
`p_pronouns`	Share of the pronouns among the words	0-1	More reference by pronouns
`pronoun_noun_ratio`	Ratio of the pronouns to the nouns	0 and above, `nan` without nouns	More pronouns per noun
`p_demonstratives`	Share of the demonstratives among the words	0-1	More pointing back
`p_given`	Share of the content words whose lemma was used before in the text	0-1, `nan` without content words	More given information
`tense_repetition`	Share of the adjacent pairs of sentences with the same dominant tense	0-1, `nan` without two adjacent sentences with the feature	A steadier tense
`mood_repetition`	Share of the adjacent pairs with the same dominant mood	0-1, `nan` without two adjacent sentences with the feature	A steadier mood
`aspect_repetition`	Share of the adjacent pairs with the same dominant aspect	0-1, `nan` without two adjacent sentences with the feature	A steadier aspect
`temporal_cohesion`	Mean of the repetitions of `temporal_features`	0-1, `nan` when one of them is `nan`	A steadier time frame
`connectors`	Connectors per 1000 words	0-1000	More explicit links
`connectors_inter`	Inter-sentential connectors per 1000 words	0-1000	More links to the sentences before
`connectors_intra`	Intra-sentential connectors per 1000 words	0-1000	More links inside the sentences
`connectors_<class>`, `connectors_<kind>`	Connectors of a class or a kind of the matcher per 1000 words	0-1000	More links of the class or the kind

Binary overlap¶

noun_overlap_adjacent · noun_overlap_all · argument_overlap_adjacent · argument_overlap_all · content_overlap_adjacent · content_overlap_all · Coh-Metrix

calc_overlap(sets, adjacent=True)

The binary overlap of Coh-Metrix (CRFNO1, CRFAO1, CRFSO1 over the adjacent pairs, CRFNOa, CRFAOa, CRFSOa over all of them): the share of the pairs of sentences that share at least one element; nan for a text shorter than two sentences.

\[ \frac{\textrm{Number of pairs sharing an element}}{\textrm{Number of pairs}} \]

Parameter	Type	Default	Description
`sets`	`list[set[str]]`	-	Elements of every sentence
`adjacent`	`bool`	`True`	Count the adjacent pairs only, otherwise all the pairs

Proportional overlap¶

content_overlap_prop_adjacent · content_overlap_prop_all · Coh-Metrix

calc_proportional_overlap(sets, adjacent=True)

The proportional overlap of Coh-Metrix (CRFCWO1, CRFCWOa): the Dice coefficient of a pair of sentences (dice) averaged over the pairs; nan for a text shorter than two sentences.

\[ \frac{1}{\textrm{Number of pairs}} \sum_{(A, B)} \textrm{Dice}(A, B) \]

Parameter	Type	Default	Description
`sets`	`list[set[str]]`	-	Elements of every sentence
`adjacent`	`bool`	`True`	Count the adjacent pairs only, otherwise all the pairs

dice(first, second)

The Dice coefficient of two sets, 0 for two empty sets.

\[ \frac{2 \times |A \cap B|}{|A| + |B|} \]

Parameter	Type	Default	Description
`first`	`frozenset[str]`	-	First set
`second`	`frozenset[str]`	-	Second set

Overlap with the next sentences and paragraphs¶

noun_overlap_2_sent · argument_overlap_2_sent · content_overlap_2_sent · noun_overlap_para · argument_overlap_para · content_overlap_para · TAACO

The adjacent overlap of TAACO, by the lemmas of the nouns, of the arguments and of the content words: the share of the distinct lemmas of a sentence repeated in one of the next two sentences (adjacent_overlap_2_*_sent), and of a paragraph repeated in the next paragraph (adjacent_overlap_*_para), over all the sentences or paragraphs that have them (calc_forward_overlap). Unlike the binary overlap, a pair of segments counts by the share of the lemmas they share. A paragraph is a line of the text, as in TAACO, or a part between blank lines with blank_lines, for a text whose lines are wrapped inside its paragraphs; a word that carries paragraph_start opens one, inside a sentence too, as when a parser joins a heading to the next sentence. A paragraph without content words, such as the number of a chapter, joins the next one, the last such the one before it.

calc_forward_overlap(sets, span=1)

The share of the elements of a segment repeated in the next ones: the distinct elements of every segment with span segments after it that one of these segments holds, over the distinct elements of those segments; nan for no more segments than span or for segments without elements.

\[ \frac{\sum_{i=1}^{n-k} |A_i \cap (A_{i+1} \cup \dots \cup A_{i+k})|}{\sum_{i=1}^{n-k} |A_i|} \]

Parameter	Type	Default	Description
`sets`	`list[set[str]]`	-	Elements of every segment
`span`	`int`	`1`	Number \(k\) of the next segments a segment is compared with

Segments that are a string, a Doc, an iterator or a segment that is not a list of strings raise SourceTypeError, a span that is not an integer above 0 ParameterError.

In ruTS both a string and a Doc are split into paragraphs. In a poem a paragraph is a line of verse by default, and blank_lines=True gives the stanzas set off by blank lines, as in VerseStats.

CodeResult

from ruts.cohesion_stats import calc_forward_overlap

calc_forward_overlap([{"кот", "окно"}, {"птица"}, {"кот"}], span=2)
text = "Кот сидел на окне. Он смотрел на птиц.\nПтицы улетели, и кот уснул. Завтра он снова будет сидеть на этом окне."
paragraphs = CohesionStats(text)
paragraphs.n_paragraphs, paragraphs.noun_overlap_para, paragraphs.content_overlap_para
CohesionStats(text, blank_lines=True).n_paragraphs

0.5
(2, 1.0, 0.8)
1

Pronouns and demonstratives¶

p_pronouns · pronoun_noun_ratio · p_demonstratives

The pronouns, the nouns and the demonstratives are the words analyze marks so, with the features of word_info by default.

\[ \frac{\textrm{Number of pronouns}}{\textrm{Number of words}}, \quad \frac{\textrm{Number of pronouns}}{\textrm{Number of nouns}}, \quad \frac{\textrm{Number of demonstratives}}{\textrm{Number of words}} \]

In ruTS they are marked by the functions of the word features after the table of the classes of words.

Givenness¶

p_given

count_given(sents)

The number of given elements: an element is given when the same lemma was used earlier in any sentence, the current one included; the first occurrence is new.

Parameter	Type	Default	Description
`sents`	`list[list[str]]`	-	Lemmas of every sentence in the order of the text

Temporal cohesion¶

tense_repetition · mood_repetition · aspect_repetition · temporal_cohesion · Coh-Metrix

calc_repetition(sents)

The repetition of the tense and of the aspect of Coh-Metrix (SMTEMP): the share of the adjacent pairs of sentences whose verbs have the same dominant value of the feature; a pair where one of the sentences has no verb with the feature is skipped, and without a single such pair the value is nan.

\[ \frac{\textrm{Number of pairs with the same dominant value}}{\textrm{Number of pairs with the feature}} \]

Parameter	Type	Default	Description
`sents`	`list[list[str]]`	-	Values of the feature of the verbs of every sentence

dominant(values)

The dominant value: the most frequent one, the first of the equally frequent; None for an empty list.

Parameter	Type	Default	Description
`values`	`list[str]`	-	Values

ruTS does not compare the mood: mood_repetition is always nan, and temporal_cohesion is the mean of tense_repetition and aspect_repetition.

Connectors¶

connectors · connectors_inter · connectors_intra · connectors_<class> · connectors_<kind>

The occurrences of the connectors found in every sentence by the matcher of connector_matcher_class, per 1000 words. An inter-sentential connector links a sentence to the ones before: it opens a sentence, unless the matcher takes it for a subordinator (is_subordinator), whose clause or phrase belongs to the same sentence; the others are intra-sentential. Every class and kind of the matcher gives a density: connectors_causal, connectors_adversative, connectors_concessive, connectors_temporal, connectors_additive, connectors_conditional, connectors_reformulative, connectors_primary and connectors_secondary by default.

\[ 1000 \cdot \frac{\textrm{Number of connectors}}{\textrm{Number of words}} \]

load_connectors()

The connector dictionary of ruTS: the file ruts/resources/connectors.tsv, 316 connectors with a class of CONNECTOR_CLASSES and a kind of CONNECTOR_TYPES of ruts.constants, compiled after the classification of Krioni, Nikin and Filippova (2008) and checked against the Ru-RSTreebank marker list, the Ruscon database and the causal marker list of Toldova et al. (2018). The one-word «и», «а», «но» are in the dictionary, while «что» and «как» are no connectors.

Class	Key	Examples
Causal (consequential and purposive included)	`causal`	потому что, поэтому, чтобы
Adversative	`adversative`	но, однако, с другой стороны
Concessive	`concessive`	хотя, несмотря на, тем не менее
Temporal	`temporal`	когда, затем, после того как
Additive	`additive`	и, кроме того, например
Conditional	`conditional`	если, при условии что, в противном случае
Reformulative	`reformulative`	то есть, иными словами, подводя итог

The primary connectors (primary) are conjunctions, adverbs and particles (потому что, однако), the secondary ones (secondary) lexicalized phrases with nouns and verbs (с другой стороны, иными словами). The rules by which a word of the dictionary counts as a connector are those of the matcher.

A list of sentences that is a string, a Doc, an iterator, a set or a table raises SourceTypeError, and so does a sentence that is not a list of strings (check_words), such as a list of spaCy tokens. The overlaps (calc_overlap, calc_proportional_overlap, calc_overlaps) take a sentence as a set too; count_given and calc_repetition count the words in order and refuse a set.

CodeResult

from ruts.cohesion_stats import calc_overlap, calc_repetition, count_given

sets = [{"кот", "окно"}, {"птица"}, {"птица", "кот"}, {"окно"}]
calc_overlap(sets), calc_overlap(sets, adjacent=False)
count_given([["кот", "окно"], ["птица"], ["птица", "кот"]])
calc_repetition([["past"], ["past", "pres", "past"], ["futr"]])

(0.3333333333333333, 0.5)
2
0.5

Methods¶

The methods get_stats, to_dataframe, print_stats and describe are those of the statistics interface of the anyTS core; describe gives None for the statistics of get_stats: the cohesion statistics have no scales.

CodeResult

cs.get_stats()

{'noun_overlap_adjacent': 0.3333333333333333,
 'noun_overlap_all': 0.5,
 'argument_overlap_adjacent': 0.3333333333333333,
 'argument_overlap_all': 0.6666666666666666,
 'content_overlap_adjacent': 0.3333333333333333,
 'content_overlap_all': 0.5,
 'content_overlap_prop_adjacent': 0.1111111111111111,
 'content_overlap_prop_all': 0.18650793650793648,
 'noun_overlap_2_sent': 0.6666666666666666,
 'argument_overlap_2_sent': 0.75,
 'content_overlap_2_sent': 0.4,
 'noun_overlap_para': nan,
 'argument_overlap_para': nan,
 'content_overlap_para': nan,
 'p_pronouns': 0.14285714285714285,
 'pronoun_noun_ratio': 0.5,
 'p_demonstratives': 0.047619047619047616,
 'p_given': 0.2857142857142857,
 'tense_repetition': 0.6666666666666666,
 'aspect_repetition': 0.3333333333333333,
 'temporal_cohesion': 0.5,
 'connectors': 47.61904761904762,
 'connectors_inter': 0.0,
 'connectors_intra': 47.61904761904762,
 'connectors_causal': 0.0,
 'connectors_adversative': 0.0,
 'connectors_concessive': 0.0,
 'connectors_temporal': 0.0,
 'connectors_additive': 47.61904761904762,
 'connectors_conditional': 0.0,
 'connectors_reformulative': 0.0,
 'connectors_primary': 47.61904761904762,
 'connectors_secondary': 0.0}

Overlaps at once¶

calc_overlaps(sets, proportional=True)

The values of calc_overlap and calc_proportional_overlap over the adjacent and over all the pairs at once, as an Overlap named tuple with the fields adjacent, all, prop_adjacent and prop_all; without proportional the last two are nan. The values over all the pairs come without going through every pair, and none of them depends on the order of the elements in the sets.

Parameter	Type	Default	Description
`sets`	`list[set[str]]`	-	Elements of every sentence
`proportional`	`bool`	`True`	Compute the proportional overlap as well

CodeResult

from ruts.cohesion_stats import calc_overlaps

calc_overlaps([{"кот", "окно"}, {"птица"}, {"птица", "кот"}, {"окно"}])

Overlap(adjacent=0.3333333333333333, all=0.5, prop_adjacent=0.2222222222222222, prop_all=0.3055555555555555)

Connector matcher¶

RussianConnectorMatcher(connectors)

The search of the connectors (discourse markers: because, however, on the other hand) of a dictionary in the sentences of a text. The dictionary gives the class of every connector - causal, adversative, concessive, temporal, additive, conditional or reformulative (CONNECTOR_CLASSES) - and its kind, primary or secondary (CONNECTOR_KINDS). A connector is matched by the normalized forms of its words, the longest one at every position, without overlaps; a period or a hyphen inside a connector may be split off by the tokenizer (e.g. as e, g), and the period of an abbreviation may stay with each of its parts, as in the tokens of UD (e., g.). An occurrence is a Connector: the number of the sentence sent, the positions start and end of its words, the connector text as the dictionary writes it, its class cls and kind kind.

Raises:

SourceTypeError - a dictionary that is not a mapping of a phrase to a pair of strings;
ParameterError - an unknown class or kind.

RussianConnectorMatcher is a subclass of the core ConnectorMatcher with the rules of Russian - lists of ruts.constants that become attributes of the matcher named without CONNECTOR_ in lower case (CONNECTOR_BLOCKED_AFTER is blocked_after, the core CONNECTOR_POS is pos_tags); the attributes not_after_pos and start_only_pos are empty in ruTS. Words and connectors are compared in lower case without the letter ё and stress marks. A one-word connector takes a part of speech of CONNECTOR_POS (CCONJ, SCONJ, PART, ADV, ADP, INTJ, X) or of CONNECTOR_POS_EXTRA for its form (словом - NOUN, допустим - VERB): «раз» as a noun and «значит» as a verb are no connectors. A connector does not count with a word after it of CONNECTOR_BLOCKED_AFTER (пока что), before it of CONNECTOR_BLOCKED_BEFORE (ещё раз) or with a part of speech of the next word of CONNECTOR_BLOCKED_AFTER_POS (после обеда); a connector of CONNECTOR_ONLY_AFTER counts inside a sentence only after the words given (и отсюда), and one of CONNECTOR_NOT_AT_START does not count at its start (the interrogative отчего). The rules read through a punctuation mark only for CONNECTOR_READ_THROUGH_MARKS (не знал, отчего), and a mark between the words of a connector of CONNECTOR_SPLIT_BY_MARKS splits it (так, как).

find(words, sent_index, pos, breaks, blocked) returns the occurrences of the connectors in a sentence in the order of its words. The words are those of one sentence without punctuation, the first one its start. With the parts of speech, a one-word connector counts only with one of pos_tags or of pos_extra for its form, never after one of not_after_pos, and with one of start_only_pos only at the start of the sentence; a word without a part of speech (None) passes. Without the parts of speech the rules that read them are skipped. The positions of the punctuation marks breaks cut a connector off its neighbours: a rule does not read a word across a mark, so in so, the end the word after the comma neither blocks nor lets the connector count, unless the connector is one of read_through_marks, and a mark between the words of a connector of split_by_marks makes it none; without them the words are read as adjacent. No connector starts at a word of blocked, which a language rules out by more than the words and their parts of speech, such as its parse.

Parameter	Type	Default	Description
`words`	`list[str]`	-	Words of the sentence
`sent_index`	`int`	`0`	Number of the sentence, written into the occurrences
`pos`	`list[str]`	`None`	Parts of speech of the words
`breaks`	`set[int]`	`None`	Positions of the words a punctuation mark stands before
`blocked`	`set[int]`	`None`	Positions of the words no connector starts at

Words that are not a list of strings and parts of speech that are not a list of strings and None (the tags token.pos_ of spaCy, not the numbers token.pos) and positions that are not a collection raise SourceTypeError, a number of the sentence or a position that is not an integer, a position of a mark that is not that of a word after the first, a blocked position that is not that of a word and parts of speech not as many as the words ParameterError.

is_subordinator(connector, pos=None) checks whether an occurrence of a connector ties its clause or phrase to its own sentence: a connector of subordinators, or a one-word connector whose part of speech pos is SCONJ or ADP, but none of parcellation. The clause of a subordinating conjunction and the phrase of a preposition belong to the sentence of the connector wherever it stands, so a subordinator opening a sentence (when he came, ..., despite the rain, ...) does not link it to the ones before. A connector that is not a Connector raises SourceTypeError.

The subordinators of ruTS besides the one-word ones tagged SCONJ and ADP are CONNECTOR_SUBORDINATORS: the multi-word conjunctions and prepositions (так как, несмотря на) and the one-word ones the spaCy models or ruMorph tag otherwise (ежели, пока). A conjunction of CONNECTOR_PARCELLATION (ибо) opening a sentence attaches it to the one before and is no subordinator.

Connector search¶

find_connectors(words, connectors=None, sent_index=0, pos=None, breaks=None, blocked=None)

Searches a sentence for connectors with the find method of RussianConnectorMatcher over the dictionary connectors, that of load_connectors by default; the other parameters and the errors are those of the matcher and of find.

Parameter	Type	Default	Description
`connectors`	`dict[str, tuple[str, str]]`	`None`	Class and kind by connector

connector_pos(word)

The UD part of speech of a word by ruMorph for the check of a connector: the first analysis with a part of speech that suits a connector (CONNECTOR_POS or CONNECTOR_POS_EXTRA for the word), otherwise the first analysis.

Parameter	Type	Default	Description
`word`	`str`	-	Word

CodeResult

from ruts.cohesion_stats import find_connectors

find_connectors(["Затем", "иными", "словами", "он", "уснул"])
find_connectors(["Раз", "он", "пришел"])
find_connectors(["Раз", "он", "пришел"], pos=["NOUN", "PRON", "VERB"])

[Connector(sent=0, start=0, end=1, text='затем', cls='temporal', kind='primary'),
 Connector(sent=0, start=1, end=3, text='иными словами', cls='reformulative', kind='secondary')]
[Connector(sent=0, start=0, end=1, text='раз', cls='conditional', kind='primary')]
[]

Word features¶

word_info(word)

words_info(sents, text, analyses=None)

analysis_info(analysis)

token_info(token)

ud_info(word, pos, tense, aspect, feats=None)

unit_info(unit)

The features of a word for the cohesion statistics, a WordInfo named tuple: lemma, noun, pronoun, demonstrative, argument, content, tense and aspect after the table of the classes of words. word_info takes them from the first ruMorph analysis, analysis_info from a given analysis, for example one chosen by context, words_info for the words of the sentences of a text: by the analysis of analyses when it is given, otherwise by the first one. token_info takes them from the ruMorph analysis the token is annotated with (the rumorph component), and without it from the spaCy annotation (token.pos_, token.morph), ud_info from a given part of speech, tense, aspect and features of UD; the lemma then comes from the ruMorph analysis with that part of speech, not from token.lemma_. unit_info gives the features of a word from its tokens: a single token by token_info, a hyphenated word of several tokens (жили-были) by the ruMorph analysis of the joined text.

Parameter	Type	Default	Description
`word`	`str`	-	Word (`word_info`, `ud_info`)
`sents`	`list[list[str]]`	-	Words of the sentences of a text (`words_info`)
`text`	`str`	-	Text the words are extracted from (`words_info`)
`analyses`	`list[list[Analysis \\| None]]`	`None`	Analyses of the words of the sentences, `None` for the analysis by word (`words_info`)
`analysis`	`Analysis`	-	ruMorph analysis (`analysis_info`)
`token`	`Token`	-	spaCy token (`token_info`)
`pos`, `tense`, `aspect`	`str`	-	Part of speech, tense and aspect of UD (`ud_info`)
`feats`	`dict[str, str]`	`None`	Features of UD of the word, `None` - unknown (`ud_info`)
`unit`	`list[Token]`	-	Tokens of a word (`unit_info`)

CodeResult

from ruts.cohesion_stats import word_info

word_info("смотрел")

WordInfo(lemma='смотреть', noun=False, pronoun=False, demonstrative=False, argument=False, content=True, tense='Past', aspect='Imp')

Words of a Doc¶

split_doc_units(source, sents_extractor, join_hyphens=False)

Splits the words of a Doc or a Span into the sentences of a sentence extractor, for a Doc without sentence boundaries or one to split otherwise: the sentences are found in its text in their order, and a word of iter_doc_units belongs to the sentence its first token lies in. The words outside the sentences found, those the extractor drops, are left out, so no sentence found gives no words.

Parameter	Type	Default	Description
`source`	`Doc` / `Span`	-	Doc or Span object
`sents_extractor`	`SentsExtractor`	-	Sentence extraction tool
`join_hyphens`	`bool`	`False`	Join the parts of hyphenated words

unit_text(unit)

unit_pos(unit)

unit_text gives the text of a word from its tokens without a byte order mark, stress marks, soft hyphens and the period of an abbreviation when the word is the same without it, unit_pos its part of speech of UD: that of the token for a single token, and that of connector_pos of the joined text for a hyphenated word of several tokens.

Parameter	Type	Default	Description
`unit`	`list[Token]`	-	Tokens of a word

CodeResult

import spacy
from ruts import SentsExtractor
from ruts.cohesion_stats import split_doc_units, unit_text

doc = spacy.blank("ru")("Жили-были дед да баба. Во-первых, кот спит.")
units = split_doc_units(doc, SentsExtractor())
[[unit_text(unit) for unit in sent] for sent in units]
units = split_doc_units(doc, SentsExtractor(), join_hyphens=True)
[[unit_text(unit) for unit in sent] for sent in units]

[['Жили', 'были', 'дед', 'да', 'баба'], ['Во', 'первых', 'кот', 'спит']]
[['Жили-были', 'дед', 'да', 'баба'], ['Во-первых', 'кот', 'спит']]

Pronoun¶

is_pronoun(word)

Checks whether a word is a pronoun by the ruMorph analysis: PRON (он, себя, кто) or DET (этот, который, мой, весь).

Parameter	Type	Default	Description
`word`	`str`	-	Word

Content word¶

is_content_word(word)

Checks whether a word is a content word by the ruMorph analysis: a part of speech of CONTENT_UD_POS (nouns, proper nouns, adjectives, verbs in all forms, adverbs) and the copula быть, except pronominal and parenthetical words, predicatives and numbers in digits.

Parameter	Type	Default	Description
`word`	`str`	-	Word