Text highlighting

HighlightedText(source, layers=None, long_sent_word_factor=30, complex_syl_factor=4, stopwords=None, cliches=None, alliteration_threshold=0.001)

Highlighting of a text by layers, in the manner of the style checkers: every layer marks the fragments of a text that a statistic counts - long sentences, complex words, the passive. The data source is a text or a Doc object of spaCy. The result shows in Jupyter as HTML with styles and a legend; to_html returns the markup, and highlights keeps the fragments for a rendering of one's own. Fragments of different layers may overlap.

Raises:

• SourceTypeError - a source that is neither a string nor a Doc;
• SourceError - a source without words;
• ParameterError - layers that are not a name or a list of names, a layer that is unknown or one that needs an annotation the source lacks.

highlight(source, layers=None, long_sent_word_factor=30, complex_syl_factor=4, stopwords=None, cliches=None, alliteration_threshold=0.001)

The function highlight returns a HighlightedText object of esTS with the layers of the statistics of the library:

Group	Layer	What it marks	Statistic
Readability	long_sents	Sentences of long_sent_word_factor words or more	BasicStats
	complex_words	Words of complex_syl_factor syllables or more	BasicStats
	rare_words	Words with a lemma beyond the embedded top 10000; numbers, words with a hyphen or a digit and stopwords are not highlighted	LexicalStats
Syntax	passive	Passive verb forms with their auxiliary ser or their se; the note tells the two apart and marks the passive with ser without an agent	SyntaxStats
	participle_clauses	Participial clauses	SyntaxStats
	gerund_clauses	Gerund clauses	SyntaxStats
	de_chains	Chains of complements with de with their head	SyntaxStats
	split_predicates	Split predicates from the verb to the noun	SyntaxStats
Officialese	verbal_nouns	Nouns derived from a verb	StyleStats
	compound_prepositions	Compound prepositions of COMPOUND_PREPOSITIONS	StyleStats
	cliches	Clichés of OFFICIALESE_CLICHES or of the parameter cliches	StyleStats
	anaphoric_mismo	The anaphoric el mismo with its article (se archivó el mismo, la duración del mismo)	-
Style	stopwords	Stopwords of STOPWORDS or of the parameter stopwords, the water of the text	StyleStats
	parentheticals	Parenthetical expressions	StyleStats
	connectors	Discourse markers, the class and the kind in the note	CohesionStats
Phonics	alliteration	Repetitions of a consonant sound in neighbouring words, unlikely by the frequencies of the Spanish sounds; the note gives the sound and the letters that write it	PhonStats

The groups are defined in ests.constants.HIGHLIGHT_LAYER_GROUPS. The layers of the group "Syntax" need a Doc with a dependency parse and the lemmas (the models es_core_news_sm, es_core_news_md, es_core_news_lg), as SyntaxStats does; verbal_nouns needs a Doc with the parts of speech and the lemmas, and anaphoric_mismo one with the dependency parse, the parts of speech and the lemmas. The layers of HIGHLIGHT_DEFAULT_LAYERS the source allows are on by default - long sentences, complex words, passive, chains of de, split predicates, clichés; layers="all" turns on every layer allowed. The layers overlap (a compound preposition is made of stopwords), so pick the ones you need.

A sentence is long from 30 words, the bound of the Spanish-language guides to plain language (the Comunidad de Madrid and the Gobierno de la Ciudad de Buenos Aires), and a word is complex from four syllables; complex_syl_factor=3, the bound of BasicStats, shows the words of n_complex_words.

Alliteration

An alliteration is looked for inside a sentence as a run of two neighbouring words or more with the same consonant sound in their stems. The sounds are the ones of the transcription, so casa and queso repeat k; the stem is the common start of the transcriptions of the word form and of its lemma (cantaban - k a n t a), which leaves out the endings that repeat by agreement (las casas blancas). The probability of a run under an independent spread of the sounds is the product over its words of the probability to meet the consonant among the sounds of the stem, \(1 - (1 - f)^n\), where \(f\) is the frequency of the sound in the corpus of literature and \(n\) the number of the sounds of the stem; a run is highlighted when the probability is below alliteration_threshold. A repetition of a rare sound shows in two or three words (deje la abeja), while the s of los suspiros se escapan is too frequent to count in two words. Words shorter than three letters (de, la, el), stopwords (que, los, con) and words without vowels neither break nor continue a run. The alliteration index of PhonStats measures how the repetitions cluster over the whole text; the highlighting shows where they are.

The class extends the HighlightedText of the anyTS core with the hooks of Spanish from ests.constants: the layers with their names in the legend are HIGHLIGHT_LAYERS_DESC, the layers on by default HIGHLIGHT_DEFAULT_LAYERS, the annotations of a Doc a layer needs HIGHLIGHT_LAYER_ANNOTATIONS and the styles of the layers HIGHLIGHT_LAYER_STYLES; the prefix of the CSS classes is ests. The words of a string are those of WordsExtractor, without the punctuation, and its sentences those of SentsExtractor; a Doc without the sentence boundaries (a blank pipeline, a pipeline without the parser) is split into sentences by the rules of SentsExtractor too. A string with a character without a UTF-8 form raises SourceError from the Spanish tokenizer. The fragments of every layer are found by the finders of esTS.

CodeResult

import spacy

from ests.visualizers import highlight

nlp = spacy.load("es_core_news_sm")
text = (
    "El proyecto, elaborado durante el verano, fue aprobado por el consejo sin debate. "
    "El aumento de la eficiencia del uso de los recursos públicos se analizó, siguiendo "
    "las normas del reglamento. "
    "Los representantes de los ministerios regionales no consiguieron hacer una revisión "
    "conjunta de las cuestiones de financiación y de reparto de responsabilidades entre los "
    "organismos, puesto que cada uno de ellos defendía su propia interpretación de las "
    "disposiciones del acuerdo. "
    "En el marco de la reunión se procedió a la votación, y la decisión quedó aplazada "
    "hasta la próxima sesión."
)
ht = highlight(nlp(text))
ht.counts
ht.highlights[:2]
highlight(nlp(text), layers="all").counts

{'long_sents': 1,
 'complex_words': 16,
 'passive': 2,
 'de_chains': 3,
 'split_predicates': 2,
 'cliches': 1}
(Highlight(start=13, end=22, layer='complex_words', note='complex word, 5 syllables'),
 Highlight(start=42, end=54, layer='passive', note='passive with ser'))
{'long_sents': 1,
 'complex_words': 16,
 'rare_words': 0,
 'passive': 2,
 'participle_clauses': 2,
 'gerund_clauses': 1,
 'de_chains': 3,
 'split_predicates': 2,
 'verbal_nouns': 9,
 'compound_prepositions': 1,
 'cliches': 1,
 'anaphoric_mismo': 0,
 'stopwords': 48,
 'parentheticals': 0,
 'connectors': 3,
 'alliteration': 0}

In Jupyter the object ht displays like this (hover over a fragment to see its note):

Long sentences1Complex words16Passive2Chains of de3Split predicates2Clichés1

El proyecto, elaborado durante el verano, fue aprobado por el consejo sin debate. El aumento de la eficiencia del uso de los recursos públicos se analizó, siguiendo las normas del reglamento. Los representantes de los ministerios regionales no consiguieron hacer una revisión conjunta de las cuestiones de financiación y de reparto de responsabilidades entre los organismos, puesto que cada uno de ellos defendía su propia interpretación de las disposiciones del acuerdo. En el marco de la reunión se procedió a la votación, y la decisión quedó aplazada hasta la próxima sesión.

A string without a model of spaCy allows every layer but the syntactic ones and verbal_nouns: long sentences, complex words and clichés by default, and the alliteration on request.

CodeResult

from ests.visualizers import highlight

# Antonio Machado, Poesías completas (the corpus of literature)
poem = "El cierzo corre por el campo yerto alborotando en blancos torbellinos la nieve silenciosa."
highlight(poem, layers="alliteration").highlights
text = "Se procedió a la revisión del expediente en el marco del plan a la mayor brevedad."
highlight(text, layers=["compound_prepositions", "cliches"]).highlights

(Highlight(start=35, end=78, layer='alliteration', note='alliteration on /b/ (b, v)'),)
(Highlight(start=3, end=13, layer='cliches', note='cliché: «proceder a»'),
 Highlight(start=41, end=56, layer='compound_prepositions', note='compound preposition: «en el marco de»'),
 Highlight(start=62, end=81, layer='cliches', note='cliché: «a la mayor brevedad»'))

Parameters

Parameter	Type	Default	Description
source	str / Doc	-	Data source (a string or a Doc object)
layers	list[str] / str	None	Layers of the highlighting; if not given, the default layers the source allows; "all" - every layer allowed

The parameters of the layers of esTS:

Parameter	Type	Default	Description
long_sent_word_factor	int	30	Minimum number of words of a long sentence
complex_syl_factor	int	4	Minimum number of syllables of a complex word
stopwords	list[str] / set[str]	None	List or set of stopwords; if not given, STOPWORDS and the one-word parenthetical expressions
cliches	list[str] / set[str]	None	List or set of clichés; if not given, OFFICIALESE_CLICHES
alliteration_threshold	float	0.001	Probability of a repetition of a consonant under an independent spread of the sounds, below which the repetition is alliteration

A number of words or syllables that is not an integer of at least one or a threshold outside (0, 1] raises ParameterError, stopwords or clichés that are not a list or a set of strings SourceTypeError.

Attributes

Attribute	Type	Description
text	str	Text of the data source
layers	tuple[str]	Layers turned on, in the order of drawing
highlights	tuple[Highlight]	Highlighted fragments, sorted by their start and then by descending end
counts	dict[str, int]	Number of fragments of every layer

Highlight(start, end, layer, note='')

A fragment of the highlighting: an immutable object with the parameters as its fields.

Field	Type	Default	Description
start	int	-	Position of the first character of the fragment in the text
end	int	-	Position after the last character of the fragment
layer	str	-	Layer of the highlighting
note	str	''	Explanation of the fragment for the tooltip

The notes of esTS are in English: they give the number of words of the sentence or of the clause, of syllables of the word, the length of the chain, the phrase of the list, the class and the kind of the connector, the sound of the alliteration and its letters.

Methods

to_html(legend=True, css=True) returns the HTML markup of the highlighted text: a div of the class <prefix>-highlight with the legend, its counts and the text, where a highlighted segment is a span of the classes <prefix>-hl and <prefix>-hl-<layer> with the notes in its title; <prefix> is css_prefix. Overlapping fragments of different layers give segments with several classes, in the order of drawing. Line breaks (\n, \r\n, \r) become character references, one per break, so the markup can go into Markdown.

Parameter	Type	Default	Description
legend	bool	True	Add the legend with the counts of the fragments
css	bool	True	Add the styles of the layers

The class method css() returns the styles to_html adds: the container, the legend and the text under the classes of css_prefix, the declarations of layer_styles for every layer, and a dark colour of the text on the layers with a background.

CodeResult

from ests.visualizers import highlight

poem = "El cierzo corre por el campo yerto alborotando en blancos torbellinos la nieve silenciosa."
highlight(poem, layers="alliteration").to_html(legend=False, css=False)

'<div class="ests-highlight"><div class="ests-highlight-text">El cierzo corre por el campo yerto <span class="ests-hl ests-hl-alliteration" title="alliteration on /b/ (b, v)">alborotando en blancos torbellinos la nieve</span> silenciosa.</div></div>'

Helpers of the layers

The words and the sentences the finders of the layers take; the helpers are in ests.visualizers.highlight.

Word(start, end, text, pos=None, lemma=None)

A word with its position in the text and, when known, its part of speech and lemma.

Sent(start, end, n_words)

A sentence with its position and number of words.

group_words_by_sents(words, sents)

The words of every sentence; the words out of the sentences are skipped.

Finders of the layers

The fragments of every layer are found by a function of ests.visualizers.highlight that returns a list of Highlight: from the words and the sentences of the text, or from the Doc for the syntactic layers (SYNTAX_FINDERS gives them by layer). All but the passive, the participial and gerund clauses, the chains of de and the anaphoric el mismo are the finders of the core with the hooks of Spanish.

Long sentences and complex words

find_long_sents(sents, long_sent_word_factor)

The layer long_sents: the sentences of long_sent_word_factor words or more, the note with their number of words.

find_complex_words(words, complex_syl_factor)

The layer complex_words: the words of complex_syl_factor syllables or more by count_syllables, the note with their number of syllables.

Parameter	Type	Default	Description
sents	list[Sent]	-	Sentences with their positions and numbers of words
long_sent_word_factor	int	-	Minimum number of words of a long sentence
words	list[Word]	-	Words with their positions
complex_syl_factor	int	-	Minimum number of syllables of a complex word

Raises:

• ParameterError - a number of words or syllables that is not an integer of at least one.

CodeResult

from ests.utils import iter_text_words
from ests.visualizers.highlight import Sent, Word, find_complex_words, find_long_sents

text = "El felinólogo examinó al gato. Lo devolvieron."
words = [Word(*word) for word in iter_text_words(text)]
find_long_sents([Sent(0, 30, 5), Sent(31, 46, 2)], 5)
find_complex_words(words, 4)

[Highlight(start=0, end=30, layer='long_sents', note='long sentence, 5 words')]
[Highlight(start=3, end=13, layer='complex_words', note='complex word, 5 syllables'),
 Highlight(start=14, end=21, layer='complex_words', note='complex word, 4 syllables'),
 Highlight(start=34, end=45, layer='complex_words', note='complex word, 4 syllables')]

Single words

find_rare_words(words)

The layer rare_words: the words whose key (lemma_key) is beyond the 10,000 most frequent lemmas (get_rank), as p_beyond_top10000 of LexicalStats counts them; a word tagged PROPN keeps its form. Only the words of two Spanish letters or more that are not stopwords are looked at.

find_stopwords(words, stopwords=None)

The layer stopwords: the words of stopwords in any case or, without the list, the stopwords of is_stopword.

find_verbal_nouns(words)

The layer verbal_nouns: the words tagged NOUN whose lemma is derived from a verb, as for the verbal nouns of StyleStats.

Parameter	Type	Default	Description
words	list[Word]	-	Words with their positions; with their parts of speech and lemmas for find_verbal_nouns
stopwords	list[str] / set[str]	None	List or set of stopwords; is_stopword if not given

Raises:

• SourceTypeError - stopwords that are not a list or a set of strings.

CodeResult

from ests.utils import iter_text_words
from ests.visualizers.highlight import Word, find_rare_words, find_stopwords

text = "El felinólogo examinó minuciosamente al gato."
words = [Word(*word) for word in iter_text_words(text)]
find_rare_words(words)
find_stopwords(words)

[Highlight(start=3, end=13, layer='rare_words', note='rare word: beyond the top 10000'),
 Highlight(start=22, end=36, layer='rare_words', note='rare word: beyond the top 10000')]
[Highlight(start=0, end=2, layer='stopwords', note='stopword'),
 Highlight(start=37, end=39, layer='stopwords', note='stopword')]

Phrases

find_phrase_highlights(words, phrases, layer, label)

The fragments of the phrases of a list, matched with their contractions and the forms of their verbs (expand_phrases); the note gives the label and the phrase of the list (cliché: «proceder a»).

find_compound_prepositions(words)

The layer compound_prepositions: the phrases of COMPOUND_PREPOSITIONS.

find_cliches(words, cliches=None)

The layer cliches: the phrases of cliches, of OFFICIALESE_CLICHES if not given.

find_parentheticals(words)

The layer parentheticals: the phrases of PARENTHETICALS.

Parameter	Type	Default	Description
words	list[Word]	-	Words with their positions
phrases	list[str] / set[str]	-	Phrases, words separated by spaces
layer	str	-	Layer of the highlighting
label	str	-	Label of the fragment in the note
cliches	list[str] / set[str]	None	List or set of clichés

Raises:

• SourceTypeError - phrases or clichés that are not a list or a set of strings.

CodeResult

from ests.utils import iter_text_words
from ests.visualizers.highlight import Word, find_cliches, find_compound_prepositions

text = "Se procedió a la revisión del expediente en el marco del plan."
words = [Word(*word) for word in iter_text_words(text)]
find_cliches(words)
find_compound_prepositions(words)

[Highlight(start=3, end=13, layer='cliches', note='cliché: «proceder a»')]
[Highlight(start=41, end=56, layer='compound_prepositions', note='compound preposition: «en el marco de»')]

Connectors

find_connector_highlights(words, sents, text=None, doc=None)

The layer connectors: the connectors of CohesionStats inside every sentence, found by its matcher SpanishConnectorMatcher, a one-word connector checked by its part of speech when the words have it; the note gives the connector, its class and its kind. With the text, the rules of the matcher read no word across a punctuation mark between two words, so así, que is no connector; with the Doc, no connector starts at a si that opens no condition, as in CohesionStats (is_connector_blocked: no sé si vendrá). HighlightedText passes the text of its source and its Doc; in a string, which it does not parse, only the rules of the matcher drop a si (como si fuera).

Parameter	Type	Default	Description
words	list[Word]	-	Words with their positions and parts of speech
sents	list[Sent]	-	Sentences with their positions
text	str	None	Text of the words, for the punctuation marks between them
doc	Doc	None	Doc of the words, for the rules by its tokens

Raises:

• SourceTypeError - a text that is not a string.

CodeResult

from ests.utils import iter_text_words
from ests.visualizers.highlight import Sent, Word, find_connector_highlights

text = "Sin embargo, el plan no se aprobó."
words = [Word(*word) for word in iter_text_words(text)]
find_connector_highlights(words, [Sent(0, len(text), len(words))], text)

[Highlight(start=0, end=11, layer='connectors', note='connector «sin embargo»: adversative, secondary')]

Alliteration

get_stem_sounds(word)

The sounds of the stem of a word form: the common start of the transcriptions of the form and of its lemma (cantaban - k a n t a, cogió - k o x). When it is shorter than two sounds, as for the suppletive forms (fue - ser), the stem is the whole form.

calc_alliteration_runs(text, threshold=0.001)

The runs of calc_alliteration_runs of the core over the stems of get_stem_sounds and the consonant sounds of ALLITERATION_FREQUENCIES, their frequencies in SOUND_FREQUENCIES; the words shorter than three letters, the stopwords and the words without vowels neither break nor continue a run. A run is a triple of the index of its first word, the index after its last word and its sound.

find_alliteration(words, threshold=0.001, sents=None)

The layer alliteration: the runs of calc_alliteration_runs inside every sentence, in the whole text without the sentences; the note gives the sound and the letters that write it (alliteration on /b/ (b, v)).

Parameter	Type	Default	Description
word	str	-	Word form in lower case
text	list[str]	-	List of words
threshold	float	0.001	Probability threshold (ALLITERATION_THRESHOLD of anyts.constants)
words	list[Word]	-	Words with their positions
sents	list[Sent]	None	Sentences with their positions

Raises:

• SourceTypeError - a word that is not a string, words of calc_alliteration_runs that are not a list of strings;
• ParameterError - a threshold outside (0, 1].

CodeResult

from ests.visualizers.highlight import calc_alliteration_runs, get_stem_sounds

get_stem_sounds("cantaban"), get_stem_sounds("fue")
calc_alliteration_runs(["deje", "la", "abeja"])
calc_alliteration_runs(["los", "suspiros", "se", "escapan"])

(('k', 'a', 'n', 't', 'a'), ('f', 'u', 'e'))
[(0, 3, 'x')]
[]

Syntactic layers

find_passive(doc)

The layer passive: every passive form of is_passive with its auxiliary ser (fue construida) or its se (se construyó); the note tells the passive with se from the one with ser and marks a passive with ser without an agent (is_agentless).

find_participle_clauses(doc)

The layer participle_clauses: the subtree of every head of a participial clause, the note with its number of words.

find_gerund_clauses(doc)

The layer gerund_clauses: the subtree of every head of a gerund clause, the note with its number of words.

find_de_chains(doc)

The layer de_chains: the chains of de of two complements or more, a fragment from the head to the last complement (el aumento de la eficiencia del uso de los recursos), the note with the length of the chain.

find_split_predicate_highlights(doc)

The layer split_predicates: the pairs of a light verb and a noun of find_split_predicates, a fragment from the first word of the pair to the last (hacer una revisión), the note with their lemmas.

Parameter	Type	Default	Description
doc	Doc	-	Doc object with a dependency parse and the lemmas

CodeResult

from ests.utils import get_nlp
from ests.visualizers.highlight import (
    find_de_chains,
    find_gerund_clauses,
    find_participle_clauses,
    find_passive,
    find_split_predicate_highlights,
)

doc = get_nlp()(
    "El informe, redactado por el comité, fue aprobado tras hacer una revisión, "
    "siguiendo las normas del uso de los recursos."
)
find_passive(doc)
find_participle_clauses(doc)
find_gerund_clauses(doc)
find_de_chains(doc)
find_split_predicate_highlights(doc)

[Highlight(start=37, end=49, layer='passive', note='passive with ser, no agent')]
[Highlight(start=12, end=35, layer='participle_clauses', note='participial clause, 4 words')]
[Highlight(start=75, end=119, layer='gerund_clauses', note='gerund clause, 8 words')]
[Highlight(start=89, end=119, layer='de_chains', note='chain of 2 complements with de')]
[Highlight(start=55, end=73, layer='split_predicates', note='split predicate: hacer revisión')]

Anaphoric el mismo

is_anaphoric_mismo(token)

Checks whether a word is an anaphoric el mismo: a form of mismo that heads a noun phrase of its own with its article (el, la, los, las, del, al) and no other dependent but a clause after a comma, as an argument of a verb or a noun (se archivó el mismo, la duración del mismo, contra la misma, que pone fin a la vía administrativa, cabe recurso). The Diccionario panhispánico de dudas takes this use of the administrative and the journalistic style for unnecessary and suggests a demonstrative, a possessive or a personal pronoun instead (su duración, dentro de ella). The identity is no anaphora: el mismo día, es el mismo, lo mismo, él mismo, a form after ser, estar, parecer, permanecer or resultar, by the lemma of the model or of simplemma (no parecía la misma, siempre serás el mismo), right before que (el mismo que el año pasado, while del mismo, que... is anaphoric) and before a conjunction and an adjective or a determiner other than an article or a possessive, which share its noun (de la misma o superior categoría, a la misma u otra sede). The models attach some anaphoric forms to their article (en la misma hay oficinas): such a form counts when no noun, adjective, numeral or pronoun follows it in its sentence; an adjective whose noun after it disagrees with the form in gender or number is a verb the model misread (contra la misma cabe recurso, cabe tagged as an adjective of recurso), while la misma angosta calle stays a noun phrase. A token of a Doc without a parse is none. Whitespace and punctuation are no dependents of the form, so a line break after it changes nothing.

find_anaphoric_mismo(doc)

The layer anaphoric_mismo: every form of is_anaphoric_mismo with its article (la misma, del mismo), the two words in the note.

Parameter	Type	Default	Description
token	Token	-	Token
doc	Doc	-	Doc object with a dependency parse, the parts of speech and the lemmas

CodeResult

from ests.utils import get_nlp
from ests.visualizers.highlight import find_anaphoric_mismo, is_anaphoric_mismo

doc = get_nlp()("Se recibió la solicitud el mismo día, y se procedió al estudio de la misma.")
[token.text for token in doc if is_anaphoric_mismo(token)]
find_anaphoric_mismo(doc)

['misma']
[Highlight(start=66, end=74, layer='anaphoric_mismo', note='anaphoric la misma')]