Sentence extraction¶

SentsExtractor(tokenizer=None, min_len=0, max_len=0)

A class for extracting sentences from a text.

Raises:

ParameterError - a length bound that is not an integer, a negative bound, a minimum above the maximum.

The class extends the SentsExtractor of the anyTS core with the hook of Spanish: its default tokenizer, the method sentenize(text), is the rule-based splitter ests.utils.sentenize.

Note

A sentence ends with a period, an exclamation or question mark or an ellipsis, optionally followed by closing quotes or brackets, when the next word starts with an upper-case letter, a digit, an inverted mark ¿ ¡, an opening quote or bracket or a dash, a dash glued to the mark included when a letter or an opening mark follows it on the same line (baja.--Tiene, as in the editions of Project Gutenberg), while a dash glued to the mark before whitespace closes its line of dialogue and stays with it (¡Traidores!--, viuda.-- El niño); a blank line ends a sentence too. A dash before a lower-case word opens a remark of the narrator and keeps the sentence going: ¿Vienes? —preguntó él. is one sentence, ¿Vienes? —Sí. two. Abbreviations (Sr., Dra., p. ej., EE. UU., a. m.), capital initials (J. L. Borges) and list markers at the start of a sentence or a line (1., 2.1., IV.) do not end a sentence, even before an upper-case word: Llegó a las 5 p. m. Luego se fue. stays one sentence. A lower-case word after an ellipsis or an exclamation mark continues the sentence, and a single line break does not split it, so hard-wrapped texts are handled.

CodeResult

from ests import SentsExtractor

text = "¿No tienes cien euros? ¡Ten cien amigos! El Sr. García lo dijo... Y se fue."
SentsExtractor().extract(text)

('¿No tienes cien euros?', '¡Ten cien amigos!', 'El Sr. García lo dijo...', 'Y se fue.')

Parameters¶

Parameter	Type	Default	Description
`tokenizer`	`Pattern` / `Callable`	`None`	Tokenizer or regular expression; by default the `sentenize` method
`min_len`	`int`	`0`	Minimum length of an extracted sentence in characters, `0` for no bound
`max_len`	`int`	`0`	Maximum length of an extracted sentence in characters, `0` for no bound

Note

A regular expression as the tokenizer is a separator for re.split. A blank line - a line break, optional whitespace and another line break - ends a sentence of any tokenizer: the text is split at blank lines first and every part goes to the tokenizer, so a heading or a paragraph without a final mark is a sentence of its own, and so is a part without words, such as a line of asterisks, which the statistics leave out. The sentences are stripped of whitespace at the edges before the length bounds, and empty ones are dropped.

Note

A spaCy pipeline can be passed as the tokenizer: tokenizer=lambda text: (sent.text for sent in nlp(text).sents).

Methods¶

extract(text) extracts sentences from a text.

Parameter	Type	Default	Description
`text`	`str`	-	Text string

A text that is not a string and a tokenizer that cannot be called or does not return strings raise SourceTypeError.

CodeResult

import re

SentsExtractor().extract("Novedades\n\nImportación más rápida\nMenos errores\n\n¡Gracias!")
SentsExtractor(tokenizer=re.compile(r", ")).extract("No tengas 100 euros, ten 100 amigos")

('Novedades', 'Importación más rápida\nMenos errores', '¡Gracias!')
('No tengas 100 euros', 'ten 100 amigos')

Paragraphs¶

split_paragraphs(text)

The paragraphs of a text: its parts between blank lines, stripped of whitespace at their edges; a part of whitespace alone is none. A blank line (BLANK_LINE) is two line breaks - \n, \r\n, \r, U+0085 or U+2028 - with whitespace alone between them, the zero-width space and the byte order mark counting as whitespace there, or a paragraph separator U+2029; it also ends the sentences of SentsExtractor and the paragraphs of CohesionStats with blank_lines. Their words give the sizes of the parts of dispersion by paragraph; a paragraph without words, such as a scene break of asterisks, is still one. A text that is not a string raises SourceTypeError.

Parameter	Type	Default	Description
`text`	`str`	-	Text string

In esTS the function is imported from ests.extractors; the words of the paragraphs give dispersion the sizes of its parts.

CodeResult

from ests.extractors import split_paragraphs

split_paragraphs("Capítulo I\n\nEra una noche oscura.\nLlovía.\n\n* * *\n\nAmaneció.")

['Capítulo I', 'Era una noche oscura.\nLlovía.', '* * *', 'Amaneció.']