Sentence extraction

SentsExtractor(tokenizer=None, min_len=0, max_len=0)

A class for extracting sentences from a text.

Raises:

• ParameterError - a length bound that is not an integer, a negative bound, a minimum above the maximum.

CodeResult

from anyts import SentsExtractor

SentsExtractor().extract('It rains. Does it? He said "Yes!" Then he left.')

('It rains.', 'Does it?', 'He said "Yes!"', 'Then he left.')

Parameters

Parameter	Type	Default	Description
tokenizer	Pattern / Callable	None	Tokenizer or regular expression; by default the sentenize method
min_len	int	0	Minimum length of an extracted sentence in characters, 0 for no bound
max_len	int	0	Maximum length of an extracted sentence in characters, 0 for no bound

Note

A regular expression as the tokenizer is a separator for re.split. A blank line (BLANK_LINE) ends a sentence of any tokenizer: the text is split at blank lines first and every part goes to the tokenizer, so a heading or a paragraph without a final mark is a sentence of its own, and so is a part without words, such as a line of asterisks, which the statistics leave out. The sentences are stripped of whitespace, control characters and invisible separators (the soft hyphen, zero-width characters, the byte order mark) at the edges before the length bounds, and empty ones are dropped.

Note

A spaCy pipeline can be passed as the tokenizer: tokenizer=lambda text: (sent.text for sent in nlp(text).sents).

Methods

extract(text) extracts sentences from a text.

Parameter	Type	Default	Description
text	str	-	Text string

A text that is not a string and a tokenizer that cannot be called or does not return strings raise SourceTypeError.

CodeResult

import re

SentsExtractor().extract("What is new\n\nFaster import\nFewer bugs\n\nThanks for reading!")
SentsExtractor(tokenizer=re.compile(r", ")).extract("Rain today, sun tomorrow")

('What is new', 'Faster import\nFewer bugs', 'Thanks for reading!')
('Rain today', 'sun tomorrow')

Sentences of the parts of a text

iter_part_sents(iter_sents, text)

Runs the splitter iter_sents on every part of a text between blank lines (BLANK_LINE) and gives the sentences at their positions in the whole text, so that a blank line ends a sentence whatever the splitter, as in SentsExtractor.

Parameter	Type	Default	Description
iter_sents	Callable	-	Splitter giving the start, the end and the text of every sentence of a string
text	str	-	Text string

A text that is not a string raises SourceTypeError, and so does an iter_sents that cannot be called or does not give the sentences of a part in order, as triples of integer bounds within it and a string.

CodeResult

import re

from anyts.extractors import iter_part_sents


def periods(text):
    return [(m.start(), m.end(), m.group()) for m in re.finditer(r"[^.\s][^.]*\.?", text)]


list(iter_part_sents(periods, "One. Two\n\nThree."))

[(0, 4, 'One.'), (5, 8, 'Two'), (10, 16, 'Three.')]

Paragraphs

split_paragraphs(text)

The paragraphs of a text: its parts between blank lines, stripped of whitespace, control characters and invisible separators (the soft hyphen, zero-width characters, the byte order mark) at their edges; a part of nothing else is none. A blank line (BLANK_LINE) is two line breaks - \n, \r\n, \r, the vertical tab, the form feed, U+0085 or U+2028 - with nothing but such characters between them, or a paragraph separator U+2029: a paragraph break of anySeg, so these are the paragraphs of a document too; it also ends the sentences of SentsExtractor and the paragraphs of CohesionStats with blank_lines. Their words give the sizes of the parts of dispersion by paragraph; a paragraph without words, such as a scene break of asterisks, is still one. A text that is not a string raises SourceTypeError.

Parameter	Type	Default	Description
text	str	-	Text string

CodeResult

from anyts.extractors import split_paragraphs

split_paragraphs("Chapter 1\n\nThe cat slept.\nThen it woke.\n \n* * *\n\nThe end.")

['Chapter 1', 'The cat slept.\nThen it woke.', '* * *', 'The end.']

Ready sentences

FixedSentsExtractor(sents)

A SentsExtractor that gives the sentences it was made with for any text, so a class that takes a sentence extractor gets known sentences without splitting the text again.

Parameter	Type	Default	Description
sents	list[str]	-	Sentences

Sentences that are not a list of strings raise SourceTypeError.

CodeResult

from anyts.extractors import FixedSentsExtractor

FixedSentsExtractor(["It rains.", "Yes!"]).extract("any text")

('It rains.', 'Yes!')

For library developers

Language hooks of a subclass

The default tokenizer is the method sentenize(text), which a language library overrides in a subclass. In the core it splits the text at whitespace after ., !, ? or …, optionally followed by a closing quote or bracket; it knows no abbreviations, so e.g. this is two sentences. A piece without words, such as the dots of a spaced ellipsis . . . or a lone !, stays with the sentence before it, or with the one after it at the start of the text. The messages of its errors come from the class attribute messages, MESSAGES by default.