Sentence extraction¶
SentsExtractor(tokenizer=None, min_len=0, max_len=0)
A class for extracting sentences from a text.
Raises:
ParameterError- a length bound that is not an integer, a negative bound, a minimum above the maximum.
The class extends the SentsExtractor of the anyTS core with the hook of Russian: its default tokenizer, the method sentenize(text), gives the sentences of the anySeg segmenter: a line without a final mark (a heading, a list item) and a sentence after a period without a space are sentences of their own.
from ruts import SentsExtractor
text = "Не имей 100 рублей, а имей 100 друзей. Так говорил проф. Иванов... А вы?"
SentsExtractor().extract(text)
('Не имей 100 рублей, а имей 100 друзей.', 'Так говорил проф. Иванов...', 'А вы?')
Parameters¶
| Parameter | Type | Default | Description |
|---|---|---|---|
tokenizer |
Pattern / Callable |
None |
Tokenizer or regular expression; by default the sentenize method |
min_len |
int |
0 |
Minimum length of an extracted sentence in characters, 0 for no bound |
max_len |
int |
0 |
Maximum length of an extracted sentence in characters, 0 for no bound |
Note
A regular expression as the tokenizer is a separator for re.split. A blank line - a line break, optional whitespace and another line break - ends a sentence of any tokenizer: the text is split at blank lines first and every part goes to the tokenizer, so a heading or a paragraph without a final mark is a sentence of its own, and so is a part without words, such as a line of asterisks, which the statistics leave out. The sentences are stripped of whitespace at the edges before the length bounds, and empty ones are dropped.
Note
A spaCy pipeline can be passed as the tokenizer: tokenizer=lambda text: (sent.text for sent in nlp(text).sents).
Methods¶
extract(text) extracts sentences from a text.
| Parameter | Type | Default | Description |
|---|---|---|---|
text |
str |
- | Text string |
A text that is not a string and a tokenizer that cannot be called or does not return strings raise SourceTypeError.
import re
SentsExtractor().extract("Что нового\n\nБыстрый импорт\nМеньше ошибок\n\nСпасибо за внимание!")
SentsExtractor(tokenizer=re.compile(r", ")).extract("Не имей 100 рублей, а имей 100 друзей")
('Что нового', 'Быстрый импорт', 'Меньше ошибок', 'Спасибо за внимание!')
('Не имей 100 рублей', 'а имей 100 друзей')