Word extraction¶

WordsExtractor(tokenizer=None, filter_punct=True, filter_nums=False, use_lexemes=False, stopwords=None, lowercase=False, ngram_range=(1, 1), min_len=0, max_len=0)

A class for extracting words from a text.

Raises:

ParameterError - an ngram_range that is not a pair of integers or whose lower bound is below one or above the upper one, a length bound that is not an integer, a negative bound, a minimum above the maximum, a switch that is not a bool;
SourceTypeError - stop words that are not a collection of strings.

The class extends the WordsExtractor of the anyTS core with the hooks of Russian: the tokenizer tokenize(text) gives the tokens of the anySeg segmenter by the conventions of UD, the lemma lemmatize(word) is the analysis of the ruMorph morphological analyzer in the context of the sentence (with a tokenizer given, the first analysis), and the number pattern number_pattern also matches ranges, fractions, ordinals and numbers with spaces between the digit groups (2020-2021, 1,5, 3-й, 1 250 000). The period of an abbreviation stays with the word only when it tells the words apart: руб. and руб are one word, им. keeps its period. Stress marks and soft hyphens are removed from the text before any tokenizer.

CodeResult

from ruts import WordsExtractor

text = "Не имей 100 рублей, а имей 100 друзей"

we = WordsExtractor(use_lexemes=True, stopwords=["не", "а"], filter_nums=True, ngram_range=(1, 2))
we.extract(text)

('иметь', 'рубль', 'иметь', 'друг', 'иметь_рубль', 'рубль_иметь', 'иметь_друг')

Parameters¶

Parameter	Type	Default	Description
`tokenizer`	`Pattern` / `Callable`	`None`	Tokenizer or regular expression; by default the `tokenize` method
`filter_punct`	`bool`	`True`	Filter punctuation marks
`filter_nums`	`bool`	`False`	Filter numbers matched by `number_pattern`
`use_lexemes`	`bool`	`False`	Use word lemmas of the `lemmatize` method
`stopwords`	`Collection[str]`	`None`	Stop words, compared case-insensitively
`lowercase`	`bool`	`False`	Convert words to lower case
`ngram_range`	`tuple[int, int]`	`(1, 1)`	Lower and upper bound of the N-gram size
`min_len`	`int`	`0`	Minimum length of an extracted word in characters, `0` for no bound
`max_len`	`int`	`0`	Maximum length of an extracted word in characters, `0` for no bound

Note

A regular expression as the tokenizer is a separator for re.split. The filters are applied in order: punctuation, numbers, lemmatization, lower case, stop words, word length. A punctuation mark is a token of marks and symbols only, multi-character ones included: ?!, --, …, € (anyts.utils.is_punctuation). Empty and whitespace tokens are dropped before the filters. N-grams join the words with _.

Methods¶

extract(text) extracts words from a text.

Parameter	Type	Default	Description
`text`	`str`	-	Text string

A text that is not a string and a tokenizer that cannot be called or does not return strings raise SourceTypeError.

get_most_common(n=10) returns the top words of the text as a list of (word, frequency) pairs, the most frequent first.

Parameter	Type	Default	Description
`n`	`int`	`10`	Number of top words

A number that is not an integer of at least one raises ParameterError.

Warning

The method must be called after words have been extracted with extract.

CodeResult

we.get_most_common(3)

[('иметь', 2), ('рубль', 1), ('друг', 1)]