Word extraction

WordsExtractor(tokenizer=None, filter_punct=True, filter_nums=False, use_lexemes=False, stopwords=None, lowercase=False, ngram_range=(1, 1), min_len=0, max_len=0)

A class for extracting words from a text.

Raises:

• ParameterError - an ngram_range that is not a pair of integers or whose lower bound is below one or above the upper one, a length bound that is not an integer, a negative bound, a minimum above the maximum, a switch that is not a bool;
• SourceTypeError - stop words that are not a collection of strings.

The class extends the WordsExtractor of the anyTS core with the hooks of Spanish:

Hook	Spanish
tokenize(text)	the rule-based tokenizer of the spaCy Spanish language class, ests.utils.tokenize
lemmatize(word)	the lemmas of simplemma, ests.utils.lemmatize
number_pattern	signed numbers, ranges, fractions, dates, times, percentages and ordinals: -5, +7, 1990-1995, 1.500,50, 12/03/2020, 3:30, 10%, 3.º, 1.ª, 2do

Note

The tokenizer needs no trained model. Punctuation marks, including ¿ and ¡, numbers like 1.500,50, 3.º, 1990-1995 and abbreviations like Sr., EE. UU. are single tokens, and the abbreviations of ests.constants.TOKENIZER_ABBREVIATIONS in every case (sr., SR., ee. uu., Pág., a.m.); words with enclitic pronouns (dámelo) are not split. The dashes of a dialogue glued to the words are split off: --No, sí--dijo, ―dijo él―. give the words No, sí, dijo, él, also next to the underscores of italics (--_Siguro_), while a hyphen between letters (franco-alemán) or before a digit (-5) stays in its token; a suffix quoted with its hyphen (-mente) loses it as well, and so does the number of an item of a list (Artículo 1.- El objeto gives the number 1). A byte order mark at the start of a text, as in a file read with utf-8 instead of utf-8-sig, is split off and dropped. ests.utils.add_dash_rules(nlp) adds the same rules to the ones a spaCy pipeline of one's own has; the model of get_nlp has them already, and the components of the library add them to their pipeline.

Note

simplemma works from a dictionary without a trained model: a known form is mapped to its lower-case lemma (Tienes - tener, NIÑOS - niño), an unknown form is returned unchanged (Madrid, dámelo).

CodeResult

from ests import WordsExtractor

text = "No tengas 100 euros, ten 100 amigos"

we = WordsExtractor(use_lexemes=True, stopwords=["no"], filter_nums=True, ngram_range=(1, 2))
we.extract(text)

('tener', 'euro', 'tener', 'amigo', 'tener_euro', 'euro_tener', 'tener_amigo')

Parameters

Parameter	Type	Default	Description
tokenizer	Pattern / Callable	None	Tokenizer or regular expression; by default the tokenize method
filter_punct	bool	True	Filter punctuation marks
filter_nums	bool	False	Filter numbers matched by number_pattern
use_lexemes	bool	False	Use word lemmas of the lemmatize method
stopwords	Collection[str]	None	Stop words, compared case-insensitively
lowercase	bool	False	Convert words to lower case
ngram_range	tuple[int, int]	(1, 1)	Lower and upper bound of the N-gram size
min_len	int	0	Minimum length of an extracted word in characters, 0 for no bound
max_len	int	0	Maximum length of an extracted word in characters, 0 for no bound

Note

A regular expression as the tokenizer is a separator for re.split. The filters are applied in order: punctuation, numbers, lemmatization, lower case, stop words, word length. A punctuation mark is a token of marks and symbols only, multi-character ones included: ?!, --, …, € (anyts.utils.is_punctuation). Empty and whitespace tokens are dropped before the filters. N-grams join the words with _.

Note

A ready stop word list is spacy.lang.es.stop_words.STOP_WORDS; note that it also holds frequent verbs like tener.

Methods

extract(text) extracts words from a text.

Parameter	Type	Default	Description
text	str	-	Text string

A text that is not a string and a tokenizer that cannot be called or does not return strings raise SourceTypeError.

With the default tokenizer, a text with a character without a UTF-8 form, as a file read with the wrong encoding gives, raises SourceError, and so do the classes and functions that extract the words of a string with it.

get_most_common(n=10) returns the top words of the text as a list of (word, frequency) pairs, the most frequent first.

Parameter	Type	Default	Description
n	int	10	Number of top words

A number that is not an integer of at least one raises ParameterError.

Warning

The method must be called after words have been extracted with extract.

CodeResult

we.get_most_common(3)

[('tener', 2), ('euro', 1), ('amigo', 1)]