Skip to content

Character N-gram extraction

CharNgramsExtractor(n=2, lowercase=False, within_words=False, tokenizer=None)

A class for extracting character N-grams from a text - sequences of N characters taken with a sliding window over the string. Character N-grams are a standard feature of stylometry and authorship attribution (Stamatatos 2009) and can replace words in measures such as Burrows's Delta.

Whitespace runs are collapsed into a single space, punctuation marks are kept. With within_words=True N-grams do not cross word boundaries: the text is split into words by the tokenizer, punctuation is dropped, and words shorter than N yield no N-grams.

Raises:

  • ParameterError - an n that is not an integer of at least one, a lowercase or within_words that is not a bool.

The class extends the CharNgramsExtractor of the anyTS core with the hook of Spanish: its default word tokenizer for within_words, the method tokenize(text), is the default tokenizer of WordsExtractor, the rule-based tokenizer of the spaCy Spanish language class ests.utils.tokenize. The list of N-grams is passed to delta as text units instead of words.

from ests import CharNgramsExtractor

text = "El gato dormía  en la ventana, y el perro - en el suelo."

ce = CharNgramsExtractor(n=3, lowercase=True)
ce.extract(text)[:6]
CharNgramsExtractor().extract(text)[:8]
CharNgramsExtractor(n=4, lowercase=True, within_words=True).extract(text)

('el ', 'l g', ' ga', 'gat', 'ato', 'to ')
('El', 'l ', ' g', 'ga', 'at', 'to', 'o ', ' d')
('gato', 'dorm', 'ormí', 'rmía', 'vent', 'enta', 'ntan', 'tana', 'perr', 'erro', 'suel', 'uelo')

Parameters

Parameter Type Default Description
n int 2 N-gram length in characters
lowercase bool False Convert the text to lower case
within_words bool False Take N-grams only inside words
tokenizer Pattern / Callable None Word tokenizer for within_words or a regular expression; by default the tokenize method

Methods

extract(text) extracts N-grams from a text.

Parameter Type Default Description
text str - Text string

A text that is not a string raises SourceTypeError, and so does, with within_words, a tokenizer that cannot be called or does not return strings.

get_most_common(n=10) returns the top N-grams of the text as a list of (N-gram, frequency) pairs, the most frequent first.

Parameter Type Default Description
n int 10 Number of top N-grams

A number that is not an integer of at least one raises ParameterError.

Warning

The method must be called after N-grams have been extracted with extract.

ce.get_most_common(2)

[('el ', 3), (' en', 2)]