Skip to content

Character N-gram extraction

CharNgramsExtractor(n=2, lowercase=False, within_words=False, tokenizer=None)

A class for extracting character N-grams from a text - sequences of N characters taken with a sliding window over the string. Character N-grams are a standard feature of stylometry and authorship attribution (Stamatatos 2009) and can replace words in measures such as Burrows's Delta.

Whitespace runs are collapsed into a single space, punctuation marks are kept. With within_words=True N-grams do not cross word boundaries: the text is split into words by the tokenizer, punctuation is dropped, and words shorter than N yield no N-grams.

Raises:

  • ParameterError - an n that is not an integer of at least one, a lowercase or within_words that is not a bool.

The class extends the CharNgramsExtractor of the anyTS core with the hook of Russian: the words for within_words (the method tokenize(text)) are those of WordsExtractor, the tokens of the anySeg segmenter. Stress marks and soft hyphens are removed from the text before extraction. The list of N-grams is passed to delta as text units instead of words.

from ruts import CharNgramsExtractor

text = "Кот сидел  на окне, а пёс - на полу."

ce = CharNgramsExtractor(n=3, lowercase=True)
ce.extract(text)[:6]
CharNgramsExtractor().extract(text)[:8]
CharNgramsExtractor(n=3, lowercase=True, within_words=True).extract(text)

('кот', 'от ', 'т с', ' си', 'сид', 'иде')
('Ко', 'от', 'т ', ' с', 'си', 'ид', 'де', 'ел')
('кот', 'сид', 'иде', 'дел', 'окн', 'кне', 'пёс', 'пол', 'олу')

Parameters

Parameter Type Default Description
n int 2 N-gram length in characters
lowercase bool False Convert the text to lower case
within_words bool False Take N-grams only inside words
tokenizer Pattern / Callable None Word tokenizer for within_words or a regular expression; by default the tokenize method

Methods

extract(text) extracts N-grams from a text.

Parameter Type Default Description
text str - Text string

A text that is not a string raises SourceTypeError, and so does, with within_words, a tokenizer that cannot be called or does not return strings.

get_most_common(n=10) returns the top N-grams of the text as a list of (N-gram, frequency) pairs, the most frequent first.

Parameter Type Default Description
n int 10 Number of top N-grams

A number that is not an integer of at least one raises ParameterError.

Warning

The method must be called after N-grams have been extracted with extract.

ce.get_most_common(2)

[(' на', 2), ('на ', 2)]