Vocabulary growth and frequency spectrum

Two plots about the word distribution of a text complementing Zipf's law: vocabulary growth with text length by Heaps' law and the frequency spectrum - how many lexemes occur exactly once, twice, three times. The functions take axes ax and return Axes.

The functions are those of the anyTS core with the Russian labels of ruts.constants.VISUALIZER_LABELS by default; labels puts the given labels over them. The fit of Heaps' law is described in fit_heaps and the spectrum in calc_frequency_spectrum.

Heaps' law

heaps_plot(words, ax=None, labels=None)

The size of the vocabulary \(V\) after every word of the text and the fitted curve \(V(N) = K \cdot N^{\beta}\) of fit_heaps with its parameters in the legend; the curve depends on the order of the words.

Parameter	Type	Default	Description
words	list[str]	-	Words of the text in order
ax	Axes	None	Axes for the plot
labels	dict[str, str]	None	Labels over the defaults: title, xlabel, ylabel, growth (the curve), fit (a format string with k and beta)

Words that are not a list of strings raise SourceTypeError, fewer than two words SourceError; labels that merge_labels refuses and an ax that is not a matplotlib Axes raise ParameterError.

Frequency spectrum

frequency_spectrum_plot(words, ax=None, labels=None)

The number of word types \(V(m)\) that occur exactly \(m\) times (calc_frequency_spectrum) in logarithmic coordinates; the left edge is the hapaxes. The spectrum underlies the measures of diversity of Yule, Sichel, Michéa and Honoré, and its shape shows how far the vocabulary of the text is from being exhausted.

Parameter	Type	Default	Description
words	list[str]	-	Words of the text
ax	Axes	None	Axes for the plot
labels	dict[str, str]	None	Labels over the defaults: title, xlabel, ylabel

Words that are not a list of strings raise SourceTypeError, no words SourceError; labels that merge_labels refuses and an ax that is not a matplotlib Axes raise ParameterError.

The plots of 12 texts of the StalinWorks dataset:

import matplotlib.pyplot as plt

from ruts import WordsExtractor
from ruts.datasets import StalinWorks
from ruts.visualizers import frequency_spectrum_plot, heaps_plot

we = WordsExtractor(use_lexemes=True, lowercase=True, filter_nums=True)
lemmas = [lemma for text in StalinWorks().get_texts(limit=12) for lemma in we.extract(text)]

fig, (left, right) = plt.subplots(1, 2, figsize=(13, 4.5))
heaps_plot(lemmas, ax=left)
frequency_spectrum_plot(lemmas, ax=right)