Zipf's law

zipf(counter, num_words=None, num_labels=10, log=True, show_theory=False, alpha=1.5, show_fit=False, ax=None, labels=None)

Plotting Zipf's law from a counter of the frequencies of words.

Definition

Zipf's law (the rank-frequency law) is an empirical regularity of the frequencies of words in a natural language: with the words of a language, or of a long enough text, ordered by descending frequency, the frequency of the n-th word is roughly inversely proportional to its rank n. The second most frequent word occurs about half as often as the first, the third a third as often, and so on.

The functions are those of the anyTS core with the Russian labels of ruts.constants.VISUALIZER_LABELS by default; labels puts the given labels over them, for instance labels={"title": "Закон Ципфа в SovChLit"}.

The plot of 100 texts of the SovChLit dataset:

from collections import Counter

from ruts import WordsExtractor
from ruts.datasets import SovChLit
from ruts.style_stats import is_stopword
from ruts.visualizers import zipf

text = "\n".join(SovChLit().get_texts(limit=100))
we = WordsExtractor(use_lexemes=True, filter_nums=True)
counts = Counter(word for word in we.extract(text) if not is_stopword(word))
ax = zipf(counts, num_words=100, num_labels=10, log=False, show_theory=True, alpha=1.1)
ax.figure.savefig("zipf.png")

Parameters

Parameter	Type	Default	Description
counter	Counter	-	Counter of the frequencies of words
num_words	int	None	Number of the most frequent words
num_labels	int	10	Number of the words labelled on the plot
log	bool	True	Use a logarithmic scale
show_theory	bool	False	Plot the theoretical Zipf's law
alpha	float	1.5	Exponent α of the theoretical Zipf's law, greater than zero
show_fit	bool	False	Plot the Zipf-Mandelbrot fit \(f(r) = C / (r + q)^s\) of fit_zipf_mandelbrot
ax	Axes	None	Axes of matplotlib for the plot; if not given, a new figure is created
labels	dict[str, str]	None	Labels of the plot over the defaults: title, xlabel, ylabel, experimental and theoretical (the curves), fit (a format string with q and s)

The function returns the Axes with the plot; a num_words greater than the number of word types does not extend the curves beyond the data. A counter that is not a Counter of strings raises SourceTypeError, and an empty counter or a frequency that is not above zero SourceError. A num_words below one, a negative num_labels, with show_theory an exponent that is not a finite number above zero, labels that merge_labels refuses and an ax that is not a matplotlib Axes raise ParameterError. All of them are raised before a figure is created.

The Zipf-Mandelbrot fit is described in fit_zipf_mandelbrot.

Theoretical law

zipf_theory(size, num_ranks, alpha=1.5, ax=None, labels=None)

The theoretical curve alone, \(f(r) = size \cdot r^{-\alpha}\) for the ranks from 1 to num_ranks.

Parameter	Type	Default	Description
size	int / float	-	Frequency of the first rank
num_ranks	int	-	Number of ranks
alpha	float	1.5	Exponent α
ax	Axes	None	Axes of matplotlib for the plot; if not given, a new figure is created
labels	dict[str, str]	None	Labels over the defaults: theoretical (the curve)

A size or an alpha that is not a finite number above zero, a num_ranks that is not an integer of at least one, labels that merge_labels refuses and an ax that is not a matplotlib Axes raise ParameterError.