Lexical sophistication statistics¶
LexicalStats(source, words_extractor=None, freq_dict=None)
Lexical sophistication in the manner of TAALES (Kyle, Crossley and Berger, 2018) - how rare the words of a text are: statistics by a frequency dictionary and its unigram model, the shares of the frequency bands of a list of the most frequent lemmas and the lexical density. The statistics by the dictionary are computed when first read, so the bands and the lexical density need no dictionary data.
Raises:
SourceTypeError- a dictionary of another class than that of the library or, withoutfreq_dict_class, one withoutlookupandipmor none at all; ananalyzethat does not giveLexicalWordobjects;SourceError- a source without words.
In ruTS a source that is neither a string nor a Doc and an extractor of another type raise SourceTypeError too, and the metrics by the dictionary without the dictionary downloaded DatasetNotFoundError.
The words of a text are compared with the frequencies of the language (TAALES), not with each other as in the lexical diversity metrics. The data source is a text or a Doc object of the spaCy library.
The class extends the LexicalStats of the anyTS core with the Russian layer: the dictionary is the Lyashevskaya and Sharoff FreqDict (freq_dict_class), and the ranks of the lemmas and the size of the list (rank, top_size) come from the embedded list of the Sharoff lemmas (get_rank). The words of a string are extracted by the Russian WordsExtractor, the words of a Doc without an extractor are its tokens, and a hyphenated word that spaCy splits stays one word. Numbers (2020, 5.5, 3-й) are not words. The lemmas come from ruMorph: for a Doc with part-of-speech annotation by the part of speech of the token, for a string in the context of the sentence, for a Doc without annotation the same in the sentences whose words are those of the string, and for the other words and with a words extractor given by the first analysis. The keys of the words in the dictionary (keys, dictionary_keys) are the lemmas brought to its conventions by dictionary_lemma: должна - должен, денег - деньги, ночью - ночь. The print_stats table and the error messages are in Russian.
from ruts import LexicalStats
from ruts.datasets import FreqDict
FreqDict().download()
ls = LexicalStats("Кот сидел на окне и смотрел на птиц")
ls.print_stats()
Статистика | Значение
-------------------------------------------------------------------
Доля слов, найденных в частотном словаре | 1.00
Средняя частотность (ipm) | 8645.54
Средняя частотность знаменательных слов (ipm) | 324.18
Средняя логарифмическая частотность (lg ipm) | 3.07
Средняя логарифмическая частотность знаменательных слов | 2.32
Средний диапазон (R) | 99.75
Средняя дисперсия (D) | 95.12
Средний сюрпризал (бит) | 9.74
Униграммная перплексия | 856.21
Доля слов из топ-1000 | 0.75
Доля слов из топ-2000 | 0.88
Доля слов из топ-5000 | 1.00
Доля слов из топ-10000 | 1.00
Доля слов вне топ-10000 | 0.00
Лексическая плотность | 0.62
Parameters¶
| Parameter | Type | Default | Description |
|---|---|---|---|
source |
str / Doc |
- | Data source |
words_extractor |
WordsExtractor |
None |
Word extraction tool; used for a Doc too, on its text |
freq_dict |
FreqDict |
None |
Frequency dictionary; FreqDict() from the default directory by default |
Attributes¶
| Attribute | Type | Description |
|---|---|---|
freq_dict |
FrequencyDictionary |
Frequency dictionary |
words, lemmas |
tuple[str] |
Words and their lemmas; the bands are counted by the lemmas |
keys |
tuple[str] |
Keys of the words in the dictionary |
entries |
tuple[FrequencyEntry / None] |
Entries of the dictionary, None for a word out of it |
ranks |
tuple[int / None] |
Ranks of the lemmas, None out of the list |
n_words, n_content_words, n_found |
int |
Number of words, of content words and of words found in the dictionary |
In ruTS the dictionary is a FreqDict, and its entries are the named tuples Entry.
Metrics¶
Every metric is an attribute or a property of the object, and get_stats gives them all. A mean is over the words found in the dictionary, and without them it is nan.
| Metric | Measures | Range | Higher means |
|---|---|---|---|
coverage |
Share of the words found in the dictionary | 0-1 | More words known to the dictionary |
mean_ipm, mean_ipm_content |
Mean frequency per million words of the words found, of the content words found | Above 0 | More frequent words |
mean_log_ipm, mean_log_ipm_content |
Mean decimal logarithm of the frequency of the words found, of the content words found | Up to 6 | More frequent words |
mean_range |
Mean range of the words found | 0 and above | Words used in more parts of the corpus |
mean_dispersion |
Mean dispersion of the words found | That of the dictionary | By the measure of the dictionary |
surprisal |
Mean surprisal of the words by the unigram model of the dictionary, in bits | 0 and above | Rarer words |
perplexity |
2 to the power of the surprisal | 1 and above | Rarer words |
p_top1000, p_top2000, p_top5000, p_top10000 |
Share of the words with a lemma in the top 1000, 2000, 5000 and 10000 of the list | 0-1, nan for a band beyond top_size |
More frequent words |
p_beyond_top10000 |
Share of the words with a lemma beyond the top 10000 | 0-1, nan for a list shorter than 10000 |
Rarer words |
lexical_density |
Share of the content words | 0-1 | More content words |
The metrics rely on two resources; always read the means by the dictionary next to coverage.
| Resource | Metrics | Availability |
|---|---|---|
FreqDict - the Lyashevskaya and Sharoff dictionary, 52,138 lemmas with ipm, R, D |
coverage, mean_ipm*, mean_log_ipm*, mean_range, mean_dispersion, surprisal, perplexity |
downloaded once with FreqDict().download() |
| The embedded list of the 10,000 most frequent lemmas of the Leeds internet corpus (S. A. Sharoff, mirror hingston/russian, CC BY 2.5) | p_top1000, p_top2000, p_top5000, p_top10000, p_beyond_top10000, band_coverage |
always |
Dictionary frequency¶
The metrics are computed from the FreqDict entries for the keys of the words (keys, entries). The mean frequency in ipm is sensitive to function words (и - 35,000 ipm, в - 31,000), so for comparing texts the log frequency or the mean over the content words is more convenient. mean_ipm_content is the FREQ2 feature of the Solovyev, Ivanov and Solnyshkina formula (sis_grade_by_freq).
Rare words lower the coverage and the log frequency and raise the surprisal and the share beyond the top 10000:
from ruts import LexicalStats
ls = LexicalStats("Фелинолог пребывал на подоконнике")
ls.coverage, ls.mean_log_ipm, ls.surprisal, ls.p_beyond_top10000
(0.75, 2.3063585317426494, 14.515882690436708, 0.25)
Surprisal and perplexity¶
calc_surprisal(lemmas, freq_dict)
The mean surprisal of the words by the unigram model of a frequency dictionary, in bits; a word out of the dictionary gets its lowest frequency min_ipm. The perplexity is 2 to its power.
| Parameter | Type | Default | Description |
|---|---|---|---|
lemmas |
list[str] |
- | Keys of the words in the dictionary |
freq_dict |
FrequencyDictionary |
- | Frequency dictionary |
No words give nan; keys that are not a list of strings and a dictionary without ipm and min_ipm raise SourceTypeError, a lowest frequency that is not above zero ParameterError.
In ruTS the dictionary is a FreqDict with the lowest frequency of 0.4 ipm: a dictionary of another class raises SourceTypeError.
from ruts.datasets import FreqDict
from ruts.lexical_stats import calc_surprisal
calc_surprisal(["кот", "сидеть", "фелинолог"], FreqDict())
15.570731790883956
Frequency bands¶
The share of the words whose lemma has a rank (rank) within the top \(N\) of the list, for the bounds 1000, 2000, 5000 and 10000 of FREQUENCY_BANDS; the share beyond the top 10000 is the rest, the words out of the list included. A band beyond the size of the list (top_size) is nan, and so is the share beyond the top 10000 for a list shorter than 10000.
This builds a frequency band profile in the manner of the Lexical Frequency Profile.
get_rank(lemma)
The rank of a lemma by the embedded Sharoff list, None for a lemma outside the list; the lemma may be in any case, with ё or without.
| Parameter | Type | Default | Description |
|---|---|---|---|
lemma |
str |
- | Lemma |
load_top_lemmas()
The whole embedded list: the rank of every lemma, lowercased and with ё replaced by е, starting from 1; duplicates keep the smaller rank.
from ruts.lexical_stats import get_rank, load_top_lemmas
get_rank("и"), get_rank("кот"), get_rank("фелинолог")
len(load_top_lemmas())
(1, 2009, None)
9989
Lexical density¶
The share of the content words (nouns, adjectives, verbs, adverbs without pronominal and parenthetical words) among all the words; a content word is defined by is_content_word for a string and by CONTENT_UD_POS for an annotated Doc.
Methods¶
band_coverage(bands, unique) gives the share of the words with a lemma in the top N of the list for every bound N of bands (FREQUENCY_BANDS by default), or with unique=True the share of the distinct lemmas. Bounds that are not a list of integers or a bound out of 1 and top_size() raise ParameterError.
from ruts import LexicalStats
ls = LexicalStats("Кот сидел на окне и смотрел на птиц")
ls.band_coverage(unique=True)
ls.band_coverage(bands=(100, 500))
{1000: 0.7142857142857143, 2000: 0.8571428571428571, 5000: 1.0, 10000: 1.0}
{100: 0.375, 500: 0.75}
The other methods - get_stats, to_dataframe, print_stats and describe - are those of the statistics interface of the anyTS core; the lexical sophistication statistics have no scales, and describe gives None for the statistics of get_stats.
ls.get_stats()
{'coverage': 1.0,
'mean_ipm': 8645.5375,
'mean_ipm_content': 324.18,
'mean_log_ipm': 3.0674194359404705,
'mean_log_ipm_content': 2.316850597556863,
'mean_range': 99.75,
'mean_dispersion': 95.125,
'surprisal': 9.74182176626998,
'perplexity': 856.2105288389297,
'p_top1000': 0.75,
'p_top2000': 0.875,
'p_top5000': 1.0,
'p_top10000': 1.0,
'p_beyond_top10000': 0.0,
'lexical_density': 0.625}