Language Models

Task definition

A statistical language model is a probability distribution over sequences of words. Given a set of sentences in Polish in a random order, each with original punctuation, the goal of the task is to create a language model for Polish. Participants are allowed to use any approach to creating the model.

Training data

As the training data set, we have prepared a corpus containing over 20 million sentences in Polish. The corpus consists of a single plain text. The order of the sentences is randomized and their sources will be revealed together with the test data. The punctuation and spelling has been left untouched from the original sources. 

DOWNLOAD: Task 3 - training data

UPDATE

We provide the segmented training corpus in the link below. All the tokens (words and punctuation marks) are separated with space.

DOWNLOAD: Task 3 - segmented training data

Data preprocessing should be performed similarly, as in Google One Billion Word Benchmark. Words are assumed to be the basic units of the language model. Punctuation marks are treated as words and they should be a part of language model. A vocabulary should be extracted from the training set. Every word and punctuation mark that occured 3 or more times in the training set should be included in the vocabulary. Additionally, the vocabulary should include sentence boundary markers (<s> - </s>) and the token <UNK> which is used to map Out Of Vocabulary (OOV) words. OOV words are tokens that are present in the training or test set, but not included in the vocabulary.

Test data

DOWNLOAD: Task 3 - test data

DOWNLOAD: Task 3 - segmented test data

Evaluation procedure

The participants are asked to evaluate perplexity of their model on the test set. The model with the lowest perplexity wins the competition. Detailed instructions about perplexity are provided below.

Perplexity is a measure commonly used to estimate the quality of a language model (see e.g.  Google One Billion Word Benchmark). Perplexity is the inverse probability of the test set, normalized by the number of words.

The language model should estimate the possibility of occurrence for every word in every sentence. The probability should be calculated for every word and the sentence ending in the test set.

Perplexity can be calculated using below equations:

$$\exp\left(\frac{\displaystyle\sum_{i=1}^{N} \log_e\left(\frac{1}{q(x_i)}\right)}{N}\right) \quad \text{or} \quad \sqrt[N]{\prod_{i=1}^{N} \frac{1}{q(x_i)}}$$

Where

• N - number of samples in the test set,
• X - discrete random variable with possible values {x1, …, xn},
• q(X) - probability function.

For instance, for the bigram language model perplexity equals:

$$\mathrm{PP}(W) = \sqrt[N]{\prod_{i=1}^{N} \frac{1}{P(w_i \mid w_{i-1})}}$$

Where

• N - number of words in the testset + number of sentence endings,
• W - the testset {w_1, w_2, …, w_N},
• P(w_i | w_{i-1}) is the conditional probability of word wi that occurs after word w (i-1).

The participants are asked to map unknown words (OOV words) to <UNK> and count them as a separate word in the vocabulary. They are also asked to count and provide the OOV (out-of-vocabulary) rate (the number of all words that occur in the test set but don’t occur in the vocabulary divided by all the words in the test set), to ensure that everyone is using the same vocabulary.

For instance, if we calculate the perplexity of the sentence: “I like bananas.” and the word “bananas” would not occur in the vocabulary and the trigam language model is used the perplexity (PPL) is computed as:

$$\mathrm{PPL} = \exp\left\{-\frac{1}{4}\left[\log P(\text{I} \mid \langle/\text{S}\rangle\,\langle\text{S}\rangle) + \log P(\text{like} \mid \langle\text{S}\rangle\,\text{I}) + \log P(\langle\text{UNK}\rangle \mid \text{I like}) + \log P(\langle/\text{S}\rangle \mid \text{like}\,\langle\text{UNK}\rangle)\right]\right\}$$

 and the out-of-vocabulary (OOV) rate is ¼.

References

Instructions provided by the authors of the 1-billion-word language modelling benchmark

Example of perplexity calculation