PolEval
2018
PolEval 2026CurrentPolEval 2025PolEval 2024PolEval 2022/2023PolEval 2021PolEval 2020PolEval 2019PolEval 2018PolEval 2017
You are viewing the archive of PolEval 2018.Go to the current edition, PolEval 2026 →
PolEval 2018›Tasks›Task 2

Named Entity Recognition

Task definition

Named entities (NE) are phrases that contain the names of things, such as persons, organizations and locations. Named Entity Recognition (NER) task is to label sequences of words in a text with appropriate named entity markup. Named entity recognition is an important task of information extraction systems.

For example:
[PER Ban Ki-moon] przebywał z wizytą w [LOC Gdańsku].

This sentence contains two named entities: Ban Ki-moon is a person [PER], Gdańsk is a location [LOC].

Entity lemmatization is not a part of this task.

There has been a lot of work on named entity recognition, especially for English. The Message Understanding Conferences (MUC), CoNLL-2002 and CoNLL-2003 [1] have offered the opportunity to evaluate systems for English on the same data in a competition. They have also produced schemes for entity annotation. The PolEval Named Entity Recognition task aims to bridge this gap and provide a reference data set for the Polish language.

Annotating conventions of both training and testing data follow the National Corpus of Polish (NKJP). For a discussion of named entity annotation in NKJP, please refer to chapter 9. by Savary et al. [2]. Please note that NKJP supports nested entity annotation and multiple sub-categories (three sub-types of persons, five sub-types of places, as on figure 9.2 on p. 133 in http://nkjp.pl/settings/papers/NKJP_ksiazka.pdf).

Training data

As the training data set, we encourage to use the manually annotated 1-million word subcorpus of the National Corpus of Polish. It is available from: http://clip.ipipan.waw.pl/NationalCorpusOfPolish

DOWNLOAD: Task 2 - training data

Test data

Test data can be downloaded from the link below. Please make sure to follow the README file to learn about the desired data format as it is different than for the training set (i.e. not TEI). It is a lightweight format with offset information.

DOWNLOAD: Task 2 - test data

Evaluation procedure

We intend to evaluate systems according to two general approaches:

  • Exact match of entities: in order to count as a true positive, offsets of detected vs golden entities need to be the same,
  • Entity overlap: in order to count as a true positive, offsets of detected vs golden entities need to have a common range, such as a word or more words.

In both cases, in order to count as a true positive, detected category of an entity need to be the same as the golden one.

Final ranking of participating systems will be based on a combined measure of the exact match and entity overlap.

The metrics we are going to use for this task is micro F1.

Other remarks [updated 8.08]

  • It is possible and acceptable to use some form of transfer learning, including but not limited to word embeddings trained on different sources than the official training data set.
  • We also accept submissions trained on other named-entity data sources.  However, systems trained on such extra named entity resources will not be evaluated in the same competition as those trained on official data only.
  • We plan not to evaluate “deriv” annotations. This type of annotations was described in chapter 9. of the NKJP book, section “derivative words” (“wyrazy pochodne”) and develop into two subtypes (relAdj oraz persDeriv). While present in NKJP, we decided not to take them into account when evaluating submitted systems. Such markings will be ignored by the evaluation script.
  • We take into account fragmented entities as such are present in NKJP. However, our data format does not allow to distinguish one fragmented entity from multiple entities, provided they are of the same named entity category. Therefore, we only check categories of all annotated fragments. 

References

  1. Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. 
    http://www.aclweb.org/anthology/W03-0419.pdf
  2. Agata Savary, Marta Chojnacka-Kuraś, Anna Wesołek, Danuta Skowrońska, Paweł Śliwiński. Anotacja jednostek nazewniczych. In: Narodowy Korpus Języka Polskiego. Editors: Adam Przepiórkowski and Mirosław Bańko and Rafał L. Górski and Barbara Lewandowska-Tomaszczyk. Wydawnictwo Naukowe PWN, Warsaw. 2012. pp 129-165
    http://nkjp.pl/settings/papers/NKJP_ksiazka.pdf