Sentiment analysis
Introduction
Sentiment analysis is a vital research area, approached at different levels: phrase-level (either in the context of opinion targets/aspects or phrases defined as syntactic sub-trees), sentence-level (related to the task of tweet-level analysis).
The aim of this task is to promote research on this topic in the context of the Polish language, provide reference data sets to work and motivation for potentially new methods.
Task definition
Given a set of syntactic dependency trees, the goal of the task is to provide the correct sentiment for each sub-tree (phrase). Phrases correspond to sub-trees of dependency parse tree. The annotations assign sentiment values to whole phrases (and in some cases, sentences), regardless of their type.
Methods applied to this task often include deep learning. Typically, applications compute sentiment recursively, starting from leaves and smaller phrases, then expanding to larger phrases and taking into account sentiment values already computed for their nested sub-phrases. This could be equivalent to recursively folding the tree in a bottom-up fashion. Sentence-level sentiment is then the value of your predictive model after folding the whole sentence.
Datasets such as this one include Stanford Sentiment Treebank.
Training data
June 12: training data is published.
Data set is a selection of:
- sentiment-bearing sentences from the Skladnica treebank, dependency version.
- product reviews of two types (perfumes and clothing) with automatic dependency parse information.
sentiment-treebank.tar.gz - for each phrase: its sentiment, dependency labels, POS, tokens
Sentiment annotations for each token corresponds to the overall sentiment of the whole phrase under it and inclusive. Specifically:
- for every leaf token or word, its sentiment corresponds to this word or token’s sentiment
- for every non-leaf token or word (node that has non-empty set of children) sentiment field describes the sentiment of the whole phrase, formed by sub-tree starting at this token (that includes this token and all tokens below it)
Test data
Aug 1: the test data is published.
poleval_test.tar.gz - Task 2 test data set
Evaluation script
Please use the linked sent-eval.py script to evaluate the performance of your method against the provided test data. The script expects two files passed as arguments: one wirth your results, the other with gold (reference) annotation. It computes micro accuracy between your system and gold annotations. Micro accuracy is a global sum of correctly labelled sentiment scores, over cases belonging for each class. The scripts expects data formatted exactly the same as in training data.
We count sentiment scores for each leaf and each phrase. We assign the same weights (1) to all sentiment scores, regardless whether they represent sentiments of leafs, phrases or sentences. We do not distinguish between types of errors.