Please enable JavaScript.

Coggle requires JavaScript to display documents.

Predicting Brexit: Classifying Agreement is Better than Sentiment and…

- - - - automatic task of assigning a polarity to a topic in context (Wiebe et al., 2005)
      - To polarity classification and sentiment analysis as the tasks for the extraction of emotive polarity or scores from text; to agreement/disagreement classification as the task of recognizing the opinion of a message towards others in a thread (Wang and Cardie, 2014)
  - - - We crawled the web for conversations about Brexit using handcrafted lists of keywords, hashtags and mentions and created daily data dumps
      - Collected from more than 4000 sources such as newspaper blogs, social network sites and other types of social media in 20 languages and from 14 countries of the European Union
      - Collected data was automatically processed with cross-language algorithms for extracting topics (Leave/Remain) and opinions towards them
    - - As the first step, the posts in the collected data are automatically assigned Leave or Remain topics.
      - The task is performed by means of simple hand-crafted rules that use keywords, hashtags and mentions to map the posts to classes.
      - If a post contains keywords, hashtags, or mentions for Leave and not for Remain, it is mapped to Leave; and if it contains keywords, hashtags or mentions for Remain and not for Leave, it is mapped to Remain.
      - Unclassified posts were not used for the prediction. The sets of keywords, hashtags and mentions used for each class were selected such that they yield balanced probabilities
    - - Just assignment of a topic to a post is not enough for the prediction of authors’ opinions.
      - Aggrement/Disagreement
        
        Automatic labeling of posts with agreement/disagreement makes use of language independent stylometric features
        
        e.g., character-based ratios of upper and lowercase letters, numbers, various punctuation marks and special characters; word-based ratios of URLs, Twitter mentions and hashtags, negative and positive emoticons
        
        Additionally, the model considers ratios of character and word ngrams (bigrams to tetragrams). All the features have their numerical values between 0 and 1.
        
        The model is trained and evaluated on the Italian CorEA corpus (Celli et al., 2014) using 66% of the data for training and 33% for evaluation.
        
        The corpus consists of about 2900 posts to online news articles that were manually annotated with respect to agreement, disagreement and neutrality/not applicability to the parent posts. The system was trained only on agreement and disagreement labels
        
        The inter-annotator agreement on two classes is k=0.85 and the manually annotated posts used for training and testing are approximately 2000.
        
        The task is cast as a regression with the well balanced bimodal distribution. The performance of the Support Vector Regressor (Shevade et al., 2000) on the CorEA test set has a Mean Absolute Error (MAE) of 0.32.
        
        Even though the model is trained and tested on Italian data, the features are language independent: semantics of features such as emoticons and punctuation are similar at least across European languages; thus, we believe that the model is applicable to other languages as well.
      - Sentiment Polarity
        
        Sentiment polarity prediction system is lexicon-based
        
        We used OpenNER polarity lexicon to label each post as either negative, positive, or neutral
        
        The posts classified as neutral were removed for the prediction
      - The authors’ opinions towards topics could be predicted either as a sentiment polarity expressed in a post, or as an agreement or disagreement with the topic expressed in a post
  - - - Sentiment polarity based
      - Agreement/disagreement classification based
        
        More accurate than the sentiment polarity based prediction. One reason can be that the agreement/disagreement based system considers significantly more posts ( 50K more) than the sentiment polarity system.
      - Baseline
        
        Volume of posts about Leave and Remain topics, obtained by
        topic labeling with the hand-crafted rules described
        
        Significantly offset and overestimates Leave (60.97%).
      - For the final prediction of each system we compute the percentage of posts that are positive towards one class and negative towards the other.