Visual Sentiment Prediction based on Automatic Discovery of Affective Regions

Abstract

Automatic assessment of sentiment from visual content has gained considerable attention with the increasing tendency of expressing opinions via images and videos online. This paper investigates the problem of visual sentiment analysis, which involves a high-level abstraction in the recognition process. While most of the current methods focus on improving holistic representations, we aim to utilize the local information, which is inspired by the observation that both the whole image and local regions convey significant sentiment information. We propose a framework to leverage affective regions, where we first use an off-the-shelf objectness tool to generate the candidates, and employ a candidate selection method to remove redundant and noisy proposals. Then a convolutional neural network (CNN) is connected with each candidate to compute the sentiment scores, and the affective regions are automatically discovered, taking the objectness score as well as the sentiment score into consideration. Finally, the CNN outputs from local regions are aggregated with the whole images to produce the final predictions. Our framework only requires image-level labels, thereby significantly reducing the annotation burden otherwise required for training. This is especially important for sentiment analysis as sentiment can be abstract, and labeling affective regions is too subjective and labor-consuming. Extensive experiments show that the proposed algorithm outperforms the state-of-the-art approaches on eight popular benchmark datasets.

Paper

Visual Sentiment Prediction based on Automatic Discovery of Affective Regions, Jufeng Yang, Dongyu She, Ming Sun, Ming-Ming Cheng, Paul L. Rosin and Liang Wang, TMM, 2018. [pdf] [project] [CN ver.]

If you find our work is helpful, please cite

@Article{yang2018visual,  
  author={Jufeng Yang and Dongyu She and Ming Sun and Ming-Ming Cheng and Paul L. Rosin and Liang Wang},  
  journal={IEEE Transactions on Multimedia}, 
  title={Visual sentiment prediction based on automatic discovery of affective regions},  
  year={2018},  
  volume={20},   
  number={9},    
  pages={2513-2525},
  doi={10.1109/TMM.2018.2803520}, 
}

Contact

Sherry6656 AT 163 DOT com  (Dongyu She)

Algorithm Pipeline

Given the input image, thousands of candidates along with the objectness scores are generated, and the candidate selection method is applied to remove the candidates which are overlapped and less important. The sentiment score of each proposal is roughly computed through CNN, which is then combined with the objectness score to discover affective regions. Finally, the sentiment label is predicted by fusing the local information with the holistic representation using several alternative operations.

Performance

The experiments show that the proposed algorithm outperforms the state-of-the-art approaches on eight popular benchmark datasets.

Visualization