Image emotion analysis based on semantic concepts

基于语义概念的图像情感分析

¹ School of Electronics and Information, Northwestern Polytechnical University, Xi'an 710072, China
² Content Production Center of Virtual Reality, Beijing 101318, China

Received: 20 September 2022

Abstract

With the increasing number of users express their emotions via images on social media, image emotion analysis attracts much attention of researchers. For the ambiguity and subjectivity of emotion, image emotion analysis is more challenging than other computer vision tasks. Previous methods merely learn a direct mapping between image feature and emotion. However, in emotion perception theory of psychology, it is demonstrated that human beings perceive emotion in a stepwise way. Therefore, we propose a novel image emotion analysis framework that makes use of emotional concepts as middle-level feature to bridge image and emotion. Firstly, the relationship between the concept and the emotion is organized in the form of knowledge graph. The relation between the image and the emotion in the semantic embedding space is explored where the knowledge is encoded into. On the other hand, a multi-level deep metric learning method to optimize the model from both label level and instance level is proposed. Extensive experimental results on two image emotion datasets, demonstrate that the present approach performs favorably against the state-of-the-art methods on both affective image retrieval and classification tasks.

摘要

随着越来越多的用户通过社交媒体表达自己的情感, 图像情感分析技术受到了研究人员的密切关注。但是由于情感的模糊性和主观性, 相比较于其他计算机视觉任务, 图像情感分析更具挑战性。该领域既有的工作仅研究了图像到情感之间的直接映射关系。然而, 心理学中有关情感感知的理论揭示了人们感知情感的过程是分步式的。因此, 提出了一种新的图像情感分析框架, 利用情感概念作为中级语义来辅助建立图像和情感之间的关系。将情感和概念的关系用知识图谱来描述并嵌入到语义空间中, 再将图像的视觉特征投影至该语义空间与情感进行对齐, 从而学习图像和情感之间的关系。另一方面, 提出了一种多层次深度度量学习方法, 从标记层面以及示例层面同时对模型进行优化。在2个情感图像数据集上进行实验, 结果表明提出的方法在情感图像检索以及分类任务上, 相对于现有方法表现良好。