graviti
产品服务
解决方案
知识库
公开数据集
关于我们
Memotion Dataset 7k
2D Classification
许可协议: CC-BY-SA 4.0

Overview

The dataset is allowed to be used in any paper, only upon citation.

Bibtex:
@inproceedings{chhavi2020memotion,
title={{Task Report: Memotion Analysis 1.0 @SemEval 2020: The Visuo-Lingual Metaphor!}},
author="Sharma, Chhavi and
Paka, Scott, William and
Bhageria, Deepesh and
Das, Amitava and
Poria, Soujanya and
Chakraborty, Tanmoy and
Gamb{"a}ck, Bj{"o}rn",
booktitle = "Proceedings of the 14th International Workshop on Semantic Evaluation ({S}em{E}val-2020)",
year = {2020},
month = {Sep},
address = "Barcelona, Spain",
publisher = "Association for Computational Linguistics"
}

Introduction

This dataset is from the semeval challenge called "Memotion Analysis" in 2020. To participate click on the following link: https://competitions.codalab.org/competitions/20629

Abstract

Information on social media comprises of various modalities such as textual, visual and audio. NLP and Computer Vision communities often leverage only one prominent modality in isolation to study social media. However, the computational processing of Internet memes needs a hybrid approach. The growing ubiquity of Internet memes on social media platforms such as Facebook, Instagram, and Twitter further suggests that we can not ignore such multimodal content anymore. To the best of our knowledge, there is not much attention towards meme emotion analysis. The objective of this proposal is to bring the attention of the research community towards the automatic processing of Internet memes. The task Memotion analysis will release 8K annotated memes - with human-annotated tags namely sentiment, and type of humor that is, sarcastic, humorous, or offensive.

The Multimodal Social Media

In the last few years, the growing ubiquity of Internet memes on social media platforms such as Facebook, Instagram, and Twitter has become a topic of immense interest. Memes, one of the most typed English words (Sonnad, 2018) in recent times. Memes are often derived from our prior social and cultural experiences such as TV series or a popular cartoon character (think: One Does Not Simply - a now immensely popular meme taken from the movie Lord of the Rings). These digital constructs are so deeply ingrained in our Internet culture that to understand the opinion of a community, we need to understand the type of memes it shares. (Gal et al., 2016) aptly describes them as performative acts, which involve a conscious decision to either support or reject an ongoing social discourse. Online Hate - A brutal Job: The prevalence of hate speech in online social media is a nightmare and a great societal responsibility for many social media companies. However, the latest entrant Internet memes (Williams et al., 2016) has doubled the challenge. When malicious users upload something offensive to torment or disturb people, it traditionally has to be seen and flagged by at least one human, either an user or a paid worker. Even today, companies like Facebook and Twitter rely extensively on outside human contractors from start-ups like CrowdFlower, or companies in the Philippines. But with the growing volume of multimodal social media, it is becoming impossible to scale. The detection of offensive content on online social media is an ongoing struggle. OffenseEval (Zampieri et al., 2019) is a shared task which is being organized since the last two years at SemEval. But, detecting an offensive meme is more complex than detecting an offensive text – it involves visual cue and language understanding. This is one of the motivating aspects which encourages us to propose this task. Multimodal Social Media Analysis - The Necessity: Analogous to textual content on social media, memes also need to be analyzed and processed to extract the conveyed message. A few researchers have tried to automate the meme generation (Peirson et al., 2018; Oliveira et al., 2016) process, while a few others tried to extract its inherent sentiment (French, 2017) in the recent past. Nevertheless, a lot more needs to be done to distinguish their finer aspects such as type of humor or offense. We hope Memotion analysis - the task will bring research attention towards the topic and the forum will be the place to continue relevant discussions on the topic among researchers.

The Memotion Analysis Task

Task A- Sentiment Classification: Given an Internet meme, the first task is to classify it as a positive, negative or neutral meme.

Task B- Humor Classification: Given an Internet meme, the system has to identify the type of humor expressed. The categories are sarcastic, humorous, and offensive meme. If a meme does not fall under any of these categories, then it is marked as another meme. A meme can have more than one category.

Task C- Scales of Semantic Classes: The third task is to quantify the extent to which a particular effect is being expressed. Details of such quantifications are reported in Table 1. Appropriate annotated data will be provided.

Evaluation Criteria

For Task A: macro F1

For Task B and C: macro F1 for each of the subtasks, and then average.

Baseline Results

Task A: 0.2176489217
Task B: 0.5118483395
Task C: 0.2483801837

Labels submission format for different tasks

Task A: Negative and Very Negative => -1
Positive and Very Positive => 1
Neutral => 0

Task B: Not humorous => 0 and Humorous (funny, very funny, hilarious) => 1
Not Sarcastic => 0 and Sarcastic (general, twisted meaning, very twisted) => 1
Not offensive => 0 and Offensive (slight, very offensive, hateful offensive) => 1
Not Motivational => 0 and Motivational => 1

Task C:
Humour :
Not funny => 0
Funny => 1
Very funny => 2
Hilarious => 3

Sarcasm:
Not Sarcastic => 0
General => 1
Twisted Meaning => 2
Very Twisted => 3

Offense:
Not offensive => 0
Slight => 1
Very Offensive => 2
Hateful Offensive => 3

Motivation:
Not Motivational => 0
Motivational => 1

数据概要
数据格式
image,
数据量
6.998K
文件大小
86.94MB
发布方
William Scott
| 数据量 6.998K | 大小 86.94MB
Memotion Dataset 7k
2D Classification
许可协议: CC-BY-SA 4.0

Overview

The dataset is allowed to be used in any paper, only upon citation.

Bibtex:
@inproceedings{chhavi2020memotion,
title={{Task Report: Memotion Analysis 1.0 @SemEval 2020: The Visuo-Lingual Metaphor!}},
author="Sharma, Chhavi and
Paka, Scott, William and
Bhageria, Deepesh and
Das, Amitava and
Poria, Soujanya and
Chakraborty, Tanmoy and
Gamb{"a}ck, Bj{"o}rn",
booktitle = "Proceedings of the 14th International Workshop on Semantic Evaluation ({S}em{E}val-2020)",
year = {2020},
month = {Sep},
address = "Barcelona, Spain",
publisher = "Association for Computational Linguistics"
}

Introduction

This dataset is from the semeval challenge called "Memotion Analysis" in 2020. To participate click on the following link: https://competitions.codalab.org/competitions/20629

Abstract

Information on social media comprises of various modalities such as textual, visual and audio. NLP and Computer Vision communities often leverage only one prominent modality in isolation to study social media. However, the computational processing of Internet memes needs a hybrid approach. The growing ubiquity of Internet memes on social media platforms such as Facebook, Instagram, and Twitter further suggests that we can not ignore such multimodal content anymore. To the best of our knowledge, there is not much attention towards meme emotion analysis. The objective of this proposal is to bring the attention of the research community towards the automatic processing of Internet memes. The task Memotion analysis will release 8K annotated memes - with human-annotated tags namely sentiment, and type of humor that is, sarcastic, humorous, or offensive.

The Multimodal Social Media

In the last few years, the growing ubiquity of Internet memes on social media platforms such as Facebook, Instagram, and Twitter has become a topic of immense interest. Memes, one of the most typed English words (Sonnad, 2018) in recent times. Memes are often derived from our prior social and cultural experiences such as TV series or a popular cartoon character (think: One Does Not Simply - a now immensely popular meme taken from the movie Lord of the Rings). These digital constructs are so deeply ingrained in our Internet culture that to understand the opinion of a community, we need to understand the type of memes it shares. (Gal et al., 2016) aptly describes them as performative acts, which involve a conscious decision to either support or reject an ongoing social discourse. Online Hate - A brutal Job: The prevalence of hate speech in online social media is a nightmare and a great societal responsibility for many social media companies. However, the latest entrant Internet memes (Williams et al., 2016) has doubled the challenge. When malicious users upload something offensive to torment or disturb people, it traditionally has to be seen and flagged by at least one human, either an user or a paid worker. Even today, companies like Facebook and Twitter rely extensively on outside human contractors from start-ups like CrowdFlower, or companies in the Philippines. But with the growing volume of multimodal social media, it is becoming impossible to scale. The detection of offensive content on online social media is an ongoing struggle. OffenseEval (Zampieri et al., 2019) is a shared task which is being organized since the last two years at SemEval. But, detecting an offensive meme is more complex than detecting an offensive text – it involves visual cue and language understanding. This is one of the motivating aspects which encourages us to propose this task. Multimodal Social Media Analysis - The Necessity: Analogous to textual content on social media, memes also need to be analyzed and processed to extract the conveyed message. A few researchers have tried to automate the meme generation (Peirson et al., 2018; Oliveira et al., 2016) process, while a few others tried to extract its inherent sentiment (French, 2017) in the recent past. Nevertheless, a lot more needs to be done to distinguish their finer aspects such as type of humor or offense. We hope Memotion analysis - the task will bring research attention towards the topic and the forum will be the place to continue relevant discussions on the topic among researchers.

The Memotion Analysis Task

Task A- Sentiment Classification: Given an Internet meme, the first task is to classify it as a positive, negative or neutral meme.

Task B- Humor Classification: Given an Internet meme, the system has to identify the type of humor expressed. The categories are sarcastic, humorous, and offensive meme. If a meme does not fall under any of these categories, then it is marked as another meme. A meme can have more than one category.

Task C- Scales of Semantic Classes: The third task is to quantify the extent to which a particular effect is being expressed. Details of such quantifications are reported in Table 1. Appropriate annotated data will be provided.

Evaluation Criteria

For Task A: macro F1

For Task B and C: macro F1 for each of the subtasks, and then average.

Baseline Results

Task A: 0.2176489217
Task B: 0.5118483395
Task C: 0.2483801837

Labels submission format for different tasks

Task A: Negative and Very Negative => -1
Positive and Very Positive => 1
Neutral => 0

Task B: Not humorous => 0 and Humorous (funny, very funny, hilarious) => 1
Not Sarcastic => 0 and Sarcastic (general, twisted meaning, very twisted) => 1
Not offensive => 0 and Offensive (slight, very offensive, hateful offensive) => 1
Not Motivational => 0 and Motivational => 1

Task C:
Humour :
Not funny => 0
Funny => 1
Very funny => 2
Hilarious => 3

Sarcasm:
Not Sarcastic => 0
General => 1
Twisted Meaning => 2
Very Twisted => 3

Offense:
Not offensive => 0
Slight => 1
Very Offensive => 2
Hateful Offensive => 3

Motivation:
Not Motivational => 0
Motivational => 1

0
立即开始构建AI
graviti
wechat-QR
长按保存识别二维码,关注Graviti公众号

Copyright@Graviti
沪ICP备19019574号
沪公网安备 31011002004865号