CROWDSOURCING

Sensing and Learning Human Annotators Engaged in Narrative Sensemaking (N18-4)

Challenge:	a lack of gold datasets and knowledge about PAS analysis makes it difficult to create accurate PAS analyses.
Approach:	They construct a Japanese blog-QA dataset and a reading comprehension QA dataset using crowdsourcing.
Outcome:	The proposed method is most effective, pre-training model to acquire domain knowledge and fine-tuning model based on PAS-QA dataset.

Copied to clipboard

McKenna Tornblad, Luke Lapresi, Christopher Homan, Raymond Ptucha, Cecilia Ovesdotter Alm

CoRefi: A Crowd Sourcing Suite for Coreference Annotation (2020.emnlp-demos)

Challenge:	a substantial sector of the gig economy is the use of crowdworkers to annotate data for machine learning and analysis.
Approach:	They propose a narrative-sorting annotation task that sorts tweets chronologically by topic, emotional content, and length.
Outcome:	The proposed task enables readers to sort sequential, target-topical, and emotionally emotional tweets.

Copied to clipboard

Ari Bornstein, Arie Cattan, Ido Dagan

Information Extraction & Text Mining

Challenge:	Using a web-based coreference annotation suite, we demonstrate that non-expert annotators can be trained to perform and review coreference resolution tasks.
Approach:	They propose a web-based coreference annotation suite oriented for crowdsourcing that provides guided onboarding and a novel algorithm for a reviewing phase.
Outcome:	The proposed tool provides guided onboarding and a novel algorithm for a review phase.

Implicitly Abusive Comparisons – A New Dataset and Linguistic Analysis (2021.eacl-main)

Copied to clipboard

Michael Wiegand, Maja Geulig, Josef Ruppenhofer

Challenge:	Using crowdsourcing, we can detect implicitly abusive comparisons . Abusive language is defined as hurtful, derogatory or obscene utterances made by one person to another .
Approach:	They propose to use crowdsourcing to generate a dataset for detecting implicitly abusive comparisons . they also use a range of linguistic features to better understand abusive comparison mechanisms .
Outcome:	The proposed dataset includes measures to obtain representative and unbiased comparisons.

ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters (N19-1)

Copied to clipboard

Abdalghani Abujabal, Rishiraj Saha Roy, Mohamed Yahya, Gerhard Weikum

Question Answering

Challenge:	ComQA dataset captures question phenomena and the diverse ways in which they are formulated.
Approach:	They propose a large dataset of real user questions that captures question phenomena and the diverse ways in which they are formulated.
Outcome:	The proposed dataset can be a driver of future research on factoid question answering (QA).

Developing a Corpus of Indirect Speech Act Schemas (2020.lrec-1)

Copied to clipboard

Antonio Roque, Alexander Tsuetaki, Vasanth Sarathy, Matthias Scheutz

Quality Estimation for Partially Subjective Classification Tasks via Crowdsourcing (2020.lrec-1)

Challenge:	Indirect speech acts (ISAs) involve utterances whose literal meanings are not identical to their intended meanings.
Approach:	They propose a formal representation of ISA Schemas required for such testing, including a measure of the difficulty of a particular schema.
Outcome:	The proposed model minimizes the amount of expert authoring needed and maximizes realism.

Copied to clipboard

Yoshinao Sato, Kouki Miyazawa

Text Classification

Challenge:	a common approach to quality estimation is to ask multiple reviewers to evaluate the same artifacts.
Approach:	They propose a probabilistic model for subjective classification tasks that incorporates the qualities of artifacts as well as the abilities and biases of creators and reviewers as latent variables to be jointly inferred.
Outcome:	The proposed model estimates the quality of speech more effectively than a vote aggregation, measured by correlation with a fine-grained classification by experts.

Crowdsourcing in the Development of a Multilingual FrameNet: A Case Study of Korean FrameNet (2020.lrec-1)

Copied to clipboard

Younggyun Hahm, Youngbin Noh, Ji Yoon Han, Tae Hwan Oh, Hyonsu Choe, Hansaem Kim, Key-Sun Choi

Multilinguality

Challenge:	Using current methods, the construction of multilingual FrameNets is expensive and complex.
Approach:	They evaluated whether crowdsourcing approaches captured cross-cultural and cross-linguistic meanings . they found that crowd workers made intuitive choices comparable to trained FrameNet experts .
Outcome:	The results are now available in Korean FrameNet 1.1.

Towards a Reliable and Robust Methodology for Crowd-Based Subjective Quality Assessment of Query-Based Extractive Text Summarization (2020.lrec-1)

Copied to clipboard

Neslihan Iskender, Tim Polzehl, Sebastian Möller

SMILE: Single-turn to Multi-turn Inclusive Language Expansion via ChatGPT for Mental Health Support (2024.findings-emnlp)

Challenge:	a growing number of documents are needed for multi-document summarization.
Approach:	They propose crowdsourcing to evaluate intrinsic and extrinsic quality of extractive text summaries . they conduct intensive comparative crowdsourcing and laboratory experiments .
Outcome:	The proposed crowdsourcing task evaluates intrinsic and extrinsic quality of extractive text summaries.

Copied to clipboard

Huachuan Qiu, Hongliang He, Shuai Zhang, Anqi Li, Zhenzhong Lan

Dialogue Systems & Conversational Agents

Challenge:	Developing specialized dialogue systems for mental health support requires multi-turn conversation data . data privacy protection, time and cost involved in crowdsourcing are challenges . a new method for rewriting public single-turn dialogues into multi-turned ones is needed .
Approach:	They propose a single-turn to multi-turn inclusive language expansion technique that prompts ChatGPT to rewrite public single-turned dialogues into multi-turned ones.
Outcome:	The proposed method generates a large-scale, lifelike, and diverse dialogue dataset . it also develops SMILECHAT, a mental health chatbot .

Neural Text Simplification of Clinical Letters with a Domain Specific Phrase Table (P19-1)

Copied to clipboard

Matthew Shardlow, Raheel Nawaz

k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations (2022.acl-short)

Challenge:	Clinical letters are written by doctors and typically contain complex medical language that is beyond the scope of the lay reader.
Approach:	They propose to augment existing neural text simplification software with a phrase table that links medical terminology to simpler vocabulary by mining SNOMED-CT data.
Outcome:	The proposed system is easier to understand than the existing system and the phrase table without it.

Copied to clipboard

Ka Wong, Praveen Paritosh

Challenge:	aggregation has been a common strategy for dealing with unreliable data since the inception of crowdsourcing . however, many applications that rely on aggregate ratings only report the reliability of individual ratings, which is the incorrect unit of analysis.
Approach:	They propose to use k-rater reliability (kRR) to determine the correct data reliability for crowdsourced datasets.
Outcome:	The proposed method produces similar results on WordSim-353 datasets.

Breaking Down the Invisible Wall of Informal Fallacies in Online Discussions (2021.acl-long)

Copied to clipboard

Saumya Sahai, Oana Balalau, Roxana Horincar

Reasoning

Challenge:	a number of people engage in unsound argumentation techniques to prove a claim on online platforms . fallacies are weak arguments that seem convincing, but their evidence does not prove or disprove the conclusion .
Approach:	They propose to use user comments containing fallacy mentions as noisy labels to classify fallacies . they use the pragma-dialectical theory of argumentation to study the most common fallacias on Reddit .
Outcome:	The proposed dataset of fallacies on reddit shows that neural models perform better in conversational context.

Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design (2023.tacl-1)

Copied to clipboard

Valentina Pyatkin, Frances Yung, Merel C. J. Scholman, Reut Tsarfaty, Ido Dagan, Vera Demberg

Challenge:	Disagreement in natural language annotation has been studied from a perspective of biases introduced by the annotators and the annotation frameworks.
Approach:	They propose to analyze task design bias in crowdsourced annotations where lay annotators are used to elicit interpretations.
Outcome:	The proposed methods can push annotators towards certain relations and some discourse relation senses can be better elicited with one or the other approach.

Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation (N19-1)

Copied to clipboard

Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, Ido Dagan

Challenge:	Manual evaluation methods are perceived as insufficient due to the high cost of the Pyramid method and the required expertise.
Approach:	They propose a crowdsourced method that compares system summaries to references and uses crowdsourced scripts to analyze the results.
Outcome:	The proposed method shows higher correlation relative to the original Pyramid method.

A Multilingual Wikified Data Set of Educational Material (L18-1)

Copied to clipboard

Iris Hendrickx, Eirini Takoulidou, Thanasis Naskos, Katia Lida Kermanidis, Vilelmini Sosoni, Hugo de Vos, Maria Stasimioti, Menno van Zaanen, Panayota Georgakopoulou, Valia Kordoni, Maja Popovic, Markus Egg, Antal van den Bosch

Multilinguality

Challenge:	a crowdsourcing effort to annotate and link parallel texts has been unsuccessful . a data set of parallel texts in eleven languages is presented .
Approach:	They present a wikified data set of English sentences linked to Wikipedia pages . they use crowdsourcing to annotate the texts and perform crowdsourcing for complex annotations .
Outcome:	The proposed data set is valuable as it constitutes a rich resource . it includes annotated data of English sentences linked to translations in eleven languages .

Using Crowd Agreement for Wordnet Localization (L18-1)

Copied to clipboard

Amarsanaa Ganbold, Altangerel Chagnaa, Gábor Bella

Challenge:	Lexical-semantic resources like WordNet are a fundamental resource for many NLP and semantic applications.
Approach:	They propose a crowdsourcing workflow that consists of synset localization and validation . they use inter-rater agreement metrics to estimate the precision of the results .
Outcome:	The proposed method is cost-effective and provides a good trade-off between quality and speed of progress.

Building an English Vocabulary Knowledge Dataset of Japanese English-as-a-Second-Language Learners Using Crowdsourcing (L18-1)

Copied to clipboard

Yo Ehara

Linguistic Theories

Challenge:	a dataset for analyzing the English vocabulary of English-as-a-second language learners is available . a vocabulary size test was performed by 100 test takers hired via crowdsourcing .
Approach:	They propose a dataset for analyzing the English vocabulary of English-as-a-second language learners.
Outcome:	a dataset for analyzing the English vocabulary of English-as-a-second language learners is available online . the results show that the test is reliable and can be predicted with high accuracy .

A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition (2023.findings-acl)

Copied to clipboard

Limao Xiong, Jie Zhou, Qunxi Zhu, Xiao Wang, Yuanbin Wu, Qi Zhang, Tao Gui, Xuanjing Huang, Jin Ma, Ying Shan

Information Extraction & Text Mining

Challenge:	Existing models for named entity recognition (NER) are based on large-scale labeled datasets, which always obtain using crowdsourcing.
Approach:	They propose a CONfidence-based partial Label Learning method to integrate prior and posterior confidences for crowd-annotated named entity recognition models.
Outcome:	The proposed model improves on real-world and synthetic datasets compared with baselines.

Small but Mighty: New Benchmarks for Split and Rephrase (2020.emnlp-main)

Copied to clipboard

Li Zhang, Huaiyu Zhu, Siddhartha Brahma, Yunyao Li

Paraphrasing

Challenge:	Split and Rephrase is a text simplification task that requires a strong evaluation benchmark and metric . despite its relatively new nature, the benchmark dataset contains easily exploitable syntactic cues .
Approach:	They propose to use crowdsourced datasets to evaluate split and rephrase models . they find that the widely used benchmark dataset universally contains exploitable syntactic cues .
Outcome:	The proposed model performs better than the state-of-the-art model, the authors say . they show that the datasets contain significantly more diverse syntax .

ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness (2023.emnlp-main)

Copied to clipboard

Jan Cegin, Jakub Simko, Peter Brusilovsky

Language Models

Challenge:	generative large language models (LLMs) are replacing human workers for some tasks . crowdsourcing has several downsides: 1) the workforce is costly, 2) output quality is difficult to achieve, and 3) there are overheads related to the design and organization of the process.
Approach:	They investigate whether ChatGPT-created paraphrases are more diverse and robust . they use a crowdsourcing tool to collect training or validation examples .
Outcome:	The proposed models are more diverse and robust than the existing models.

Oddballs and Misfits: Detecting Implicit Abuse in Which Identity Groups are Depicted as Deviating from the Norm (2024.emnlp-main)

Copied to clipboard

Michael Wiegand, Josef Ruppenhofer

Ethical NLP

Challenge:	Abusive language is often defined as hurtful, derogatory or obscene utterances made by one person to another.
Approach:	They propose to use a dataset to detect abusive sentences in identity groups . they also report on classification experiments.
Outcome:	The proposed dataset includes 7 identity groups and includes classification experiments.

DialCrowd 2.0: A Quality-Focused Dialog System Crowdsourcing Toolkit (2022.lrec-1)

Copied to clipboard

Jessica Huynh, Ting-Rui Chiang, Jeffrey Bigham, Maxine Eskenazi

Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types (C18-1)

Challenge:	DialCrowd 2.0 helps requesters obtain higher quality data from human intelligence tasks.
Approach:	They propose to use DialCrowd 2.0 to help requesters obtain higher quality data . they aim to improve the way requesters present tasks and facilitate effective communication with workers.
Outcome:	The proposed toolkit enables requesters to obtain higher quality data by presenting tasks more clearly and facilitating effective communication with workers.

Copied to clipboard

Paul Felt, Eric Ringger, Jordan Boyd-Graber, Kevin Seppi

Challenge:	Annotated corpora are often assigned to internet workers whose judgments are reconciled by crowdsourcing models.
Approach:	They propose a framework for learning from rich prior knowledge to combine annotations with different structures.
Outcome:	The proposed model compares favorably with previous work and enables active sample selection to reduce annotation effort.

Combining Feature and Instance Attribution to Detect Artifacts (2022.findings-acl)

Copied to clipboard

Pouya Pezeshkpour, Sarthak Jain, Sameer Singh, Byron Wallace

ViLPAct: A Benchmark for Compositional Generalization on Multimodal Human Activities (2023.findings-eacl)

Challenge:	In this paper, we evaluate use of different attribution methods for aiding identification of training data artifacts.
Approach:	They propose hybrid methods that combine saliency maps and instance attribution methods to aid in identifying training data artifacts.
Outcome:	The proposed methods can be used to efficiently uncover artifacts in training data when a challenging validation set is available.

Copied to clipboard

Terry Yue Zhuo, Yaqing Liao, Yuecheng Lei, Lizhen Qu, Gerard de Melo, Xiaojun Chang, Yazhou Ren, Zenglin Xu

Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models (2024.naacl-long)

Challenge:	a vision-language benchmark for human activity planning is designed for humans . the task is easy for humans, but challenging for SOTA deep learning models .
Approach:	They propose a vision-language benchmark for human activity planning that extends Charades with intents and builds on a multi-choice question test set.
Outcome:	The proposed benchmark evaluates the ability of systems to anticipate and plan human actions in a multimodal visionlanguage setting.

Copied to clipboard

Jiashu Xu, Mingyu Ma, Fei Wang, Chaowei Xiao, Muhao Chen

Dialogue Scenario Collection of Persuasive Dialogue with Emotional Expressions via Crowdsourcing (L18-1)

Challenge:	et al., 2021) show that instruction models can be trained on crowdsourced datasets with task instructions to achieve superior performance.
Approach:	They examine security concerns of emergent instruction tuning paradigm that models are trained on crowdsourced datasets with task instructions to achieve superior performance.
Outcome:	The proposed model can achieve 90% success rate across four commonly used datasets.

Copied to clipboard

Koichiro Yoshino, Yoko Ishikawa, Masahiro Mizukami, Yu Suzuki, Sakriani Sakti, Satoshi Nakamura

WikiArt Emotions: An Annotated Dataset of Emotions Evoked by Art (L18-1)

Challenge:	Existing methods for data collection and annotation are costly and prevent launching new dialogue systems.
Approach:	They asked crowd workers to create persuasive dialogue systems using emotional expressions . they annotated emotional states and users' acceptance for system persuasion .
Outcome:	The proposed system has sufficient agreement even without training, the researchers found . the experiment showed that the collected data are comparable to real-world dialogue recording methods .

Copied to clipboard

Saif Mohammad, Svetlana Kiritchenko

PO-EMO: Conceptualization, Annotation, and Modeling of Aesthetic Emotions in German and English Poetry (2020.lrec-1)

Challenge:	a dataset of 4,000 pieces of art has annotations for emotions evoked in the observer . the dataset can help answer questions about what makes art evocative, how does art convey different emotions, what attributes of a painting make it well liked, and how much does the title impact the affectual response to art.
Approach:	They create a dataset of 4,000 western art pieces that has annotations for emotions . they use crowdsourcing to annotate the art for one or more of twenty emotion categories . fear, happiness, love, sadness were the dominant emotions that obtained consistent annotations .
Outcome:	The dataset shows that the most popular emotions are fear, happiness, love and sadness . the dataset can be used to develop systems that detect emotions evoked by art .

Copied to clipboard

Thomas Haider, Steffen Eger, Evgeny Kim, Roman Klinger, Winfried Menninghaus

Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference (P19-1)

Challenge:	a new study shows that literature enables engagement in a broader range of complex and subtle emotions.
Approach:	They propose to use multiple emotion labels to capture mixed emotions in poetry . they evaluate an annotation experiment with experts and crowdsourcing .
Outcome:	The proposed method shows that identifying aesthetic emotions is challenging in the German subset.

Copied to clipboard

Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, Iryna Gurevych

Textual Inference

Challenge:	Recent advances on abstractive summarization have led to fluent summaries, but factual errors in generated summary still severely limit their use in practice.
Approach:	They evaluate summaries produced by state-of-the-art models via crowdsourcing and show that factual errors occur frequently.
Outcome:	The proposed models can detect errors and reduce them by reranking alternative summaries.

Towards an Automatic Assessment of Crowdsourced Data for NLU (L18-1)

Copied to clipboard

Patricia Braunger, Wolfgang Maier, Jan Wessling, Maria Schmidt

Challenge:	Recent development of spoken dialog systems aims at allowing a natural input style.
Approach:	They investigate how crowdsourced data can be assessed with respect to its naturalness and usefulness by using a word based language model to identify valid data.
Outcome:	The proposed methods show that valid data can be identified with the help of a word based language model.

Controllable Dialogue Simulation with In-context Learning (2022.findings-emnlp)

Copied to clipboard

Zekun Li, Wenhu Chen, Shiyang Li, Hong Wang, Jing Qian, Xifeng Yan

Dialogue Systems & Conversational Agents

Challenge:	Existing methods to generate annotated dialogues require crowdsourcing, which is expensive and time-consuming.
Approach:	They propose a dialogue simulation method based on large language model in-context learning that generates new dialogues and annotations in a controllable way.
Outcome:	The proposed method can expand a small set of dialogue data with minimum or zero human involvement and parameter update.

Summary Grounded Conversation Generation (2021.findings-acl)

Copied to clipboard

Chulaka Gunasekara, Guy Feigenblat, Benjamin Sznajder, Sachindra Joshi, David Konopnicki

A Web-based System for Crowd-in-the-Loop Dependency Treebanking (L18-1)

Challenge:	Existing datasets for conversation summarization are small due to the lack of large-scale datasets.
Approach:	They propose three approaches to generate summary grounded conversations, and evaluate the generated conversations using automatic measures and human judgements.
Outcome:	The proposed models can generate entire conversations with only a summary of a conversation as the input.

Copied to clipboard

Stephen Tratz, Nhien Phan

Syntactic Text Processing

Challenge:	Existing treebanks are limited in size, genre, and topic coverage, making manual annotation time-consuming and expensive.
Approach:	They propose a web-based interactive tool for editing dependency trees that uses machine learning to accelerate annotation.
Outcome:	CROWDTREE is a web-based interactive tool for editing dependency trees . it can train a parsing model during the annotation process and can even be compatible with Mechanical Turk.

Effects of Naturalistic Variation in Goal-Oriented Dialog (2020.findings-emnlp)

Copied to clipboard

Jatin Ganhotra, Robert Moore, Sachindra Joshi, Kahini Wadhawan

A Computational Exploration of Exaggeration (D18-1)

Challenge:	Existing benchmarks for end-to-end neural dialog systems lack a key component: natural variation.
Approach:	They propose new and more effective testbeds by introducing naturalistic variation by the user.
Outcome:	The proposed testbeds incorporate natural variation by the user.

Copied to clipboard

Enrica Troiano, Carlo Strapparava, Gözde Özbal, Serra Sinem Tekiroğlu

Challenge:	a new computational approach to exaggeration detection is needed for non-literal phenomena . a corpus of overstatements (or hyperboles) is used to detect exaggrements .
Approach:	They propose a computational approach to detect exaggerated sentences using crowdsourcing data . they build a corpus containing overstatements and then evaluate models trained on HYPO .
Outcome:	The proposed approach can detect exaggerated sentences using a crowdsourced dataset.

ProxyQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models (2024.acl-long)

Copied to clipboard

Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, Linqi Song

Language Models

Challenge:	Existing evaluation methods for large language models are labor-intensive and lack efficiency.
Approach:	They propose a framework dedicated to assessing long-text generation that includes in-depth human-curated meta-questions spanning various domains . they use a set of proxy-quests with pre-annotated answers to assess the content's quality by incorporating the generated texts as contextual background.
Outcome:	The proposed framework assesses the quality of long-text content by matching it with references through human evaluation or automated metrics.

SLIDE - a Sentiment Lexicon of Common Idioms (L18-1)

Copied to clipboard

Charles Jochim, Francesca Bonin, Roy Bar-Haim, Noam Slonim

Barch: an English Dataset of Bar Chart Summaries (2022.lrec-1)

Challenge:	Compositional solutions for phrase sentiment are not able to handle idioms because their sentiment is not derived from the sentiment of the individual words.
Approach:	They propose a crowdsourcing approach for collecting sentiment annotations of idiomatic expressions using crowdsourcing.
Outcome:	The proposed approach is able to capture sentiment strength and ambiguity in idiomatic expressions using crowdsourcing.

Copied to clipboard

Iza Škrjanec, Muhammad Salman Edhi, Vera Demberg

Learning to Correct Noisy Labels for Fine-Grained Entity Typing via Co-Prediction Prompt Tuning (2023.findings-emnlp)

Challenge:	a new dataset of human-written summaries of bar charts is available in english . a chart summary is a textual description of a data point, which is often analytical .
Approach:	They propose a dataset of human-written summaries describing bar charts in english . a total of 47 charts are presented in the dataset, which includes 47 charts .
Outcome:	a new dataset of human-written summaries describing bar charts is presented in english . the dataset shows that human speakers often include such statements into chart summary .

Copied to clipboard

Minghao Tang, Yongquan He, Yongxiu Xu, Hongbo Xu, Wenyuan Zhang, Yang Lin

Language Models

Challenge:	Experimental results show that noise correction in fine-grained entity typing improves quality of training samples.
Approach:	They propose a method that leverages multiple prediction results to correct noisy labels . they integrate prediction results and utilize a differentiated margin to identify inaccurate labels a .
Outcome:	The proposed model improves quality of training samples annotated using distant supervision, ChatGPT, and crowdsourcing.

Construction of Large-scale English Verbal Multiword Expression Annotated Corpus (L18-1)

Copied to clipboard

Akihiko Kato, Hiroyuki Shindo, Yuji Matsumoto

Syntactic Text Processing

Challenge:	In this paper, we focus on verbal MWEs, whose accurate recognition is challenging because they could be discontinuous.
Approach:	They conduct large-scale annotations of VMWEs on the Wall Street Journal portion of Ontonotes . they first construct a VMwe dictionary based on the english-language Wiktionary .
Outcome:	The proposed resource annotates 7,833 VMWE instances belonging to various categories . the authors hope the results will help to develop models for MWE recognition and dependency parsing .

Crowdsourcing Learning as Domain Adaptation: A Case Study on Named Entity Recognition (2021.acl-long)

Copied to clipboard

Xin Zhang, Guangwei Xu, Yueheng Sun, Meishan Zhang, Pengjun Xie

Low-Resource NLP

Challenge:	Experimental results show that crowdsourced annotations are highly effective under supervised conditions.
Approach:	They propose an annotator-aware representation learning model that is inspired by domain adaptation methods which attempt to capture effective domain-alike features.
Outcome:	The proposed model is highly effective on a benchmark dataset and achieves state-of-the-art performance with only a very small scale of expert annotations.

Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements (2020.emnlp-main)

Copied to clipboard

Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, Zhiwei Guan

Speech Foundation Models and Crowdsourcing for Efficient, High-Quality Data Collection (2025.coling-main)

Challenge:	Existing tools for examining and fixing missing captions are lacking in mobile UIs.
Approach:	They propose a task for automatically generating language descriptions for UI elements from multimodal input including both the image and structural representations of user interfaces.
Outcome:	The proposed task can generate captions from image and structural representations of UI elements.

Copied to clipboard

Beomseok Lee, Marco Gaido, Ioan Calapodescu, Laurent Besacier, Matteo Negri

NAP2: A Benchmark for Naturalness and Privacy-Preserving Text Rewriting by Learning from Human (2025.findings-emnlp)

Challenge:	Existing methods for crowdsourcing data collection require a human workforce, which is hard to sustain.
Approach:	They propose to use Speech Foundation Models to automate validation processes . they find that SFMs can reduce reliance on human validation .
Outcome:	The proposed model reduces the reliance on human validation without degrading the quality of the final data.

Copied to clipboard

Shuo Huang, William Maclean, Xiaoxi Kang, Qiongkai Xu, Zhuang Li, Xingliang Yuan, Gholamreza Haffari, Lizhen Qu

Semantic Text Processing

Challenge:	a large number of large language models are being used to protect user privacy . sanitizing sensitive text using two common strategies is the answer .
Approach:	They propose sanitizing sensitive text using deleting expressions and abstracting them . they propose a tool for text rewriting that uses crowdsourcing and large language models .
Outcome:	The proposed approach protects privacy before sending sensitive data to large language models . it combines crowdsourcing and large language modeling to create a text rewrite tool .

Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text (2022.acl-long)

Copied to clipboard

Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, Yejin Choi

Semantic Text Processing

Challenge:	a recent study has reported that crowdsourcing cannot distinguish between machine-authored and human-authored text.
Approach:	They propose a framework called Scarecrow for scrutinizing machine text via crowd annotation . they use crowd annotation to identify redundancy, commonsense errors, and incoherence .
Outcome:	The proposed method quantifies gaps between human-authored and machine-generated text . it can detect redundancy, commonsense errors, and incoherence .

Beyond Generic Summarization: A Multi-faceted Hierarchical Summarization Corpus of Large Heterogeneous Data (L18-1)

Copied to clipboard

Christopher Tauchmann, Thomas Arnold, Andreas Hanselowski, Christian M. Meyer, Margot Mieskes

A Dataset for Tracking Entities in Open Domain Procedural Text (2020.emnlp-main)

Challenge:	Automated summarization has focused on ten to twenty documents, typically news articles, but could in theory analyze hundreds of documents from a wide range of sources and provide an overview to the interested reader.
Approach:	They propose a method for creating hierarchical summarization corpora from large, heterogeneous document collections by crowdsourcing relevant content and asking trained annotators to order the relevant information hierarchically.
Outcome:	The proposed method can be used to develop and evaluate hierarchical summarization systems.

Copied to clipboard

Niket Tandon, Keisuke Sakaguchi, Bhavana Dalvi, Dheeraj Rajagopal, Peter Clark, Michal Guerquin, Kyle Richardson, Eduard Hovy

Challenge:	Existing tasks require only a small set of attributes to track state changes in procedural text.
Approach:	They propose a task where given a procedural text as input, the task is to generate a set of state change tuples for each step.
Outcome:	The proposed task generates state change tuples from a set of pre-defined attributes for each step and predicts them from an open vocabulary.

Improving Machine Translation of Educational Content via Crowdsourcing (L18-1)

Copied to clipboard

Maximiliana Behnke, Antonio Valerio Miceli Barone, Rico Sennrich, Vilelmini Sosoni, Thanasis Naskos, Eirini Takoulidou, Maria Stasimioti, Menno van Zaanen, Sheila Castilho, Federico Gaspari, Panayota Georgakopoulou, Valia Kordoni, Markus Egg, Katia Lida Kermanidis

Machine Translation

Challenge:	Using crowdsourcing to train neural machine translation models is expensive and expensive . professional outsourcing of bilingual data is expensive if the translations are of a lower quality .
Approach:	They analyze the impact of crowdsourcing on the quality of in-domain training data . they use translations of MOOCs from English to eleven languages to fine-tune machine translation models .
Outcome:	The proposed method improves on general-domain training data and with pre-existing in-domain corpora.

A Systematic Approach to Derive a Refined Speech Corpus for Sinhala (2022.lrec-1)

Copied to clipboard

Disura Warusawithana, Nilmani Kulaweera, Lakshan Weerasinghe, Buddhika Karunarathne

DART: A Large Dataset of Dialectal Arabic Tweets (L18-1)

Challenge:	Despite being large and generic, some languages such as Sinhala are left to underutilize the technology due to the lack of adequate resources.
Approach:	They propose to derive a corpus from a publicly available corpus for Sinhala speech recognition using crowdsourcing and web scraping techniques.
Outcome:	The proposed corpus reduces the Word-Error-Rate by 15.9%.

Copied to clipboard

Israa Alsarsour, Esraa Mohamed, Reem Suwaileh, Tamer Elsayed

Challenge:	The Arabic language is the fifth most widely spoken language in the world; more than 380 million people speak and write in Arabic.
Approach:	They propose to build a large manually-annotated multi-dialect dataset of Arabic tweets that is publicly available.
Outcome:	The proposed dataset is well-balanced over five main Arabic dialects: Egyptian, Maghrebi, Levantine, Gulf, and Iraqi.

DoQA - Accessing Domain-Specific FAQs via Conversational QA (2020.acl-main)

Copied to clipboard

Jon Ander Campos, Arantxa Otegi, Aitor Soroa, Jan Deriu, Mark Cieliebak, Eneko Agirre

Quality Control for Crowdsourced Bilingual Dictionary in Low-Resource Languages (2022.lrec-1)

Challenge:	a dataset of 2,437 dialogues and 10,917 QA pairs is used to access domain-specific FAQ information.
Approach:	They present a dataset with 2,437 dialogues and 10,917 QA pairs for FAQs . they use the Wizard of Oz method with crowdsourcing to create dialogues using the original post and the original reply.
Outcome:	The proposed system can access domain-specific FAQ information without training data.

Copied to clipboard

Hiroki Chida, Yohei Murakami, Mondheera Pituxcoosuvarn

Low-Resource NLP

Challenge:	a recent study shows that crowdsourcing is becoming mainstream to create bilingual dictionaries . the number of people who can speak multiple low-resource languages is limited and the average ability of workers is low.
Approach:	They propose a method to aggregate the answers of evaluation tasks by majority voting . they use hyper questions to evaluate the reliability of workers and task-allocation method to select high-quality workers .
Outcome:	The proposed method improves quality of bilingual dictionaries by integrating answers by majority voting.

Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation (2024.acl-long)

Copied to clipboard

Jan Cegin, Branislav Pecher, Jakub Simko, Ivan Srba, Maria Bielikova, Peter Brusilovsky

LLMs to Replace Crowdsourcing For Parallel Data Creation? The Case of Text Detoxification (2024.findings-emnlp)

Challenge:	generative large language models (LLMs) have found their application in data augmentation tasks, where small numbers of text samples are paraphrased and then used to fine-tune downstream models.
Approach:	They propose to use taboo words, hints by previous outlier solutions, and chaining on previous outliest solutions to augment text datasets as part of instructions to LLMs augmenting text dataset.
Outcome:	The proposed methods increase diversity of generated texts, but performance is highest with hints.

Copied to clipboard

Daniil Moskovskiy, Sergey Pletenev, Alexander Panchenko

Konidioms Corpus: A Dataset of Idioms in Konkani Language (2024.lrec-main)

Challenge:	Manual annotation methods, such as crowdsourcing, are costly and require intricate task design skills.
Approach:	They propose to use open source LLMs to annotate parallel data for text detoxification . they generate a pseudo-parallel detoxification dataset using activation patching .
Outcome:	The proposed model performs comparable to the original dataset in automatic detoxification evaluation metrics and superior quality in manual evaluation and side-by-side comparisons.

Copied to clipboard

Naziya Mahamdul Shaikh, Jyoti D. Pawar, Mubarak Banu Sayed

Low-Resource NLP

Challenge:	Konkani is a low-resource language spoken by 2.5 million speakers . idiomatic sense processing is challenging due to the nature of idioms .
Approach:	They propose to use crowdsourced idiomatic sentence identification to build a corpus for idioms in the Konkani language.
Outcome:	The proposed corpus consists of 6520 sentences written in the Konkani language.

Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asia (2025.acl-long)

Copied to clipboard

Samuel Cahyawijaya, Holy Lovenia, Joel Ruben Antony Moniz, Tack Hwa Wong, Mohammad Rifqi Farhansyah, Thant Thiri Maung, Frederikus Hudi, David Anugraha, Muhammad Ravi Shulthan Habibi, Muhammad Reza Qorib, Amit Agarwal, Joseph Marvin Imperial, Hitesh Laxmichand Patel, Vicky Feliren, Bahrul Ilmi Nasution, Manuel Antonio Rufino, Genta Indra Winata, Rian Adam Rajagede, Carlos Rafael Catalan, Mohamed Fazli Mohamed Imam, Priyaranjan Pattnayak, Salsabila Zahirah Pranida, Kevin Pratama, Yeshil Bangera, Adisai Na-Thalang, Patricia Nicole Monderin, Yueqi Song, Christian Simon, Lynnette Hui Xian Ng, Richardy Lobo Sapan, Taki Hasan Rafi, Bin Wang, null Supryadi, Kanyakorn Veerakanjana, Piyalitt Ittichaiwong, Matthew Theodore Roque, Karissa Vincentio, Takdanai Kreangphet, Phakphum Artkaew, Kadek Hendrawan Palgunadi, Yanzhi Yu, Rochana Prih Hastuti, William Nixon, Mithil Bangera, Adrian Xuan Wei Lim, Aye Hninn Khine, Hanif Muhammad Zhafran, Teddy Ferdinan, Audra Aurora Izzani, Ayushman Singh, Evan Evan, Jauza Akbar Krito, Michael Anugraha, Fenal Ashokbhai Ilasariya, Haochen Li, John Amadeo Daniswara, Filbert Aurelian Tjiaranata, Eryawan Presma Yulianrifat, Can Udomcharoenchaikit, Fadil Risdian Ansori, Mahardika Krisna Ihsani, Giang Nguyen, Anab Maulana Barik, Dan John Velasco, Rifo Ahmad Genadi, Saptarshi Saha, Chengwei Wei, Isaiah Edri W. Flores, Kenneth Chen Ko Han, Anjela Gail D. Santos, Wan Shen Lim, Kaung Si Phyo, Tim Santos, Meisyarah Dwiastuti, Jiayun Luo, Jan Christian Blaise Cruz, Ming Shan Hee, Ikhlasul Akmal Hanif, M.Alif Al Hakim, Muhammad Rizky Sya’ban, Kun Kerdthaisong, Lester James Validad Miranda, Fajri Koto, Tirana Noor Fatyanosa, Alham Fikri Aji, Jostin Jerico Rosal, Jun Kevin, Robert Wijaya, Onno P. Kampman, Ruochen Zhang, Börje F. Karlsson, Peerat Limkonchotiwat

MentalRiskES: A New Corpus for Early Detection of Mental Disorders in Spanish (2024.lrec-main)

Challenge:	Southeast Asia is underrepresented in vision-language research . SEA-VL is an open-source initiative dedicated to developing culturally relevant datasets for SEA languages.
Approach:	They propose to use crowdsourced, automated image crawling and synthetic image generation to develop culturally relevant datasets for SEA languages.
Outcome:	The proposed datasets capture SEA cultural nuances and contexts better than existing datasets.

Copied to clipboard

Alba M. Mármol Romero, Adrián Moreno Muñoz, Flor Miriam Plaza-del-Arco, M. Dolores Molina González, María Teresa Martín Valdivia, L. Alfonso Ureña-López, Arturo Montejo Ráez

Text Classification

Challenge:	Existing studies on the prevalence of mental disorders on the Web are limited to the English language.
Approach:	They propose to use user messages posted on Telegram groups to annotate the corpus for natural language processing and to conduct experiments on text classification and regression.
Outcome:	The proposed corpus contains over 1,300 subjects with more than 45,000 messages posted in different public Telegram groups.

Euphemistic Abuse – A New Dataset and Classification Experiments for Implicitly Abusive Language (2023.emnlp-main)

Copied to clipboard

Michael Wiegand, Jana Kampfmeier, Elisabeth Eder, Josef Ruppenhofer

Text Classification

Challenge:	Currently, only explicit abuse can be reliably detected due to the increasing amount of abusive language on the Web.
Approach:	They propose a crowdsourced dataset that can detect euphemistic abuse by paraphrasing simple explicit utterances.
Outcome:	The proposed classifier augments training data with automatically-generated GPT-3 completions.

MAPLE: Multi-Aspect Panels of LLM Evaluators for Open-Ended Questions (2026.findings-acl)

Copied to clipboard

Michinori Jinji, Kyohei Atarashi, Koh Takeuchi, Hisashi Kashima

Challenge:	LLM-as-a-Judge uses LLMs to evaluate open-ended questions . however, the discrepancy between LLM generated evaluations and human evaluations remains a critical problem in this field .
Approach:	They propose a framework that orchestrates evaluations across multiple criteria using multiple LLMs.
Outcome:	The proposed framework achieves superior alignment with human evaluations compared to baselines.

UniPCM: Universal Pre-trained Conversation Model with Task-aware Automatic Prompt (2024.lrec-main)

Copied to clipboard

Yucheng Cai, Wentao Ma, Yuchuan Wu, Shuzheng Si, Yuan Shao, Zhijian Ou, Yongbin Li