SPECIFICITY

Simplifying Outcomes of Language Model Component Analyses with ELIA (2026.eacl-demo)

Challenge:	generating accurate hyper-detailed image descriptions is challenging for vision-language models trained on web-scraped image-text.
Approach:	They propose a data-centric framework for generating hyper-detailed image descriptions using web-scraped image-text.
Outcome:	The proposed framework improves on human evaluations on the data, even with only 9k samples.

Copied to clipboard

Aaron Louis Eidt, Nils Feldhus

Explainability & Interpretability in NLP

Challenge:	ELIA is an interactive web application that simplifies the outputs of various language model component analyses for a broader audience.
Approach:	They propose to use a vision-language model to automatically generate natural language explanations for the complex visualizations produced by these methods.
Outcome:	The proposed system integrates three key techniques and generates natural language explanations for complex visualizations.

Discussion Tracker: Supporting Teacher Learning about Students’ Collaborative Argumentation in High School Classrooms (2020.coling-demos)

Copied to clipboard

Luca Lugini, Christopher Olshefski, Ravneet Singh, Diane Litman, Amanda Godley

Challenge:	Discussion Tracker provides teachers with data about argument moves, specificity and collaboration .
Approach:	They have developed a classroom discussion analytics system that leverages natural language processing to classify argument moves, specificity and collaboration.
Outcome:	The proposed system performs with moderate to substantial agreement with humans in a classroom setting.

A Comparative Multidimensional Analysis of Empathetic Systems (2024.eacl-long)

Copied to clipboard

Andrew Lee, Jonathan Kummerfeld, Larry Ann, Rada Mihalcea

Answer-based Adversarial Training for Generating Clarification Questions (N19-1)

Challenge:	Empathetic dialogue systems have received significant attention, but no systematic review has verified these limitations.
Approach:	They analyze 21 empathetic dialogue systems using automated methods to examine their progress.
Outcome:	The results show that empathetic dialogue systems lack specificity, reflection levels, diversity . the results also offer guidance for developing future systems .

Copied to clipboard

Sudha Rao, Hal Daumé III

Robustness in NLP

Challenge:	a goal of natural language processing is to develop techniques that enable machines to process naturally occurring language.
Approach:	They propose a model where hypothetical answers are latent variables that can guide the model into generating more useful clarification questions.
Outcome:	The proposed model outperforms retrieval-based models and ablations that exclude utility model and adversarial training on two datasets.

Speak up, Fight Back! Detection of Social Media Disclosures of Sexual Harassment (N19-3)

Copied to clipboard

Arijit Ghosh Chowdhury, Ramit Sawhney, Puneet Mathur, Debanjan Mahata, Rajiv Ratn Shah

Text Classification

Challenge:	#MeToo movement provides platform to narrate personal experiences of sexual harassment.
Approach:	They propose a three-part ULMFiT architecture to tackle text subtleties in a classification task . they propose to annotate a manually annotated real-world dataset to test their approach .
Outcome:	The proposed model outperforms existing models that rely on handcrafted stylistic features and is more accurate than generic models.

Pre-Deployment Advertisement Ranking under Data Scarcity via Context-Aware Criteria Generation with VLMs (2026.acl-industry)

Copied to clipboard

Kyungho Kim, Yeonje Choi, Gyurim Hwang, Sejin Chung, Hongseok Lee, Myeong Ho Song, Yeongho Kim, Sunwoo Kim, Jongha Lee, Juyeon Kim, Kijung Shin

HydraSum: Disentangling Style Features in Text Summarization with Multi-Decoder Models (2022.emnlp-main)

Challenge:	Existing VLMs perform well on general multimodal tasks, but limited labeled data makes them difficult to apply to real-world business decisions.
Approach:	They propose a new task that aims to rank ads for a target brand prior to deployment . they propose 'brand-specific ad ranking' which uses brand-specific effectiveness .
Outcome:	The proposed task outperforms baselines on 10 brands on real-world advertising data.

Copied to clipboard

Tanya Goyal, Nazneen Rajani, Wenhao Liu, Wojciech Kryscinski

Summarization

Challenge:	Abstractive summarization systems implicitly encode “decisions” about summary properties, but these are not enforced.
Approach:	They propose a new summarization architecture that extends existing models to a mixture-of-experts version with multiple decoders.
Outcome:	The proposed architecture outperforms baseline models in obtaining stylistically-diverse summaries by sampling from individual decoders or their mixtures.

Improving Topic Quality by Promoting Named Entities in Topic Modeling (P18-2)

Copied to clipboard

Katsiaryna Krasnashchok, Salim Jouili

Information Extraction & Text Mining

Challenge:	Using named entities as domain-specific terms for news-centric content has not been studied extensively.
Approach:	They propose to use named entities as domain-specific terms for news-centric content . they propose a weighting model that incorporates more named entities in topic descriptors .
Outcome:	The proposed model improves the quality of news-centric topics by including more named entities in the topic descriptors.

DocFinQA: A Long-Context Financial Reasoning Dataset (2024.acl-short)

Copied to clipboard

Varshini Reddy, Rik Koncel-Kedziorski, Viet Lai, Michael Krumdick, Charles Lovering, Chris Tanner

Question Answering

Challenge:	Existing work on automating financial numerical reasoning focuses on unrealistically specific document snippets, failing to reflect the broader and more realistic scenarios faced by analysts.
Approach:	They propose a long-document financial QA task that augments 7,437 questions from existing FinQA dataset with full-document context, extending the average context length from under 700 words in FinQA to 123k words in DocFinQA.
Outcome:	The proposed task extends the average context length from under 700 words in FinQA to 123k words in DocFinQA.

Can Language Models Be Specific? How? (2023.findings-acl)

Copied to clipboard

Jie Huang, Kevin Chen-Chuan Chang, Jinjun Xiong, Wen-mei Hwu

Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers (2024.findings-acl)

Challenge:	Existing pre-trained language models have a preference for more specific answers . however, there may exist multiple answers for a query, while not all answers are equally specific.
Approach:	They propose to build a benchmark for specificity testing by forming masked token prediction tasks with prompts.
Outcome:	The proposed methods improve the specificity of pre-trained language models without additional training.

Copied to clipboard

Haowen Pan, Yixin Cao, Xiaozhi Wang, Xun Yang, Meng Wang

Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement (2026.acl-industry)

Challenge:	Existing methods to identify key neurons for interpretability of multi-modal large language models are unclear.
Approach:	They propose a method to identify key neurons for interpretability by multi-modal large language models.
Outcome:	The proposed method improves conventional works upon efficiency and applied range by removing needs of costly gradient computation.

Copied to clipboard

Wataru Hirota, Tomoki Taniguchi, Tomoko Ohkuma, Kosuke Takahashi, Takahiro Omi, Kosuke Arima, Takuto Asakura, Chung-Chi Chen, Tatsuya Ishigaki

Challenge:	Large language models (LLMs) make it easy to generate large numbers of product ideas.
Approach:	They propose to use a dataset of 3,000 individual scores across 300 patent-grounded product ideas to assess whether an automatic judge approximates an aggregate consensus.
Outcome:	The proposed model evaluators disagree on fine-grained ordinal scores, suggesting structured heterogeneity rather than random noise.

Answering Unanswered Questions through Semantic Reformulations in Spoken QA (2023.acl-industry)

Copied to clipboard

Pedro Faustini, Zhiyu Chen, Besnik Fetahu, Oleg Rokhlenko, Shervin Malmasi

Pruning as a Domain-specific LLM Extractor (2024.findings-naacl)

Challenge:	Question Answering (QA) is a longstanding NLP task, and voice assistants like Alexa have made Spoken QA ubiquitous.
Approach:	They propose a model that uses linguistically-grounded operations to rewrite questions to facilitate answering.
Outcome:	The proposed model improves answer rates on 1M unanswered questions from a leading voice assistant.

Copied to clipboard

Nan Zhang, Yanchi Liu, Xujiang Zhao, Wei Cheng, Runxue Bao, Rui Zhang, Prasenjit Mitra, Haifeng Chen

Challenge:	Large Language Models (LLMs) have exhibited remarkable proficiency across a wide array of NLP tasks.
Approach:	They propose a method for pruning large language models using general or task-specific weights to extract a compressed, task-agnostic LLM.
Outcome:	The proposed method extracts a compressed, domain-specific, and task- agnostic LLM by identifying LLM weights that are pivotal for general capabilities, like linguistic capability and multi-task solving, and domain- specific knowledge.

Event Semantic Classification in Context (2024.findings-eacl)

Copied to clipboard

Haoyu Wang, Hongming Zhang, Kaiqiang Song, Dong Yu, Dan Roth

Information Extraction & Text Mining

Challenge:	In this work, we focus on the semantic classification of events in context to help machines gain a deeper understanding of events.
Approach:	They propose to integrate event semantics into downstream tasks to help machines understand events better.
Outcome:	The proposed model improves the understanding of events in context.

Sentiment-Stance-Specificity (SSS) Dataset: Identifying Support-based Entailment among Opinions. (L18-1)

Copied to clipboard

Pavithra Rajendran, Danushka Bollegala, Simon Parsons

Learning to Control the Specificity in Neural Response Generation (P18-1)

Challenge:	Argument mining is a method for extracting argument components and structures from natural language texts.
Approach:	They propose to model arguments as a set of premises that either support each other or collectively support a conclusion.
Outcome:	The proposed rules give an overall accuracy of 0.83 for the three datasets.

Copied to clipboard

Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, Jun Xu, Xueqi Cheng

Text Generation

Challenge:	Existing generative conversational models tend to favor general and trivial responses which appear frequently.
Approach:	They propose a controlled response generation mechanism to handle different utterance-response relationships in terms of specificity.
Outcome:	The proposed model outperforms state-of-the-art models under automatic and human evaluations.

CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions (2025.findings-naacl)

Copied to clipboard

Mourad Heddaya, Kyle MacMillan, Hongyuan Mei, Chenhao Tan, Anup Malani

Summarization

Challenge:	CaseSumm is a dataset for long-context summarization in the legal domain . human groundtruth summaries are often not available for legal summarizing .
Approach:	They propose a dataset for long-context summarization that includes SCOTUS opinions and their official summaries.
Outcome:	The proposed dataset is the largest open legal case summarization dataset . it outperforms larger models on automatic metrics and human evaluation .

What Can We Learn from Noun Substitutions in Revision Histories? (2020.coling-main)

Copied to clipboard

Talita Anthonio, Michael Roth

Challenge:	Recent work shows that resulting improvements can be modelled computationally, assuming that each revision contributes to the improvement.
Approach:	They propose to model improvements in sentences using wikiHow revision histories by assuming that each revision contributes to the improvement.
Outcome:	The proposed model fails in cases where humans can resort to factual knowledge or intuitions about the required level of specificity.

ProMISe: A Proactive Multi-turn Dialogue Dataset for Information-seeking Intent Resolution (2024.findings-eacl)

Copied to clipboard

Yash Butala, Siddhant Garg, Pratyay Banerjee, Amita Misra

The Discussion Tracker Corpus of Collaborative Argumentation (2020.lrec-1)

Challenge:	Work done during internship at Amazon Alexa AI.
Approach:	They propose to use iterative suggested question-answering conversation to improve the trade-off between satisfaction of the user’s intent and keeping the information exchange natural.
Outcome:	The proposed proposed question-answering conversation improves the satisfaction of the user’s intent while keeping the information exchange natural and cognitive load of the interaction minimal on the users.

Copied to clipboard

Christopher Olshefski, Luca Lugini, Ravneet Singh, Diane Litman, Amanda Godley

Investigating Content Planning for Navigating Trade-offs in Knowledge-Grounded Dialogue (2024.eacl-long)

Challenge:	The Discussion Tracker corpus is an annotated dataset of transcripts of spoken, multi-party argumentation transcribed from 985 minutes of audio .
Approach:	They analyze 29 multi-party arguments transcribed from 985 minutes of audio . they provide descriptive statistics and code for predicting each dimension separately.
Outcome:	The Discussion Tracker corpus was collected in high school English classes and annotated for argument moves, specificity, specificities and collaboration dimensions.

Copied to clipboard

Kushal Chawla, Hannah Rashkin, Gaurav Singh Tomar, David Reitter

COMEM: In-Context Retrieval-Augmented Mass-Editing Memory in Large Language Models (2024.findings-naacl)

Challenge:	Knowledge-grounded dialogues require a balance between being specific to what the conversation partner has said and being attributable to an underlying source document.
Approach:	They propose a framework that allows to experiment with various plan variables supported by prior work . they show that metric-aware planning mechanisms are better at automatic evaluations but underperform in human judgment compared to metric agnostic mechanisms.
Outcome:	The proposed framework supports metric-agnostic and metric aware content planning, but it underperforms in human judgment.

Copied to clipboard

Shanbao Qiao, Xuebing Liu, Seung-Hoon Na

Leveraging Explicit Reasoning for Inference Integration in Commonsense-Augmented Dialogue Models (2025.coling-main)

Challenge:	Existing methods for "knowledge editing" in large language models are inadequate . authors propose a method that can be used to update outdated information or correct false information .
Approach:	They propose a unified knowledge editing method called in-COntext retrieval-augmented Mass-Editing Memory . it incorporates retrieval augmented IKE, a novel extension of IKE designed for massive editing tasks .
Outcome:	The proposed method outperforms existing methods on the zsRE and CounterFact datasets.

Copied to clipboard

Sarah E. Finch, Jinho D. Choi

Sparse Feature Coactivation Reveals Causal Semantic Modules in Large Language Models (2026.acl-long)

Challenge:	Existing approaches to commonsense-augmented dialogue rely on implicit reasoning to integrate commonsensense inferences during response generation.
Approach:	They propose to separate commonsense reasoning into explicit steps for generating, selecting, and integrating commonsensense into dialogue responses.
Outcome:	The proposed model infers commonsense knowledge from dialogue contexts to improve response quality and naturalness of dialogue interactions.

Copied to clipboard

Ruixuan Deng, Xiaoyang Hu, Miles Gilberti, Shane Storks, Aman Taxali, Mike Angstadt, Chandra Sripada, Joyce Chai

Chain-of-Specificity: Enhancing Task-Specific Constraint Adherence in Large Language Models (2025.coling-main)

Challenge:	Recent work has focused on layerwise interpretations, lacking fine-grained interpretation of specific features and their interaction.
Approach:	They identify semantically coherent, context-consistent network components in large language models . they use sparse autoencoders to coactivate sparsity features from a handful of prompts .
Outcome:	The proposed model can capture concepts and relations more comprehensively than individual features while maintaining specificity.

Copied to clipboard

Kaiwen Wei, Jiang Zhong, Hongzhi Zhang, Fuzheng Zhang, Di Zhang, Li Jin, Yue Yu, Jingyuan Zhang

What makes a good conversation? How controllable attributes affect human judgments (N19-1)

Challenge:	Existing approaches to enhancing large language models fail to emphasize specific constraints and unlock the underlying knowledge.
Approach:	They propose a method that emphasizes specific constraints and unlocks knowledge within LLMs by iteratively emphasising on specific constraints.
Outcome:	The proposed method outperforms existing methods in enhancing generated content, especially in terms of specificity.

Copied to clipboard

Abigail See, Stephen Roller, Douwe Kiela, Jason Weston

Mitigating Data Scarceness through Data Synthesis, Augmentation and Curriculum for Abstractive Summarization (2021.findings-emnlp)

Challenge:	Existing work on dialogue models for conversational quality is incompletely understanding the relationship between quality and individual attributes.
Approach:	They propose to use conditional training and weighted decoding to control four attributes for chit-chat dialogue: repetition, specificity, response-relatedness and question-asking.
Outcome:	The proposed methods improve human quality judgments by controlling combinations of these variables.

Copied to clipboard

Ahmed Magooda, Diane Litman

Text Generation

Challenge:	a new study explores data manipulation techniques for improving abstractive summarization models without the need for any additional data.
Approach:	They propose a method of data synthesis with paraphrasing, data augmentation with sample mixing and curriculum learning with new difficulty metrics based on specificity and abstractiveness.
Outcome:	The proposed techniques improve abstractive summarization models without additional data . the proposed techniques can be applied in isolation and when combined .

Deep Ordinal Regression for Pledge Specificity Prediction (D19-1)

Copied to clipboard

Shivashankar Subramanian, Trevor Cohn, Timothy Baldwin

Definition Modelling for Appropriate Specificity (2021.emnlp-main)

Challenge:	Currently, there are no publicly available annotated datasets of pledges . a novel approach to specificity prediction is needed to predict the specificity of pledged issues.
Approach:	They propose deep ordinal regression approaches for specificity prediction using supervised and semi-supervised settings.
Outcome:	The proposed methods demonstrate their utility over several baseline approaches.

Copied to clipboard

Han Huang, Tomoyuki Kajiwara, Yuki Arase

Challenge:	Existing definition generation techniques have faced various problems such as the out-of-vocabulary problem and over/under-specificity problems.
Approach:	They propose to leverage a pre-trained encoder-decoder model and introduce a re-ranking mechanism to model specificity in definitions.
Outcome:	The proposed method significantly outperforms the state-of-the-art method on standard evaluation datasets and shows that it addresses the over/under-specificity problems.

Personalizing Dialogue Agents: I have a dog, do you have pets too? (P18-1)

Copied to clipboard

Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, Jason Weston

Achieving Conversational Goals with Unsupervised Post-hoc Knowledge Injection (2022.acl-long)

Challenge:	chit-chat models lack specificity, do not display a consistent personality and are often not very captivating.
Approach:	They propose to train chit-chat models to condition on profile information and profile information about the interlocutors.
Outcome:	The proposed model can predict profile information about the interlocutors based on the data . the proposed model is able to generate meaningful responses in a chit-chat setting .

Copied to clipboard

Bodhisattwa Prasad Majumder, Harsh Jhamtani, Taylor Berg-Kirkpatrick, Julian McAuley

Assessing the Human Likeness of AI-Generated Counterspeech (2025.coling-main)

Challenge:	Existing neural dialog models lack specificity and informativeness due to limited knowledge available during training.
Approach:	They propose a method to extract relevant knowledge from external sources at decoding time and incorporate it into a dialog response.
Outcome:	The proposed method in goal-oriented and knowledge-grounded dialog settings shows that human annotators judge the outputs more engaging and informative compared to responses from prior dialog systems.

Copied to clipboard

Xiaoying Song, Sujana Mamidisetty, Eduardo Blanco, Lingzi Hong

Challenge:	Existing studies have focused on relevance, surface form, and other shallow linguistic characteristics.
Approach:	They propose to evaluate the human likeness of AI-generated counterspeech . they implement and evaluate several LLM-based generation strategies .
Outcome:	The proposed models show that human-written counterspeech can be distinguished by both simple classifiers and humans.

Steering Safely or Off a Cliff? Rethinking Specificity and Robustness in Inference-Time Interventions (2026.eacl-long)

Copied to clipboard

Navita Goyal, Hal Daumé III

Robustness in NLP

Challenge:	Existing studies have shown that model steering can preserve fluency and unrelated abilities, but it fails to preserve robustness specificity.
Approach:	They propose a framework that distinguishes three dimensions of specificity: general, control, and robustness.
Outcome:	The proposed framework distinguishes three dimensions of specificity: general (preserving fluency and unrelated abilities), control (preserving related control properties), and robustness (preserving control properties under distribution shifts).

Open-Set Living Need Prediction with Large Language Models (2025.findings-acl)

Copied to clipboard

Xiaochong Lan, Jie Feng, Yizhou Sun, Chen Gao, Jiahuan Lei, Xinleishi Xinleishi, Hengliang Luo, Yong Li

Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models (2022.coling-1)

Challenge:	Existing approaches to living need prediction treat it as a closed-set classification problem, severely limiting their ability to capture diversity and complexity of living needs.
Approach:	They propose a system leveraging large language models for unrestricted need prediction that leverages Maslow's hierarchy of needs to align predictions with human living needs.
Outcome:	The proposed system outperforms closed-set approaches on need-based life service recall by an average of 19.37% on real-world datasets.

Copied to clipboard

Ze-Feng Gao, Peiyu Liu, Wayne Xin Zhao, Zhong-Yi Lu, Ji-Rong Wen

A Multi-Persona Chatbot for Hotline Counselor Training (2020.findings-emnlp)

Challenge:	Recent results show that the mix-of-experts architecture is parameter inefficient . large-scale pre-trained language models can achieve excellent performance in many NLP tasks.
Approach:	They propose to build a parameter-efficient mix-of-experts architecture by sharing information across experts.
Outcome:	The proposed architecture increases model capacity without increasing computation costs.

Copied to clipboard

Orianna Demasi, Yu Li, Zhou Yu

Linguistically-Informed Specificity and Semantic Plausibility for Dialogue Generation (N19-1)

Challenge:	a chatbot cannot replace a counselor, but a simulation of intimate situations is needed to train counselors.
Approach:	They propose a counseling strategy annotation scheme and a multi-task framework that mimics prototype conversations to train counselors.
Outcome:	The proposed framework significantly increases response diversity and specificity, with limited impact to coherence.

Copied to clipboard

Wei-Jen Ko, Greg Durrett, Junyi Jessy Li

Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems (2020.coling-main)

Challenge:	Past work has focused on word frequency-based approaches to improving specificity, such as penalizing responses with only common words.
Approach:	They propose to rerank a sequence-to-sequence model to improve the informativeness, reasonableness, and grammatically of responses by using externally-trained classifiers targeting each of these factors.
Outcome:	The proposed model improves the informativeness, reasonableness, and grammatically of responses.

Copied to clipboard

Vitou Phy, Yang Zhao, Akiko Aizawa

Label Augmentation for Zero-Shot Hierarchical Text Classification (2024.acl-long)

Challenge:	Existing evaluation metrics are not designed to cope with this flexibility.
Approach:	They propose to group the qualities into three groups to obtain a single metric called USL-H.
Outcome:	The proposed metric achieves good correlations with human judgment and maintains its configurability towards different aspects and metrics.

Copied to clipboard

Lorenzo Paletto, Valerio Basile, Roberto Esposito

DistillMIKE: Editing Distillation of Massive In-Context Knowledge Editing in Large Language Models (2024.findings-acl)

Challenge:	Hierarchical Text Classification is a difficult problem due to the lack of labeled data and the cost of manually annotating data samples.
Approach:	They propose a method that uses a Large Language Model to augment the deepest layer of the labels hierarchy to enhance its specificity.
Outcome:	The proposed method achieves state-of-the-art on four public datasets and a strong correlation between the metric values and the classification performance.

Copied to clipboard

Shanbao Qiao, Xuebing Liu, Seung-Hoon Na

PersLLM: A Personified Training Approach for Large Language Models (2025.findings-emnlp)

Challenge:	In-context knowledge editing has shown respectable abilities on knowledge editing in terms of generalization and specificity.
Approach:	They propose a novel extension of in-context knowledge editing (IKE) that allows for massive edits to be injected into large language models.
Outcome:	The proposed method shows state-of-the-art perfomrances and comparable performance with MIKE.

Copied to clipboard

Zheni Zeng, Jiayi Chen, Huimin Chen, Yukun Yan, Yuxuan Chen, Zhenghao Liu, Zhiyuan Liu, Maosong Sun

Rationale-based Opinion Summarization (2024.naacl-long)

Challenge:	Large language models exhibit human-like intelligence, enabling them to simulate human behavior and support various applications that require both humanized communication and extensive knowledge reserves.
Approach:	They propose a framework for better data construction and model tuning to unlock the potential of LLM personification by using Chain-of-Thought prompting and anti-induction.
Outcome:	The proposed framework improves data construction and model tuning for insufficient data usage and rigid behavior patterns.

Copied to clipboard

Haoyuan Li, Snigdha Chaturvedi

Summarization

Challenge:	Existing methods to generate concise summaries of reviews are generic and lack supporting details.
Approach:	They propose a rationale-based opinion summarization paradigm that outputs representative opinions and corresponding rationales.
Outcome:	The proposed method is more useful than conventional summarizations.

UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations (2024.naacl-long)

Copied to clipboard

Wenting Zhao, Justin Chiu, Jena Hwang, Faeze Brahman, Jack Hessel, Sanjiban Choudhury, Yejin Choi, Xiang Li, Alane Suhr

SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluation (2025.emnlp-main)

Challenge:	Existing work evaluating commonsense reasoning focuses on making inferences about common, everyday situations.
Approach:	They propose to use an English language corpus to investigate commonsense reasoning . they characterize performance differences between human explainers and best-performing large language models .
Outcome:	The proposed method reduces the loss rate of human-written explanations on commonsense reasoning compared with the vanilla supervised fine-tuning approach .

Copied to clipboard

Xiaofu Chen, Israfel Salazar, Yova Kementchedjhieva

Emergence of Hierarchical Reference Systems in Multi-agent Communication (2022.coling-1)

Challenge:	N-gram-based evaluation metrics are unreliable due to low correlation to human judgments.
Approach:	They propose a metric that rewards correct details and penalizes incorrect ones.
Outcome:	The proposed metric matches the performance of open-source LLM-based metrics in correlation to human judgments while being far more efficient.

Copied to clipboard

Xenia Ohmer, Marko Duda, Elia Bruni

Challenge:	a hierarchical reference system allows the selection of the most appropriate level of specificity for a given context.
Approach:	They propose a hierarchical reference game to study the emergence of hierarchic reference systems in artificial agents.
Outcome:	The proposed game shows that agents can generalize to new concepts . the hierarchical reference game is based on a simplified world .

Automated Focused Feedback Generation for Scientific Writing Assistance (2024.findings-acl)

Copied to clipboard

Eric Chamoun, Michael Schlichtkrull, Andreas Vlachos

Challenge:	Recent work has focused on improving surface form and style rather than manuscript content.
Approach:	They propose to use a scientific writing focused feedback tool to generate specific, actionable and coherent comments which identify weaknesses in a paper and/or propose revisions to it.
Outcome:	The proposed tool outperforms existing approaches in specificity, reading comprehension and overall helpfulness of the generated reviews.

Out-of-Task Training for Dialog State Tracking Models (2020.coling-main)

Copied to clipboard

Michael Heck, Christian Geishauser, Hsien-chin Lin, Nurul Lubis, Marco Moresi, Carel van Niekerk, Milica Gasic

Modeling Persuasive Discourse to Adaptively Support Students’ Argumentative Writing (2022.acl-long)

Challenge:	Dialog state tracking (DST) suffers from data sparsity.
Approach:	They utilize non-dialog data from unrelated NLP tasks to train dialog state trackers . they propose to use dialog state tracking to summarise the conversation history .
Outcome:	The proposed method exploits non-dialog data from unrelated NLP tasks to train dialog state trackers.

Copied to clipboard

Thiemo Wambsganss, Christina Niklaus

Challenge:	Argumentation is an omnipresent rudiment of daily communication and thinking . humans struggle to develop argumentation skills due to a lack of individual and instant feedback in their learning process.
Approach:	They propose an argumentation annotation approach to model argumentative discourse in student-written business model pitches and embed it into an adaptive writing support system for students that provides individual argumentation feedback.
Outcome:	The proposed method annotates a corpus of 200 business model pitches in german and measures their self-efficacy and ease-of-use in a real-world writing exercise.

FeedEval: Pedagogically Aligned Evaluation of LLM-Generated Essay Feedback (2026.findings-acl)

Copied to clipboard

SeongYeub Chu, Jongwoo Kim, Mun Yong Yi

Challenge:	Recent research emphasizes the generation of high-quality feedback that provides justification and actionable guidance.
Approach:	They propose an LLM-based framework for evaluating LLM feedback along three dimensions: specificity, helpfulness, and validity.
Outcome:	The proposed framework evaluates LLM-generated feedback along three dimensions: specificity, helpfulness, and validity.

Language-specific Effects on Automatic Speech Recognition Errors for World Englishes (2022.coling-1)

Copied to clipboard

June Choe, Yiran Chen, May Pik Yu Chan, Aini Li, Xin Gao, Nicole Holliday

Fine-grained Classification of Circumstantial Meanings within the Prague Dependency Treebank Annotation Scheme (2024.lrec-main)

Challenge:	Existing systems are not able to meet the needs of speakers of different demographic groups.
Approach:	They propose to analyze the performance of Otter’s automatic captioning system on native and non-native English speakers of different language background through a linguistic analysis of segment-level errors.
Outcome:	The proposed system predicts certain errors from the phonological structure of a speaker’s native language.

Copied to clipboard

Marie Mikulova

Information Retrieval

Challenge:	a formally and semantically based fine-grained classification of circumstantial meanings is proposed for the Czech language . the methodology and principles used are language independent .
Approach:	They propose a formally and semantically based fine-grained classification of circumstantial meanings based on Prague Dependency Treebanks examples.
Outcome:	The proposed method is language independent and compares with English . it is carried out in the Czech language but not in any other annotation project .

CASSI: Contextual and Semantic Structure-based Interpolation Augmentation for Low-Resource NER (2023.findings-emnlp)

Copied to clipboard

Tanmay Surana, Thi-Nga Ho, Kyaw Tun, Eng Siong Chng

Named Entity Recognition

Challenge:	Existing methods for text augmentation suffer from annotation corruption for token-level tasks like NER.
Approach:	They propose a novel augmentation scheme that generates high-quality contextually diverse augmentations while avoiding annotation corruption.
Outcome:	The proposed scheme outperforms existing methods at multiple low resource levels, in multiple languages, and for noisy and clean text.

One Prompt To Rule Them All: LLMs for Opinion Summary Evaluation (2024.acl-long)

Copied to clipboard

Tejpalsingh Siledar, Swaroop Nath, Sankara Muddu, Rupasai Rangaraju, Swaprava Nath, Pushpak Bhattacharyya, Suman Banerjee, Amey Patil, Sudhanshu Singh, Muthusamy Chelliah, Nikesh Garera

Low-Resource NLP

Challenge:	Existing evaluation methods for opinion summarizations lack adequate opinion summary evaluation datasets.
Approach:	They propose a dataset that combines 7 dimensions crucial to opinion summaries . they propose OP-I-PROMPT, a dimension-independent prompt, and OP PROMPTS, .
Outcome:	The proposed model achieves a Spearman correlation of 0.70 with human judgments, surpassing prior methods.

ChainEdit: Propagating Ripple Effects in LLM Knowledge Editing through Logical Rule-Guided Chains (2025.acl-long)

Copied to clipboard

Zilu Dong, Xiangqing Shen, Zinong Yang, Rui Xia

Decomposing and Comparing Meaning Relations: Paraphrasing, Textual Entailment, Contradiction, and Specificity (2020.lrec-1)

Challenge:	Existing knowledge editing methods for large language models struggle to maintain logical consistency when propagating ripple effects to associated facts.
Approach:	They propose a framework that synergizes knowledge graph-derived logical rules with LLM logical reasoning capabilities to enable systematic chain updates.
Outcome:	The proposed framework improves logical generalization and specificity while maintaining reliability and specificness.

Copied to clipboard

Venelin Kovatchev, Darina Gold, M. Antonia Marti, Maria Salamo, Torsten Zesch

Syntactic Text Processing

Challenge:	SHARel is a new typology for decomposing and comparing multiple meaning relations . it consists of 26 linguistic and 8 reason-based categories and can be applied to all relations with a high inter-annotator agreement.
Approach:	They propose a new typology that consists of 26 linguistic and 8 reason-based categories and propose SHARel for decomposing and comparing multiple meaning relations.
Outcome:	The proposed method can be applied to all relations with high inter-annotator agreement.

Counterfactual Probing for the Influence of Affect and Specificity on Intergroup Bias (2023.findings-acl)

Copied to clipboard

Venkata Subrahmanyan Govindarajan, David Beaver, Kyle Mahowald, Junyi Jessy Li

CLAIR: Evaluating Image Captions with Large Language Models (2023.emnlp-main)

Challenge:	Existing work on bias in NLP only considers negative or pejorative language use.
Approach:	They propose a revised framing of bias in terms of intergroup social context and its effects on language output.
Outcome:	The proposed framework is based on a model of intergroup relationships in English language tweets.

Copied to clipboard

David Chan, Suzanne Petryk, Joseph Gonzalez, Trevor Darrell, John Canny

Model Editing at Scale leads to Gradual and Catastrophic Forgetting (2024.findings-acl)

Challenge:	Existing measures for image caption evaluation fail to capture dimensions of similarity . a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) demonstrates a stronger correlation with human judgments of caption quality compared to existing measures.
Approach:	They propose a method that leverages the zero-shot language modeling capabilities of large language models to evaluate captions.
Outcome:	The proposed method shows a stronger correlation with human judgments of caption quality compared to other measures.

Copied to clipboard

Akshat Gupta, Anurag Rao, Gopala Anumanchipalli

TamEdit: Trajectory-Aware Meta-Learning for Specificity-Preserving Continual Knowledge Editing (2026.acl-long)

Challenge:	Existing model editing methods are evaluated using metrics for reliability, specificity and generalization over one or few edits.
Approach:	They evaluate model editing methods for three crucial properties - editing proficiency, fact forgetting and downstream performance.
Outcome:	The proposed methods are based on two state-of-the-art models - ROME and MEMIT.

Copied to clipboard

Shiqiang Tian, Cheng Ding, Qin Chen, Jie Zhou, Liang He

Modelling Argumentation for an User Opinion Aggregation Tool (2024.lrec-main)

Challenge:	Existing methods for continual knowledge editing focus on single edits or preventing knowledge forgetting.
Approach:	They propose a meta-learning method that preserves specificity for continual knowledge editing by capturing relationships between different single edits within the trajectory.
Outcome:	Experiments show that TamEdit outperforms baselines in continual editing while preserving general capabilities.

Copied to clipboard

Pablo Weingart, Thiemo Wambsganss, Matthias Soellner

Challenge:	Existing methods for eliciting information from user opinion data are limited to high-level text and are prone to hallucination, degrading system performance or introduce biases.
Approach:	They propose an argumentation annotation scheme that models argumentative structure across user opinion domains.
Outcome:	The proposed model can predict arguments and contextual details from user opinions . the model can rank products based on user opinions and improve user experience .

Simulating Identity, Propagating Bias: Abstraction and Stereotypes in LLM-Generated Text (2025.findings-emnlp)

Copied to clipboard

Pia Sommerauer, Giulia Rambelli, Tommaso Caselli

Keys to Robust Edits: From Theoretical Insights to Practical Advances (2025.acl-long)

Challenge:	Persona-prompting is a growing strategy to personalize outputs, but its impact on how LLMs represent social groups remains underexplored.
Approach:	They investigate whether persona-prompting leads to different levels of linguistic abstraction . they compare 11 persona driven responses to those of a generic AI assistant .
Outcome:	The proposed method can be used to personalize outputs, but its impact on how LLMs represent social groups remains underexplored.

Copied to clipboard

Jianhao Yan, Futing Wang, Yun Luo, Yafu Li, Yue Zhang

Robustness in NLP

Challenge:	Existing methods for modifying parametric memory are prone to inaccuracies due to conflicting or outdated information.
Approach:	They propose a plug-and-play module that disentangles editing keys from native model representations and dynamically adjusts keys via contrastive learning to achieve robustness-specificity balance.
Outcome:	The proposed method improves over robustness tests by up to 66.4% while maintaining the success rate unaffected.

Qsnail: A Questionnaire Dataset for Sequential Question Generation (2024.lrec-main)

Copied to clipboard

Yan Lei, Liang Pang, Yuanzhuo Wang, Huawei Shen, Xueqi Cheng

CLAIMCHECK: How Grounded are LLM Critiques of Scientific Papers? (2025.findings-emnlp)

Challenge:	Questionnaires are a professional research methodology used for qualitative and quantitative analysis of human opinions, preferences, and behaviors.
Approach:	They propose a questionnaire-based dataset that consists of 13,168 human-written questionnaires.
Outcome:	The proposed dataset contains 13,168 human-written questionnaires gathered from online platforms.

Copied to clipboard

Jiefu Ou, William Gantt Walden, Kate Sanders, Zhengping Jiang, Kaiser Sun, Jeffrey Cheng, William Jurayj, Miriam Wanner, Shaobo Liang, Candice Morgan, Seunghoon Han, Weiqi Wang, Chandler May, Hannah Recknor, Daniel Khashabi, Benjamin Van Durme

The Contextual Variability of English Nouns: The Impact of Categorical Specificity beyond Conceptual Concreteness (2024.lrec-main)

Challenge:	CLAIMCHECK is an annotated dataset of NeurIPS 2023 and 2024 submissions and reviews from OpenReview.
Approach:	They annotate NeurIPS 2023 and 2024 submissions and reviews for weaknesses and dispute them for fine-grained labels of validity, objectivity, and type of the identified weaknesses.
Outcome:	The proposed dataset is richly annotated by ML experts for weaknesses statements in the reviews and the claims that they dispute, as well as fine-grained labels of validity, objectivity, and type of the identified weaknesses.

Copied to clipboard

Giulia Rambelli, Marianna Bolognesi

Information Extraction & Text Mining

Challenge:	Empirical studies on conceptual abstraction have examined differences in contextual distributions of abstract and concrete concept words.
Approach:	They propose to use a model to investigate the interplay between contextual variability and specificity of abstract and concrete concepts.
Outcome:	The proposed models show that more specific words have closer contexts than generic terms.

Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews (2025.emnlp-main)

Copied to clipboard

Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee, Juho Kim

Challenge:	Large Language Models (LLMs) can automatically draft reviews, but determining whether they are trustworthy requires systematic evaluation.
Approach:	They propose an automatic focus-level evaluation pipeline based on two sets of facets . authors evaluated LLM reviews at surface-level or content-level .
Outcome:	The proposed framework enables automatic evaluation of paper reviews based on two sets of facets . the framework compared open review paper reviews with human experts on validity, clarity, novelty .

LLMs as Lab Engineers: A Benchmark for Analytical Method Lifecycle Management (2026.findings-acl)

Copied to clipboard

Xiaoyi Chen, Mahsa Monshizadeh, Chaoqi Zhang, Jianjun Lang, Yang Wu, Genevieve Mortensen, Xiaozhong Liu, Haixu Tang

TabReX: Tabular Referenceless eXplainable Evaluation (2026.acl-long)

Challenge:	General-purpose commercial models outperform domain-specialized ones, while RAG and reasoning significantly improve performance.
Approach:	They propose a benchmark to evaluate LLMs' capabilities in analytical chemistry scenarios.
Outcome:	The proposed framework outperforms existing benchmarks focused on factual knowledge and provides practical guidance for analytical chemistry challenges.

Copied to clipboard

Tejas Anvekar, Junha Park, Aparna Garimella, Vivek Gupta