Papers with evaluation
Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translation (2025.naacl-srw)
Copied to clipboard
| Challenge: | Large language models excel in machine translation, but most studies focus on sentence-level translation. |
| Approach: | They propose to use LLMs as a judge paradigm to evaluate document-level translations by directly prompting them to translate entire documents in a single pass. |
| Outcome: | The proposed method improves translation quality even without document-level fine-tuning compared to translating sentences separately . |
WorkTeam: Constructing Workflows from Natural Language with Multi-Agents (2025.naacl-industry)
Copied to clipboard
| Challenge: | Existing workflow construction methods require specialized knowledge and task-switching skills. |
| Approach: | They propose a multi-agent workflow framework that incorporates a supervisor, orchestrator, and filler agent. |
| Outcome: | The proposed framework significantly increases the success rate of workflow construction . the proposed framework is based on a dataset of 3,695 real-world business samples . |
PolyNorm: Few-Shot LLM-Based Text Normalization for Text-to-Speech (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Text Normalization (TN) is a key preprocessing step in Text-to-Speech systems. |
| Approach: | They propose a prompt-based approach to TN using Large Language Models (LLMs) they propose scalable experimentation across languages to reduce the reliance on manual rules . |
| Outcome: | The proposed approach reduces the reliance on manual rules and enables broader linguistic applicability with minimal human intervention across eight languages. |
Robustness Gym: Unifying the NLP Evaluation Landscape (2021.naacl-demos)
Copied to clipboard
| Challenge: | Existing tools cater to specialized set of evaluations and provide no clear way to leverage or share findings from prior evaluations. |
| Approach: | They propose a toolkit that unifies 4 evaluation paradigms to provide a common platform for evaluation. |
| Outcome: | The proposed evaluation toolkit unifies 4 evaluation paradigms and is under active development. |
Reforging : A Method for Constructing a Linguistically Valid Japanese CCG Treebank (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing treebanks for Combinatory Categorial Grammar (CCG) are insufficient for linguistic validity of CCG . |
| Approach: | They propose to combine ABCTreebank and lightblue to generate a linguistically valid Japanese CCG treebank with detailed information by filtering lightblu's lexical items using ABCTtreebank. |
| Outcome: | The proposed method generates a linguistically valid Japanese CCG treebank with detailed information by combining the strengths of ABCTreebank and lightblue. |
SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects (2024.eacl-long)
Copied to clipboard
David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, En-Shiun Lee
| Challenge: | despite progress in building multilingual language models evaluation is limited to a few languages with available datasets . despite this, we create a large-scale open-sourced benchmark dataset for topic classification in 205 languages and dialects to address the lack of evaluation dataset for Natural Language Understanding (NLU). |
| Approach: | They create a large-scale open-sourced benchmark dataset for topic classification in 205 languages and dialects to address the lack of evaluation dataset for Natural Language Understanding (NLU). |
| Outcome: | The proposed dataset addresses the lack of evaluation dataset for Natural Language Understanding (NLU) for many languages, it is the first publicly available evaluation dataset. |
NeuroX Library for Neuron Analysis of Deep NLP Models (2023.acl-demo)
Copied to clipboard
| Challenge: | NeuroX is an open-source toolkit to conduct neuron analysis of natural language processing models. |
| Approach: | They propose a Python toolkit to conduct neuron analysis of natural language processing models. |
| Outcome: | a new open-source toolkit enables neuron analysis of natural language processing models . the framework provides a framework for data processing and evaluation, making it easier for researchers and practitioners to perform neuron analyses. |
LLM Evaluate: An Industry-Focused Evaluation Tool for Large Language Models (2025.coling-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive capability to solve a wide range of tasks in recent years. |
| Approach: | They propose to build an on-premise system for LLM evaluation to address the challenges in the evaluation of LLMs in real-world industrial settings. |
| Outcome: | The proposed evaluation system protects customer privacy and protects data integrity in real-world industrial environments. |
DVAGen: Dynamic Vocabulary Augmented Generation (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Existing dynamic vocabulary approaches struggle to generalize to novel or out-of-vocabulary words, limiting their flexibility in handling diverse token combinations. |
| Approach: | They propose an open-source framework for training, evaluation, and visualization of dynamic vocabulary-augmented language models. |
| Outcome: | The proposed framework validates the effectiveness of dynamic vocabulary-augmented language models on modern LLMs and shows support for batch inference significantly improving inference throughput. |
True Detective: A Deep Abductive Reasoning Benchmark Undoable for GPT-3 and Challenging for GPT-4 (2023.starsem-1)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated solid zero-shot reasoning capabilities, which is reflected in their performance on the current test tasks. |
| Approach: | They propose a benchmark consisting of 191 long-form mystery narratives constructed as detective puzzles. |
| Outcome: | The proposed benchmark outperforms random models on the current test tasks while state-of-the-art models only solve 38% of puzzles. |
KMatrix: A Flexible Heterogeneous Knowledge Enhancement Toolkit for Large Language Model (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Existing Knowledge-Enhanced Large Language Models (K-LLMs) toolkits focus on free-textual knowledge and lack robust datasets, models, and user-friendly experience. |
| Approach: | They propose a flexible heterogeneous knowledge enhancement toolkit to enhance Large Language Models (LLMs) using external knowledge. |
| Outcome: | KMatrix: a flexible heterogeneous knowledge enhancement toolkit for LLMs includes verbalizing-retrieval and parsing-query methods. |
ScaleBox: Enabling High-Fidelity and Scalable Code Verification for Large Language Models (2026.acl-demo)
Copied to clipboard
Jiasheng Zheng, Xin Zheng, Boxi Cao, Pengbo Wang, Zhengzhao Ma, Qiming Zhu, Jiazhen Jiang, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun
| Challenge: | Existing code sandboxes fail to provide accurate verification and efficiency under high-concurrency workloads. |
| Approach: | They propose a high-fidelity code verification system that provides sandbox feedback for RL training and evaluation. |
| Outcome: | The proposed system outperforms heuristic-matching baselines on LiveCodeBench and training stability on high-concurrency workloads. |
A Nontrivial Sentence Corpus for the Task of Sentence Readability Assessment in Portuguese (C18-1)
Copied to clipboard
| Challenge: | Effective textual communication depends on readers being proficient enough to comprehend texts . when meaning is not well conveyed, many losses and damages may occur . |
| Approach: | They propose automatic evaluation of sentence readability task in Portuguese to improve readability. |
| Outcome: | The proposed method correctly identifies the ranking of sentence pairs with an accuracy of 74.2%. |
DocEE-zh: A Fine-grained Benchmark for Chinese Document-level Event Extraction (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Chinese document-level event extraction is still largely unexplored. |
| Approach: | They propose a Chinese document-level event extraction dataset with over 36,000 events and 210,000 arguments. |
| Outcome: | The proposed dataset includes over 36,000 events and more than 210,000 arguments . it is an extension of the DocEE dataset, utilizing the same event schema and annotated by human experts. |
SPOR: A Comprehensive and Practical Evaluation Method for Compositional Generalization in Data-to-Text Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies on compositional generalization in data-to-text generation focus on one manifestation, Systematicity, Productivity, Order invariance, and Rule learnability. |
| Approach: | They propose a method for evaluation of compositional generalization in data-to-text generation that includes four aspects of manifestations and allows high-quality evaluation without additional manual annotations. |
| Outcome: | The proposed method is based on two datasets and evaluates existing language models including LLMs. |
Familarity: Better Evaluation of Zero-Shot Named Entity Recognition by Quantifying Label Shifts in Synthetic Training Data (2025.naacl-long)
Copied to clipboard
| Challenge: | Current research relies on large synthetic datasets to train zero-shot named entity recognition models. |
| Approach: | They propose a metric that captures the semantic similarity between entity types in training and evaluation to estimate label shift. |
| Outcome: | The proposed metric captures semantic similarity between entity types in training and evaluation, and their frequency in training data to provide an estimate of label shift. |
Long-term Control for Dialogue Generation: Methods and Evaluation (2022.naacl-main)
Copied to clipboard
| Challenge: | Current approaches for controlling dialogue response generation focus on high-level attributes like style, sentiment, or topic. |
| Approach: | They propose a method that allows for more fine-grained control of dialogue response generation . they propose utterances that encourage the generation of control words in the future . |
| Outcome: | The proposed method outperforms state-of-the-art constrained generation baselines on task-oriented dialogue datasets and shows that it is more fine-grained than previous methods. |
Thesis Proposal: Stability-Aware, Evidence-Grounded Knowledge Graph for Substance Use Disorders and Social Determinants of Health (2026.eacl-srw)
Copied to clipboard
| Challenge: | Existing methods for NER and RE annotation are costly and difficult to scale. |
| Approach: | They propose a semantic stability framework for constructing explainable KGs using NER and RE annotations. |
| Outcome: | The proposed framework supports multi-hop reasoning, triadic SUD–SDOH–SUD mediation patterns, and feedback loop analysis. |
Machine Translation Models are Zero-Shot Detectors of Translation Direction (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to detect the translation direction of parallel text are lacking in the machine translation community. |
| Approach: | They propose an unsupervised approach to detection of translation direction of parallel texts . they use a simple hypothesis that p(translation|original)>p(original|translation) they confirm the approach is effective for high-resource language pairs . |
| Outcome: | The proposed approach achieves document-level accuracies of 82–96% for NMT-produced translations and 60–81% for human translations, based on the model used. |
A Comparative Study on Schema-Guided Dialogue State Tracking (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent work proposes using natural language descriptions to define domain ontologies for dialog state tracking. |
| Approach: | They propose to use natural language descriptions to define domain ontologies instead of tag names for each intent or slot . they introduce a set of newly designed bench-marking descriptions and show model robustness . |
| Outcome: | The proposed model is robust on homogeneous and heterogeneously described descriptions in training and evaluation. |
Dialogue is the Plan: From Interface to Joint Action in Agentic AI (2026.acl-short)
Copied to clipboard
| Challenge: | Large Language Model agents' language use is often used as an interface for instructing and reporting results. |
| Approach: | They argue that large language models are often used as an interface for instructingactions and reporting results. |
| Outcome: | We show that large-scale language models can be used to plan and act, yet their language is often used as an interface for instructing and reporting results. |
StarFlow: Generating Structured Workflow Outputs From Sketch Images (2026.eacl-long)
Copied to clipboard
Patrice Bechard, Chao Wang, Amirhossein Abaskohi, Juan A. Rodriguez, Christopher Pal, David Vazquez, Spandana Gella, Sai Rajeswar, Perouz Taslakian
| Challenge: | Despite being widely used, building workflows can be complex, often requiring manual configuration through low-code platforms or visual programming tools. |
| Approach: | They propose a framework for generating structured workflow outputs from sketches using vision-language models to automate the process. |
| Outcome: | The proposed framework outperforms large vision-language models in the task of generating structured workflow outputs from sketches and diagrams. |
Unveiling the Achilles’ Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have highlighted various neural metrics that align well with human evaluations. |
| Approach: | They propose a black-box adversarial framework that generates strong disagreements between human and victim evaluators. |
| Outcome: | The proposed framework can significantly improve the performance of human and victim evaluators. |
IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding (2020.aacl-main)
Copied to clipboard
Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, Ayu Purwarianti
| Challenge: | Despite the availability of data on Indonesian, progress on this language is slow . available datasets are scattered, with a lack of documentation and minimal community engagement. |
| Approach: | They propose a resource for training, evaluation, and benchmarking on Indonesian natural language understanding tasks. |
| Outcome: | The proposed resource includes 12 tasks ranging from single sentence classification to pair-sentences sequence labeling with different levels of complexity. |
ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA Datasets with Large Language Models (2024.emnlp-industry)
Copied to clipboard
Ronak Pradeep, Daniel Lee, Ali Mousavi, Jeffrey Pound, Yisi Sang, Jimmy Lin, Ihab Ilyas, Saloni Potdar, Mostafa Arefiyan, Yunyao Li
| Challenge: | Knowledge Graphs (KGs) are a powerful tool for capturing structured representations of the world. |
| Approach: | They propose a scalable method for generating up-to-date and configurable conversational KGQA datasets that adheres to human interaction configurations and operates at a significantly larger scale. |
| Outcome: | Qualitative psychometric analyses show that ConvKGYarn produces high-quality data comparable to popular conversational KGQA datasets across various metrics. |
ENGinius: A Bilingual LLM Optimized for Plant Construction Engineering (2025.acl-industry)
Copied to clipboard
| Challenge: | Recent advances in large language models have drawn attention for their potential to automate and optimize processes across diverse sectors. |
| Approach: | They propose a specialized LLM for plant construction engineering that delivers optimized responses to plant engineers by leveraging enriched domain knowledge. |
| Outcome: | The proposed model delivers optimized responses to plant engineers by leveraging enriched domain knowledge. |
Multimodal and Multi-view Models for Emotion Recognition (P19-1)
Copied to clipboard
| Challenge: | combining lexical and acoustic information results in more robust and accurate models . combining both modalities may be a bottleneck in a deployment pipeline due to computational complexity or privacy constraints . |
| Approach: | They propose to combine acoustic and lexical information to provide a deployable acustic model . they use multimodal models and two attention mechanisms to assess the benefits of lexicals . |
| Outcome: | The proposed model outperforms the state-of-the-art on the USC-IEMOCAP dataset . it significantly surpasses models that have been exclusively trained with acoustic features . |
RAVEN: Query-Guided Representation Alignment for Question Answering over Audio, Video, Embedded Sensors, and Natural Language (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal question answering often requires identifying which video, audio, or sensor tokens are relevant to the question. off-camera speech, background noise, or motion outside the field of view often mislead fusion models that weight all streams equally. |
| Approach: | They propose a unified architecture for multimodal question answering that assigns scalar relevance scores to each token across modalities. |
| Outcome: | The proposed model outperforms state-of-the-art multimodal large language models on seven multi-modal QA benchmarks and egocentric and exocentric tasks. |
AutoUE: Automated Generation of 3D Games in Unreal Engine via Multi-Agent Systems (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) and generative models have motivated studies on automated game generation from natural language descriptions. |
| Approach: | They propose a novel multi-agent system, AutoUE, which coordinates multiple agents to end-to-end generate 3D games, covering model retrieval, scene generation, gameplay and interaction code synthesis, and automated game testing for evaluation. |
| Outcome: | The proposed system covers model retrieval, scene generation, gameplay and interaction code synthesis, and automated game testing for evaluation. |
Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning (2023.acl-short)
Copied to clipboard
| Challenge: | Recent studies on instruction tuning (IT) have achieved great performance with zero-shot generalizability to unseen tasks. |
| Approach: | They analyze how models utilize instructions during IT by comparing model training with altered vs. original instructions. |
| Outcome: | The proposed model outperforms naive models in low resource setting. |
Skill Induction and Planning with Latent Language (2022.acl-long)
Copied to clipboard
| Challenge: | a framework for learning hierarchical policies from demonstrations is lacking . we use sparse annotations to guide the discovery of reusable skills . |
| Approach: | They propose a framework for learning hierarchical policies from demonstrations using sparse annotations. |
| Outcome: | The proposed model outperforms other models with access to ground-truth plans in the ALFRED simulation environment. |
Semantically Driven Sentence Fusion: Modeling and Evaluation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Sentence fusion is the task of joining related sentences into coherent text. |
| Approach: | They propose a method where ground-truth solutions are automatically expanded into multiple references via curated equivalence classes of connective phrases. |
| Outcome: | The proposed approach improves on state-of-the-art models by expanding ground-truth solutions into multiple references. |
NUTMEG: Separating Signal From Noise in Annotator Disagreement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work suggests that annotators may have genuine disagreements, but few models separate signal from noise in annotator disagreement. |
| Approach: | They propose a Bayesian model that removes noisy annotations from training data while preserving systematic disagreements. |
| Outcome: | The proposed model outperforms models trained on NUTMEG-aggregated data. |
Multi-Defendant Legal Judgment Prediction via Hierarchical Reasoning (2023.findings-emnlp)
Copied to clipboard
Yougang Lyu, Jitai Hao, Zihan Wang, Kai Zhao, Shen Gao, Pengjie Ren, Zhumin Chen, Fang Wang, Zhaochun Ren
| Challenge: | Existing methods for predicting judgment results for multiple defendants are ineffective. |
| Approach: | They propose a method to predict the judgment results for each defendant in multi-defendant cases . they formalize the multi-diffendant judgment process as hierarchical reasoning chains . |
| Outcome: | The proposed method can predict the judgment results for multiple defendants in multi-defendant cases. |
Hard to Be Heard: Phoneme-Level ASR Analysis of Phonologically Complex, Low-Resource Endangered Languages (2026.findings-acl)
Copied to clipboard
| Challenge: | a phoneme-level analysis of automatic speech recognition (ASR) is performed on two low-resource, typologically complex East Caucasian languages. |
| Approach: | They propose a phoneme-level analysis of automatic speech recognition for two East Caucasian languages, Archi and Rutul. |
| Outcome: | The proposed model improves on existing models and improves in low-resource settings. |
Bahasa Harmony: A Comprehensive Dataset for Bahasa Text-to-Speech Synthesis with Discrete Codec Modeling of EnGen-TTS. (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text-to-speech (TTS) systems often fail to address the needs of Bahasa, resulting in limited adaptability, linguistic richness, or efficiency. |
| Approach: | They propose a Bahasa text-to-speech dataset and a novel TTS model, EnGen-TTS, which enhance the quality and versatility of synthetic speech in the Bahasan language. |
| Outcome: | The proposed model outperforms existing models even without fine-tuning and achieves a mean opinion score of 4.45 0.13. |
Question Modifiers in Visual Question Answering (2022.lrec-1)
Copied to clipboard
| Challenge: | Visual Question Answering (VQA) is a multi-disciplinary task that requires integration of several key disciplines. |
| Approach: | They develop a model that adds modifiers to questions based on object properties and spatial relationships using Amazon Mechanical Turk data. |
| Outcome: | The proposed model can improve when questions are modified to include more details. |
You should evaluate your language model on marginal likelihood over tokenisations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Neural language models typically tokenise input text into sub-word units to achieve an open vocabulary. |
| Approach: | They propose that language models should be evaluated on their marginal likelihood over tokenisations instead. |
| Outcome: | The proposed approach is unsatisfactory and may bottleneck model out-of-domain performance. |
Asking Clarification Questions in Knowledge-Based Question Answering (D19-1)
Copied to clipboard
| Challenge: | Existing clarification datasets with limited annotated examples do not address ambiguous phenomena. |
| Approach: | They propose a dataset that allows users to ask clarification questions using open-domain examples. |
| Outcome: | The proposed model achieves better performance than strong baselines and provides new challenges. |
Probing into the Root: A Dataset for Reason Extraction of Structural Events from Financial Documents (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for event reason extraction are far from resolving this problem. |
| Approach: | They propose a task to extract causal explanations from document-level texts . they use a dataset FinReason for evaluation to provide Reasons annotation for financial events . |
| Outcome: | The proposed task performs better than existing methods on a dataset of 8,794 documents, 12,861 financial events and 11,006 reason spans. |
Gradient-Based Language Model Red Teaming (2024.eacl-long)
Copied to clipboard
| Challenge: | generative language models generate unsafe responses by producing adversarial prompts . red teaming is labor-intensive and difficult to scale when done by humans. |
| Approach: | They propose a red teaming method that generates diverse prompts that are likely to cause an LM to generate unsafe responses. |
| Outcome: | The proposed method is more effective at finding prompts that trigger an LM to generate unsafe responses than a strong reinforcement learning-based red teaming approach. |
Parsivar: A Language Processing Toolkit for Persian (L18-1)
Copied to clipboard
| Challenge: | a preprocessing step is required to convert text into a standard format for NLP tasks. |
| Approach: | They propose a Persian preprocessing toolkit that performs various kinds of activities . they use a plagiarism detection system to exploit the proposed toolkit . |
| Outcome: | The proposed tool outperforms available Persian preprocessing tools by about 8 percent in terms of F1 . the proposed toolkit performs normalization, space correction, tokenization, stemming, parts of speech tagging and shallow parsing tasks. |
A Critical Evaluation of Evaluations for Long-form Question Answering (2023.acl-long)
Copied to clipboard
| Challenge: | Long-form question answering (LFQA) is an emerging research area within QA . however, its flexibility poses enormous challenges for evaluation . |
| Approach: | They conduct the first targeted study of the evaluation of long-form answers, covering both human and automatic evaluation practices. |
| Outcome: | The proposed evaluations cover human and automatic evaluations. |
KeyGames: A Game Theoretic Approach to Automatic Keyphrase Extraction (2020.coling-main)
Copied to clipboard
| Challenge: | Automated keyphrase extraction (AKE) is a task of identifying important words and phrases that best describe a given text document. |
| Approach: | They introduce two advancements in the automatic keyphrase extraction space - KeyGames and pke+. |
| Outcome: | The proposed framework outperforms state-of-the-art models while generalizing better on documents with different domains and length. |
Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Processing (2022.emnlp-main)
Copied to clipboard
Abbas Ghaddar, Yimeng Wu, Sunyam Bagga, Ahmad Rashid, Khalil Bibi, Mehdi Rezagholizadeh, Chao Xing, Yasheng Wang, Xinyu Duan, Zhefeng Wang, Baoxing Huai, Xin Jiang, Qun Liu, Phillippe Langlais
| Challenge: | Existing pre-trained language models are not well-explored and are not reproducible in the literature. |
| Approach: | They propose to improve existing Arabic language pre-trained language models using a more methodical approach. |
| Outcome: | The proposed models outperform existing models on ALUE, a leaderboard-powered benchmark for Arabic NLU and NLG tasks. |
Exploring the Generalizability of Factual Hallucination Mitigation via Enhancing Precise Knowledge Utilization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to improve LLMs' ability to align their responses with objective facts suffer from poor generalization and trade-offs in other different capabilities. |
| Approach: | They propose to introduce PKUE (Precise Knowledge Utilization Enhancement) which fine-tunes the model on self-generated responses to precise and simple factual questions through preference optimization. |
| Outcome: | The proposed enhancements improve LLM’s ability to precisely leverage its knowledge and improve FactualBench, a comprehensive and precise factual QA dataset containing 181k Chinese data spanning 21 domains. |
The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism (2025.naacl-long)
Copied to clipboard
| Challenge: | Current evaluations of large language models (LLMs) focus on a single output per example, which limits our understanding of LLM performance variability in real-world applications. |
| Approach: | They explore the performance differences between greedy decoding and sampling and identify benchmarks’ consistency regarding non-determinism and examine unique model behaviors. |
| Outcome: | The proposed model outperforms sampling methods and greedy decoding outperformed other models. |
Re2-DocRED: Revisiting Revisited-DocRED for Joint Entity and Relation Extraction (2026.eacl-long)
Copied to clipboard
| Challenge: | Document-level Joint Entity and Relation Extraction benchmarks such as DocRED, Re-DocRED, and DocGNRE suffer from pervasive False Negatives (FN) |
| Approach: | They propose a training-free annotation pipeline that leverages user-specifiable reasoning, enriched inverse/co-occurring relation schemas, and novel entity-level constraints to address FN gaps. |
| Outcome: | The proposed pipeline improves on REDFM Mandarin dataset and shows that model recall scores drop on revised splits, whereas the training set mitigates this. |
Fine-grained Semantic Textual Similarity for Serbian (L18-1)
Copied to clipboard
| Challenge: | Semantic textual similarity (STS) is a task of assigning a numerical score to short texts based on the level of semantic equivalence between them. |
| Approach: | They propose to annotate Serbian STS dataset with fine-grained similarity scores . they propose a supervised bag-of-words model that combines part-of speech weighting with term frequency weighting . |
| Outcome: | The proposed model outperforms existing models on the Serbian STS News Corpus . the proposed model is based on a new morphologically rich language . |
Can LLMs Simulate L2-English Dialogue? An Information-Theoretic Analysis of L1-Dependent Biases (2025.acl-long)
Copied to clipboard
Rena Wei Gao, Xuetong Wu, Tatsuki Kuribayashi, Mingrui Ye, Siya Qi, Carsten Roever, Yuanxing Liu, Zheng Yuan, Jey Han Lau
| Challenge: | Large Language Models (LLMs) can simulate non-native-like English use observed in human second language (L2) learners interfered with by their native first language (N1) knowledge. |
| Approach: | They use large language models to simulate non-native-like English use observed in human second language (L2) learners, and then compare their outputs to real L2 learner data. |
| Outcome: | The proposed models replicate L1-dependent patterns observed in human second language (L2) learners, with distinct influences from various languages. |
Few Clean Instances Help Denoising Distant Supervision (2022.coling-1)
Copied to clipboard
| Challenge: | Existing distantly supervised entity relation extractors rely on noisy data for training and evaluation. |
| Approach: | They propose a criterion for clean instance selection based on influence functions to collect sample-level evidence for recognizing good instances. |
| Outcome: | The proposed method shows strong performance on real and synthetic noisy datasets. |
MCP-Guard: A Multi-Stage Defense-in-Depth Framework for Securing Model Context Protocol in Agentic AI (2026.findings-acl)
Copied to clipboard
Wenpeng Xing, Zhonghao Qi, Yupeng Qin, Yilin Li, Caini Chang, Jiahui Yu, Changting Lin, Zhenzhen Xie, Meng Han
| Challenge: | Large Language Models (LLMs) are vulnerable to jailbreak, authors say . authors propose a robust, layered defense architecture designed for LLM–tool interactions . |
| Approach: | They propose a robust, layered defense architecture designed for LLM–tool interactions . they propose XCP-Guard, which employs a three-stage detection pipeline . |
| Outcome: | The proposed model achieves 96.01% accuracy in identifying adversarial prompts . the model is based on a three-stage detection pipeline that balances efficiency with accuracy . |
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing image captioning datasets have limited cross-modal associations, preventing researchers from examining how inter-modal learning impacts intra-modal tasks. |
| Approach: | They propose to use image captioning data to support multi-modal retrieval training and evaluation to assess the impact of inter-modality learning. |
| Outcome: | The proposed model is able to measure the influence of intra- and inter-modality learning. |
DocAMR: Multi-Sentence AMR Representation and Evaluation (2022.naacl-main)
Copied to clipboard
Tahira Naseem, Austin Blodgett, Sadhana Kumaravel, Tim O’Gorman, Young-Suk Lee, Jeffrey Flanigan, Ramón Astudillo, Radu Florian, Salim Roukos, Nathan Schneider
| Challenge: | Abstract Meaning Representation (AMR) graphs are compared to gold graphs by the Smatch metric, but lack a well-defined representation and evaluation. |
| Approach: | They propose an algorithm for deriving a unified graph representation using a super-sentential annotation method. |
| Outcome: | The proposed algorithm avoids the pitfalls of over-merging and lacks coherence from under merging. |
Lessons Learned from a Citizen Science Project for Natural Language Processing (2023.eacl-main)
Copied to clipboard
Jan-Christoph Klie, Ji-Ung Lee, Kevin Stowe, Gözde Şahin, Nafise Sadat Moosavi, Luke Bates, Dominic Petrak, Richard Eckart De Castilho, Iryna Gurevych
| Challenge: | Annotations are expensive and difficult to obtain, which is why many NLP systems outsource their work to paid crowdworkers. |
| Approach: | They propose to use Citizen Science to re-annotate parts of a pre-existing crowdsourced dataset to gain high-quality annotations. |
| Outcome: | The proposed approach yields high-quality annotations and motivated volunteers, but requires consideration of scalability, participation over time, and legal and ethical issues. |
Chat-TS: Enhancing Multi-Modal Reasoning Over Time-Series and Natural Language Data (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models are being rapidly applied across many fields such as healthcare, finance, transportation, and energy. |
| Approach: | They propose a large language model framework that integrates time-series tokens into LLMs’ vocabulary, enhancing its reasoning ability over time- and textual data. |
| Outcome: | The proposed framework enhances reasoning ability over time-series and textual data without compromising core natural language capabilities. |
OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding (2026.acl-long)
Copied to clipboard
Deming Ding, Shichun Liu, Enhui Yang, Jiahang Lin, Ziying Chen, Shihan Dou, Honglin Guo, Weiyu Cheng, Pengyu Zhao, Chengjun Xiao, Qunhong Zeng, Qi Zhang, Xuanjing Huang, Qidi Xu, Tao Gui
| Challenge: | coding scaffolds that follow heterogeneous instructions remain under-examined in software engineering . coding models are capable software agents, but their ability to follow constraints remains under-explored . |
| Approach: | They introduce OctoBench, which benchmarks scaffold-aware instruction following in agentic coding. |
| Outcome: | The proposed benchmark aims to accelerate the development of more scaffold-aware agents. |
RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Efforts to ensure the safety of large language models (LLMs) include safety fine-tuning, evaluation, and red teaming. |
| Approach: | They conduct a comparative analysis of RAG and non-RAG frameworks with eleven LLMs to examine how RAG can make models less safe and change their safety profile. |
| Outcome: | The proposed methods are less effective than those used for non-RAG settings. |
Chinese Morpheme-informed Evaluation of Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing evaluations of large language models focused on the perspective of various tasks or abilities. |
| Approach: | They propose to evaluate large language models from a linguistic perspective and use morpheme to measure morphology and syntax. |
| Outcome: | The proposed model outperforms ChatGPT in Chinese scenarios with a morpheme-informed benchmark and human exam questions. |
A Gold Standard for Multilingual Automatic Term Extraction from Comparable Corpora: Term Structure and Translation Equivalents (L18-1)
Copied to clipboard
| Challenge: | Terms are notoriously difficult to identify, both automatically and manually. |
| Approach: | They propose a method to annotate terms manually from a comparable corpus . they show that the gold standard provides a tool for evaluation and a rich source of information . |
| Outcome: | The proposed method provides a tool for evaluation and rich source of information about terms. |
Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMs (2026.findings-acl)
Copied to clipboard
Yue Huang, Haomin Zhuang, Jiayi Ye, Han Bao, Yanbo Wang, Hang Hua, Siyuan Wu, Pin-Yu Chen, Xiangliang Zhang
| Challenge: | prevailing taxonomies neglect robustness and honesty, yielding safer-on-paper but less useful systems. |
| Approach: | They propose a soft-gating pipeline where a guardian predicts a binary risk label plus a concise explanation and prepends this advice to the original query for re-inference. |
| Outcome: | The proposed model maintains safety while reducing over-refusal. |
Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models (2025.findings-acl)
Copied to clipboard
Shuliang Liu, Xinze Li, Zhenghao Liu, Yukun Yan, Cheng Yang, Zheni Zeng, Zhiyuan Liu, Maosong Sun, Ge Yu
| Challenge: | Existing evaluation metrics cannot fairly evaluate the outputs of RAG models during training and evaluation. |
| Approach: | They propose a method which prompts LLMs to generate different judgments based on various combinations of judgment dimensions and utilizes the judge-consistency to evaluate these judgments. |
| Outcome: | The proposed method generates more accurate evaluations for RAG models across different RAG model and datasets. |
Towards the First Machine Translation System for Sumerian Transliterations (2020.coling-main)
Copied to clipboard
| Challenge: | Sumerian cuneiform script was invented more than 5,000 years ago and is one of the oldest in history. |
| Approach: | They propose to translate Sumerian texts into English automatically using supervised, phrase-based, and transfer learning techniques. |
| Outcome: | The proposed method accelerates the costly and time-consuming manual translation process and helps researchers better explore the relationships between Sumerian and Mesopotamian culture. |
SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature (2025.emnlp-main)
Copied to clipboard
David Wadden, Kejian Shi, Jacob Morrison, Alan Li, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, Doug Downey, Hannaneh Hajishirzi, Arman Cohan
| Challenge: | ScIRIFF is the only entirely expert-written instruction-following dataset for scientific literature understanding . it features complex instructions with long input contexts, detailed task descriptions, and structured outputs. |
| Approach: | They present a dataset of 137K instruction-following instances for training and evaluation . they finetuned large language models using a mix of general domain and ScIRIFF instructions . |
| Outcome: | The proposed dataset shows that on nine out-of-distribution held-out tasks, the model performs better than baselines trained on general domain instructions. |
Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards (2022.findings-emnlp)
Copied to clipboard
Jean-Benoit Delbrouck, Pierre Chambon, Christian Bluethgen, Emily Tsai, Omar Almusa, Curtis Langlotz
| Challenge: | Neural image-to-text radiology report generation systems have been successful on NLG metrics, but they are not factually complete or consistent due to inadequate training and evaluation. |
| Approach: | They propose a method to improve the factual completeness and correctness of generated radiology reports by using a dataset containing annotated chest X-ray images. |
| Outcome: | The proposed method significantly improves factual completeness and correctness of generated radiology reports on two open radiology report datasets. |
A Multi-Persona Chatbot for Hotline Counselor Training (2020.findings-emnlp)
Copied to clipboard
| Challenge: | a chatbot cannot replace a counselor, but a simulation of intimate situations is needed to train counselors. |
| Approach: | They propose a counseling strategy annotation scheme and a multi-task framework that mimics prototype conversations to train counselors. |
| Outcome: | The proposed framework significantly increases response diversity and specificity, with limited impact to coherence. |
Revisiting Generalization Across Difficulty Levels: It’s Not So Easy (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing research is mixed regarding whether training on easier or harder data leads to better results. |
| Approach: | They examine how well large language models generalize across different task difficulties by using a large dataset and a well-established difficulty metric. |
| Outcome: | The results show that training on hard data can't achieve consistent improvements across the full range of difficulties. |
ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | a new system that leverages the encyclopedic knowledge and linguistic reasoning capabilities of Large Language Models (LLMs) is proposed to enhance the productivity of researchers . a researcher's research idea generation process involves problem identification, method development, experiment design and iterative revision . |
| Approach: | They propose a system that leverages encyclopedic knowledge and linguistic reasoning capabilities of Large Language Models to assist researchers in their work. |
| Outcome: | The proposed system generates novel ideas based on human and model-based evaluations . it leverages encyclopedic knowledge and linguistic reasoning capabilities of Large Language Models based systems . |
Minimally-Supervised Morphological Segmentation using Adaptor Grammars with Linguistic Priors (2021.findings-acl)
Copied to clipboard
Ramy Eskander, Cass Lowry, Sujay Khandagale, Francesca Callejas, Judith Klavans, Maria Polinsky, Smaranda Muresan
| Challenge: | Unsupervised morphological segmentation is an essential subtask in many natural language processing applications. |
| Approach: | They introduce two types of priors: grammar definition and linguist-provided affixes . they show that priors boost morphological segmentation performance in a minimally-supervised manner . |
| Outcome: | The proposed priors achieve 8.9% and 34.2% error reductions over the state-of-the-art unsupervised system. |
WIKIGENBENCH:Exploring Full-length Wikipedia Generation under Real-World Scenario (2025.coling-main)
Copied to clipboard
Jiebin Zhang, Eugene J. Yu, Qinyu Chen, Chenhao Xiong, Dawei Zhu, Han Qian, Mingbo Song, Weimin Xiong, Xiaoguang Li, Qun Liu, Sujian Li
| Challenge: | Existing efforts to generate Wikipedia articles for new events fall short of real-world application. |
| Approach: | They propose a benchmark to generate Wikipedia articles for new events under real-world scenarios . they use systematic metrics and LLM-based metrics to assess verifiability, organization, and other aspects aligned with real-life scenarios. |
| Outcome: | The proposed benchmarks show that hierarchical-based methods generate more comprehensive content while fine-tuned methods achieve better verifiability. |
HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts (2023.emnlp-main)
Copied to clipboard
Truong Do, Le Khiem, Quang Pham, TrungTin Nguyen, Thanh-Nam Doan, Binh Nguyen, Chenghao Liu, Savitha Ramasamy, Xiaoli Li, Steven Hoi
| Challenge: | Recent studies suggest that fixing the routers can achieve competitive performance by alleviating the collapsing problem, where all experts eventually learn similar representations. |
| Approach: | They propose a method that dynamically generates router parameters through a fixed hypernetwork and trainable embeddings to achieve a balance between training the routers and freezing them to learn an improved routing policy. |
| Outcome: | Experiments on a wide range of tasks show that the proposed method performs better than existing methods. |
LLMs for Extremely Low-Resource Finno-Ugric Languages (2025.findings-naacl)
Copied to clipboard
| Challenge: | Low-resource languages such as those in the Finno-Ugric family are underrepresented in large language models. |
| Approach: | They propose to develop large language models for extremely low-resource languages . they focus on Vro, Livonian, and Komi, which are underrepresented . |
| Outcome: | The proposed models cover almost the entire cycle of creation, from data collection to instruction tuning and evaluation. |
Learning to Extract Structured Entities Using Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language Models (LMs) play a pivotal role in extracting structured information from unstructured text. |
| Approach: | They propose to reformulate the task to be entity-centric, enabling the use of diverse metrics that can provide more insights from various perspectives. |
| Outcome: | The proposed model outperforms baselines and human evaluations on the extracted entities. |
Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing work on translationese neglects important factors and conclusions are mostly correlational but not causal. |
| Approach: | They use a dataset where MT training data are also labeled with human translation directions to examine the impact of translationese on machine translation evaluation. |
| Outcome: | The proposed model learns in the same direction as human translation directions. |
RepEval: Effective Text Evaluation with LLM Representation (2024.emnlp-main)
Copied to clipboard
Shuqian Sheng, Yi Xu, Tianhang Zhang, Zanwei Shen, Luoyi Fu, Jiaxin Ding, Lei Zhou, Xiaoying Gan, Xinbing Wang, Chenghu Zhou
| Challenge: | Traditional metrics for automatic text evaluation are tailored to specific tasks, while LLM-based evaluation metrics are costly. |
| Approach: | They propose a metric that leverages projections of LLM representations for evaluation. |
| Outcome: | The proposed metric exhibits higher correlation with human judgments than previous methods on 14 datasets. |
SPASM: Stable Persona-driven Agent Simulation for Multi-turn Dialogue Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models are increasingly deployed in multi-turn settings such as tutoring, support, and counseling where reliability depends on preserving consistent roles, personas, and goals across long horizons. |
| Approach: | They propose a framework that decomposes LLM–LLM conversations into a modular, stability-first framework that allows for a stable persona-driven agent simulation for multi-turn dialogue generation. |
| Outcome: | The proposed framework decomposes the LLM-based model into four main components: persona creation, plausibility validation, and natural-language persona crafting. |
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022.emnlp-main)
Copied to clipboard
Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Minjoon Seo
| Challenge: | Language Models (LMs) become outdated as the world changes, a phenomenon called temporal misalignment. |
| Approach: | They propose a lifelong benchmark that utilizes the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation. |
| Outcome: | The proposed benchmark can be trained on the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation. |
EXAMS: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering (2020.emnlp-main)
Copied to clipboard
| Challenge: | EXAMS is a benchmark dataset for cross-lingual and multilingual question answering for high school examinations. |
| Approach: | They propose to use EXAMS to evaluate cross-lingual and multilingual question answering for high school examinations. |
| Outcome: | The proposed model can be used to explore multilingual reasoning and knowledge transfer methods and pre-trained models in schools in different languages, which was not possible by now. |
Benchmarking and Learning Real-World Customer Service Dialogue (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks and training pipelines for industrial intelligent customer service (ICS) focus on task completion and tool correctness. |
| Approach: | They propose a benchmark-to-optimization loop that bridges offline gains to deployment . they propose OlaMind, which distills reusable reasoning patterns from expert dialogues . |
| Outcome: | The proposed benchmark surpasses GPT-5.2 and Gemini 3 Pro on OlaBench . it delivers an average +23.67% issue resolution and -6.6% human transfer rate versus baseline . |
What Prompts Don’t Say: Understanding and Managing Underspecification in LLM Prompts (2026.findings-acl)
Copied to clipboard
| Challenge: | Under-specified prompts are 2x as likely to regress across model or prompt changes, authors show . eliot safina: a lack of explicit prompts can cause frustrations and failures . |
| Approach: | They propose requirements-aware prompt optimization mechanisms that improve performance by 4.8% over baselines. |
| Outcome: | The proposed mechanisms improve prompt performance by 4.8% over baselines. |
Automatically Select Emotion for Response via Personality-affected Emotion Transition (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on rendering specified emotions in responses, yet the individual difference in emotion expression is overlooked. |
| Approach: | They propose to equip a dialog system with personality and enable it to select emotions in responses like humans. |
| Outcome: | The proposed system can select emotions in responses like humans by simulating the emotion transition of humans in conversation. |
ASPECTNEWS: Aspect-Oriented Summarization of News Documents (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for generating generic summarizations can't be used to generalize to these domains without seeing in-domain training data. |
| Approach: | They use a dataset of real-world aspect-oriented summaries to annotate articles from two different news sub-domains. |
| Outcome: | The proposed approach produces better focused summaries than existing systems without seeing in-domain training data. |
IceBATS: An Icelandic Adaptation of the Bigger Analogy Test Set (2022.lrec-1)
Copied to clipboard
| Challenge: | a new test set that measures word embeddings' ability to recognize linguistic regularities is presented in a paper in elijsson, iran . the test sets are a good quality estimator for extrinsic evaluation of word embedded models . |
| Approach: | They propose a test set that measures language models' ability to recognize linguistic regularities in a balanced way. |
| Outcome: | The proposed set is apt at measuring the capabilities of word embedding models. |
Multimodal Table Understanding (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to understanding tables rely on textual inputs and table images are difficult to access in real-world scenarios. |
| Approach: | They propose a multimodal table understanding problem where the model needs to generate correct responses to various table-related requests based on the given table image. |
| Outcome: | The proposed model outperforms open-source MLLMs on 23 benchmarks under held-in and held-out settings. |
On the Robustness of Unsupervised and Semi-supervised Cross-lingual Word Embedding Learning (2020.lrec-1)
Copied to clipboard
| Challenge: | Cross-lingual word embeddings are vector representations of words in different languages where words with similar meaning are represented by similar vectors, regardless of the language. |
| Approach: | They propose to evaluate multiple cross-lingual word embedding models and compare their strengths and limitations to evaluate their effectiveness. |
| Outcome: | The proposed models perform well with noisy text and language pairs with major differences. |
CoLo: A Contrastive Learning Based Re-ranking Framework for One-Stage Summarization (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for extractive and abstractive summarization use token-level or sentence-level training objectives. |
| Approach: | They propose a Contrastive Learning based re-ranking framework for one-stage summarization called CoLo. |
| Outcome: | The proposed framework boosts extractive and abstractive results on CNN/DailyMail benchmarks while maintaining inference efficiency. |
Train One Sparse Autoencoder Across Multiple Sparsity Budgets to Preserve Interpretability and Accuracy (2025.emnlp-main)
Copied to clipboard
Nikita Balagansky, Yaroslav Aksenov, Daniil Laptev, Vadim Kurochkin, Gleb Gerasimov, Nikita Koriagin, Daniil Gavrilov
| Challenge: | Sparse Autoencoders (SAEs) are powerful tools for interpreting neural networks . conventional SAEs are constrained by the fixed sparsity level chosen during training . |
| Approach: | They propose a training objective that trains a single SAE to optimise reconstructions across multiple sparsity levels simultaneously. |
| Outcome: | The proposed objective achieves Pareto-optimal trade-offs between sparsity and explained variance, outperforming traditional SAEs trained at individual sparsities. |
Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset (P19-1)
Copied to clipboard
| Challenge: | EmpatheticDialogues dataset provides a benchmark for empathetic dialogue generation . human evaluators perceive dialogue models as more epathetic . |
| Approach: | They propose a benchmark for empathetic dialogue generation from a dataset of 25k conversations grounded in emotional situations. |
| Outcome: | The proposed benchmarks show that existing models are perceived to be more empathetic by human evaluators compared to models trained on large-scale Internet conversations. |
Are Checklists Really Useful for Automatic Evaluation of Generative Tasks? (2025.emnlp-main)
Copied to clipboard
| Challenge: | evaluators using large language models face ambiguous criteria and inconsistent evaluations. |
| Approach: | They investigate whether checklists should be used for all questions or selectively . they generate checklists using six methods and evaluate their effectiveness across eight models . |
| Outcome: | The proposed method improves evaluation performance in pairwise comparisons while ignoring human-written criteria. |
Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multi-document summarization (MDS) assumes a set of topic-related documents is provided as input. |
| Approach: | They formalize the task and bootstrap it using existing datasets, retrievers and summarizers. |
| Outcome: | The proposed method reduces the sensitivity of summarizers to imperfect retrieval, but is highly sensitive to other errors. |
FormLM: Recommending Creation Ideas for Online Forms by Modelling Semantic and Structural Information (2022.emnlp-main)
Copied to clipboard
| Challenge: | FormLM is a pre-trained language model for creating semi-structured forms where questions and descriptions are organized by predefined structures. |
| Approach: | They propose to enhance pre-trained language model with form structural information to model online forms and recommend form creation ideas. |
| Outcome: | The proposed model outperforms general-purpose language models on all tasks, with an improvement by 4.71 on Question Recommendation and 10.6 on Block Type Suggestion in terms of ROUGE-1 and Macro-F1, respectively. |
Cognitive-Level Adaptive Generation via Capability-Aware Retrieval and Style Adaptation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models struggle to adapt content to users with differing cognitive capacities, leading to cognitive misalignment. |
| Approach: | They propose a cognitive-level alignment framework that aligns both knowledge complexity and presentation style with user cognition. |
| Outcome: | The proposed framework aligns knowledge complexity and presentation style with user cognition. |
A Closer Look into Using Large Language Models for Automatic Evaluation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies explore the use of large language models to evaluate text quality, but they differ in some details of the evaluation process. |
| Approach: | They propose to use large language models to evaluate text quality by giving LLMs instructions to evaluate samples by giving them a rating. |
| Outcome: | The auto Chain-of-Thought (CoT) used in G-Eval does not always make it more aligned with human ratings. |
FaMTEB: Massive Text Embedding Benchmark in Persian Language (2025.findings-emnlp)
Copied to clipboard
Erfan Zinvandi, Morteza Alikhani, Mehran Sarmadi, Zahra Pourbahman, Sepehr Arvin, Reza Kazemi, Arash Amini
| Challenge: | a comprehensive benchmark for Persian text embeddings is built upon the Massive Text Embedding Benchmark (MTEB) 63 datasets are included in the benchmark, including a novel task of summary retrieval. |
| Approach: | They propose a benchmark for Persian (Farsi) text embeddings built upon the Massive Text Embedding Benchmark. |
| Outcome: | The proposed framework includes 63 datasets spanning seven different tasks . the evaluation datasets were rigorously evaluated by humans and automated systems . |
Neural Topic Modeling based on Cycle Adversarial Training and Contrastive Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Neural topic models have been widely used to extract common topics across documents. |
| Approach: | They propose a framework to apply contrastive learning directly to the decoder . they propose 'self-supervised' contrastive loss to make the generator capture similar topic information . |
| Outcome: | The proposed framework outperforms baselines on four benchmark datasets. |
Towards Advanced Mathematical Reasoning for LLMs via First-Order Logic Theorem Proving (2025.emnlp-main)
Copied to clipboard
Chuxue Cao, Mengze Li, Juntao Dai, Jinluan Yang, Zijian Zhao, Shengyu Zhang, Weijie Shi, Chengzhong Liu, Sirui Han, Yike Guo
| Challenge: | Large language models (LLMs) have shown promising first-order logic (FOL) reasoning capabilities with applications in various areas, but their effectiveness in complex mathematical reasoning involving multi-step FOL deductions remains under-explored. |
| Approach: | They propose a self-adaptive solution that enhances the Diversity and REAsonability of LLMs’ generation strategies by introducing an Axiom-Driven Strategy Diversification mechanism and a Sub-Proposition Error Feedback to help LLM reflect on and correct their proofs. |
| Outcome: | The proposed model improves diversity and REAsonability of LLMs’ generation strategies by introducing an Axiom-Driven Strategy Diversification mechanism and a Sub-Proposition Error Feedback to help LLM reflect on and correct proofs. |
Don’t Copy the Teacher: Data and Model Challenges in Embodied Dialogue (2022.emnlp-main)
Copied to clipboard
| Challenge: | Embodied dialogue instruction following requires an agent to complete a complex sequence of tasks from a natural language exchange. |
| Approach: | They argue that imitation learning and low-level metrics are misleading . they compare existing models with IL and argue evaluation should focus on higher-level semantic goals . |
| Outcome: | The proposed model evaluations are based on three models and compare them with benchmarks . they show that existing models fail to ground query utterances, which are essential for task completion . |
A Modular Architecture for Unsupervised Sarcasm Generation (D19-1)
Copied to clipboard
| Challenge: | Existing systems for sarcasm generation are elusive due to the fact that both selection of contents and training of sarcasm are based on the same data. |
| Approach: | They propose a framework that takes a literal negative opinion as input and translates it into a sarcastic version. |
| Outcome: | The proposed system outperforms baselines built using known unsupervised statistical and neural machine translation and style transfer techniques. |
MEE: A Novel Multilingual Event Extraction Dataset (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for Event Extraction are limited for non-English languages . lack of high-quality multilingual datasets has been the main hindrance . |
| Approach: | They propose a multilingual event extraction dataset that provides annotation for more than 50K event mentions in 8 typologically different languages. |
| Outcome: | The proposed dataset provides annotation for more than 50K event mentions in 8 languages . the proposed dataset will be publicly available to foster future research . |
Smart Word Suggestions for Writing Assistance (2023.findings-acl)
Copied to clipboard
| Challenge: | Using word suggestions, writing assistance is a widely used application of natural language processing (NLP) . a task is performed to identify words or phrases that require improvement and provide substitution suggestions for each improvable target. |
| Approach: | They propose a task and benchmark to help writers improve word usage . they use human-labeled data and a distantly supervised dataset for testing . |
| Outcome: | The proposed task and benchmark aims to improve word usage in writing aids. |
SafeLawBench: Towards Safe Alignment of Large Language Models (2025.findings-acl)
Copied to clipboard
Chuxue Cao, Han Zhu, Jiaming Ji, Qichao Sun, Zhenghao Zhu, Wu Yinyu, Josef Dai, Yaodong Yang, Sirui Han, Yike Guo
| Challenge: | Recent studies indicate that large language models (LLMs) may exhibit risks, including threats to the protection of private data and the generation of hallucinations. |
| Approach: | They propose to evaluate LLMs from a legal perspective using the SafeLawBench benchmark. |
| Outcome: | The proposed framework categorizes safety risks into three levels based on legal standards and includes 24,860 multi-choice questions and 1,106 open-domain question-answering tasks. |
InFoBench: Evaluating Instruction Following Ability in Large Language Models (2024.findings-acl)
Copied to clipboard
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, Dong Yu
| Challenge: | Existing methods for evaluating Large Language Models (LLMs) ability to follow instructions have not been able to provide a detailed analysis of their compliance with instructions. |
| Approach: | They propose a new metric for evaluating Large Language Models' ability to follow instructions and a benchmark for DRFR. |
| Outcome: | The proposed metric and benchmark compared with traditional scoring methods and explores annotation sources including human experts, crowd-sourced workers, and GPT-4. |
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies on compression of pretrained language models usually use preserved accuracy as the metric for evaluation. |
| Approach: | They propose two new metrics that measure how closely a compressed model mimics the original model. |
| Outcome: | The proposed metrics measure how closely a compressed model (i.e., student) mimics the original model (e.g., teacher). |
Can We Edit Multimodal Large Language Models? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to edit multimodal models have been used to incrementally infuse a language model with a new set of facts. |
| Approach: | They construct a benchmark for editing multimodal Large Language Models and establish metrics for evaluation. |
| Outcome: | The proposed benchmarks show that editing multimodal models is not as difficult as editing single-modal models. |
BIASEDTALES-ML: A Multilingual Dataset for Analyzing Narrative Attribute Distributions in LLM-Generated Stories (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on the use of Large Language Models (LLMs) focus primarily on English, leaving the cross-lingual generalization of aligned behavior underexplored. |
| Approach: | They propose a structured generator-extractor pipeline and a multi-dimensional distributional analysis framework to examine how narrative attributes vary across languages, models, and social conditions. |
| Outcome: | The proposed model reveals substantial cross-lingual variability in narrative generation patterns, indicating that distributions observed in English do not always exhibit similar characteristics in other languages, particularly in lower-resource settings. |
Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling (2024.lrec-main)
Copied to clipboard
| Challenge: | Topic modelling has found extensive use in automatically detecting significant topics within a corpus of documents, but there are certain drawbacks. |
| Approach: | They propose a framework that prompts large language models to generate topics from a given set of documents and establish evaluation protocols to assess the clustering efficacy of LLMs. |
| Outcome: | The proposed model generates relevant topic titles and adheres to human guidelines to refine and merge topics. |
FTibSuite: A Comprehensive Resource Suite for Tibetan Vision–Language Modeling (2026.findings-acl)
Copied to clipboard
| Challenge: | FTibSuite provides an end-to-end training-and-evaluation workflow for vision–language models . Tibetan is underserved due to the lack of infrastructure for reproducible training and evaluation. |
| Approach: | They propose a resource-centric workflow for Tibetan VLMs that provides an end-to-end training-and-evaluation workflow and human-verified multimodal annotations. |
| Outcome: | FTibSuite provides an end-to-end training-and-evaluation workflow and human-verified multimodal annotations. |
DEplain: A German Parallel Corpus with Intralingual Translations into Plain Language for Sentence and Document Simplification (2023.acl-long)
Copied to clipboard
| Challenge: | Current text simplification research mostly focuses on English and on sentencelevel simplification. |
| Approach: | They propose to use a dataset of parallel, professionally written and manually aligned simplifications in plain German "plain DE" and "Einfache Sprache" they build a web harvester and experiment with automatic alignment methods to facilitate integration of non-aligned and to be-published parallel documents. |
| Outcome: | The proposed dataset of parallel, professionally written and manually aligned simplifications in plain German is extended to 750 document pairs and 3.5k sentence pairs. |
Locally Measuring Cross-lingual Lexical Alignment: A Domain and Word Level Perspective (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a cognitive science research focus on aligning language spaces in their entirety . but, cognitive science has long focused on a local perspective . a new method for cross-lingual lexical alignment requires some methodology . |
| Approach: | They propose a method for analyzing kinship domain kinematics and a new method for contextualization . they propose kin-level validations and contextualizations to validate the results . |
| Outcome: | The proposed method analyzes synthetic validations and naturalistic validations using lexical gaps in the kinship domain. |
Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning (2024.emnlp-main)
Copied to clipboard
Abhinav Bandari, Lu Yin, Cheng-Yu Hsieh, Ajay Jaiswal, Tianlong Chen, Li Shen, Ranjay Krishna, Shiwei Liu
| Challenge: | Existing approaches to prune LLMs rely on the C4 dataset as calibration data . arithmetic datasets perform better than pre-training datasets for pruning, whereas chain-of-thought is only useful on certain tasks. |
| Approach: | They evaluate the selection of calibration data for LLM pruning across a wide range of datasets . they find that C4 is not the optimal calibration data, and that CoT is only useful on certain tasks. |
| Outcome: | The chosen calibration data significantly impacts the performance of pruned LLMs, the authors found . their results shed light on the importance of carefully selecting calibration data for LLM pruning . |
Argue with Me Tersely: Towards Sentence-Level Counter-Argument Generation (2023.emnlp-main)
Copied to clipboard
Jiayu Lin, Rong Ye, Meng Han, Qi Zhang, Ruofei Lai, Xinyu Zhang, Zhao Cao, Xuanjing Huang, Zhongyu Wei
| Challenge: | Existing work describes paragraph-level counter-argument generation task as paragraph-based . however, sentence-level generation can be quite different due to its unique constraints and brevity-focused challenges. |
| Approach: | They propose a benchmark framework for sentence-level counter-argument generation . they use an annotated debate forum dataset to generate high-quality counter-argments . |
| Outcome: | The proposed framework and evaluator are competitive in counter-argument generation tasks. |
Looking Beyond the One: Operationalizing and Eliciting Visual Ambiguity in VLLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Visual question answering systems typically collapse ambiguity, committing to a single interpretation during decoding and evaluation. |
| Approach: | They operationalize ambiguity as the existence of multiple answer-supporting regions in an image . they show that ambiguities are already encoded in their internal representations . |
| Outcome: | The proposed approach makes ambiguity observable without exhaustive annotations . ambiguities are already encoded in models, but not reliably expressed in outputs despite hidden states . |
Interpretable Text Embeddings and Text Similarity Explanation: A Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text embeddings are a fundamental component in many NLP tasks, but their interpretation and explanation remain challenging. |
| Approach: | They propose a framework for interpretable text embeddings and text similarity explanation . they characterize the main ideas, approaches, and trade-offs and discuss lessons learned . |
| Outcome: | The proposed methods are compared with existing models and compare them with existing ones. |
Mind the Style Gap: Meta-Evaluation of Style and Attribute Transfer Metrics (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) make it easy to rewrite a text in any style, but they are not straightforward when evaluating content preservation. |
| Approach: | They propose a large meta-evaluation of metrics for evaluating style and attribute transfer, focusing on content preservation. |
| Outcome: | The proposed method achieves higher alignment with human judgements than prompting a model of a similar size as an autorater. |
Can Out-of-Distribution Evaluations Uncover Reliance on Prediction Shortcuts? A Case Study in Question Answering (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work assesses models’ generalization capabilities through the lens of performance on out-of-distribution (OOD) datasets. |
| Approach: | They challenge this assumption by comparing OOD evaluations with failure modes documented in existing question-answering (QA) models. |
| Outcome: | The proposed evaluations show that the models' generalization capabilities are under-performing on out-of-distribution datasets, while others are underperforming on in-difference datasets. |
COVER: Context-Driven Over-Refusal Verification in LLMs (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have become increasingly prevalent in the field of Natural Language Processing (NLP), achieving unprecedented performance across linguistic tasks. |
| Approach: | They propose a framework to quantify and analyze context-driven over-refusal . they find that over-fusals depend on the task, system prompts, model family, and the number of retrieved documents. |
| Outcome: | The proposed framework quantifyes and analyzes the concept of context-driven over-refusal on two public corpora. |
SI-NLI: A Slovene Natural Language Inference Dataset and Its Evaluation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing datasets for natural language inference (NLI) are limited to English and a few other well-resourced languages. |
| Approach: | They propose to use a dataset for natural language inference to extend the resources for the task. |
| Outcome: | The proposed dataset is constructed from scratch using knowledgeable annotators with carefully crafted guidelines aiming to avoid common problems in existing datasets. |
QEVA: A Reference-Free Evaluation Metric for Narrative Video Summarization with Multimodal Question Answering (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing video-to-text summarization evaluation methods depend heavily on human-written reference summaries. |
| Approach: | They propose a reference-free metric evaluating candidate summaries directly against source videos through multimodal question answering. |
| Outcome: | The proposed metric assesses candidate summaries directly against source videos through multimodal question answering. |
tasksource: A Large Collection of NLP tasks with a Structured Dataset Preprocessing Framework (2024.lrec-main)
Copied to clipboard
| Challenge: | Several initiatives release harmonized datasets or provide harmonization codes to preprocess datasets into a consistent format. |
| Approach: | They propose an annotation framework that enables concise, readable, and reusable annotations. |
| Outcome: | The proposed framework outperforms all publicly available text encoders on all tasks. |
A Constrained Text Revision Agent via Iterative Planning and Searching (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing text revision systems are capable of generating fluent and coherent text, but struggle with constrained text revision (CTR). |
| Approach: | They propose a tool that generates revisions tailored to different scenarios using a planner, a reviser and adaptable tools. |
| Outcome: | The proposed agent outperforms baseline approaches in both constraint adherence and revision quality. |
Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | NLCO evaluates large language models for combinatorial optimization (CO) . existing evaluations emphasize relatively simple reasoning competencies . |
| Approach: | They propose a combinatorial optimization benchmark that evaluates large language models on CO reasoning. |
| Outcome: | The proposed model can handle combinatorial optimization without writing code or calling external solvers. |
AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists (2025.emnlp-main)
Copied to clipboard
Yifei Li, Hanane Nour Moussa, Ziru Chen, Shijie Chen, Botao Yu, Mingyi Xue, Benjamin Burns, Tzu-Yao Chiu, Vishal Dey, Zitong Lu, Chen Wei, Qianheng Zhang, Tianyu Zhang, Song Gao, Xuhui Huang, Xia Ning, Nesreen K. Ahmed, Ali Payani, Huan Sun
| Challenge: | AutoSDT-5K is the only automatically collected and the largest open dataset for data-driven scientific discovery. |
| Approach: | They propose an automatic pipeline that collects high-quality coding tasks in real-world data-driven discovery workflows. |
| Outcome: | The proposed pipeline synthesizes accurate tasks and tasks from a dataset of 5,404 tasks covering four scientific disciplines and 756 Python packages. |
CiteEval: Principle-Driven Citation Evaluation for Source Attribution (2025.acl-long)
Copied to clipboard
Yumo Xu, Peng Qi, Jifan Chen, Kunlun Liu, Rujun Han, Lan Liu, Bonan Min, Vittorio Castelli, Arshit Gupta, Zhiguo Wang
| Challenge: | Current evaluation frameworks rely on NLI to assess binary or ternary support from cited sources, which is suboptimal for citation evaluation. |
| Approach: | They propose a citation evaluation framework based on fine-grained citation ratings within a broad context and construct a multi-domain benchmark with high-quality human annotations. |
| Outcome: | The proposed framework provides a high-quality human annotation benchmark and a suite of model-based metrics that exhibit strong correlation with human judgments. |
MMDocIR: Benchmarking Multimodal Retrieval for Long Documents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for multimodal document retrieval are lacking for evaluating performance of systems. |
| Approach: | They propose a benchmark that evaluates page-level and layout-level retrieval tasks . they use a rich dataset featuring 1,685 questions annotated by experts . |
| Outcome: | The proposed benchmark outperforms existing benchmarks in page-level and layout-level retrieval tasks. |
Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for learning general-purpose audio representations are limited in scope and coverage of audio attributes. |
| Approach: | They propose to use a 10.7M caption dataset to compare ALP with captioning . they find that contrastive learning yields competitive, transferable representations . |
| Outcome: | The proposed model yields competitive, transferable representations, while captioning exhibits better scalability. |
From Scores to Preferences: Redefining Evaluation Paradigm for Speech Quality Reward Modeling (2026.findings-acl)
Copied to clipboard
Yifei Cao, Changhao Jiang, Jiabao Zhuang, Jiajun Sun, Ming Zhang, Zhiheng Xi, Hui Li, Shihan Dou, Yuran Wang, Yunke Zhang, Tao Ji, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Experimental results show that the MOS-aware GRM significantly improves fine-grained speech quality discrimination. |
| Approach: | They propose a MOS-aware reward model that incorporates MOS gap into reward function during reinforcement learning. |
| Outcome: | The proposed model significantly improves fine-grained speech quality discrimination. |
CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward (2025.emnlp-main)
Copied to clipboard
Shudong Liu, Hongwei Liu, Junnan Liu, Linchen Xiao, Songyang Gao, Chengqi Lyu, Yuzhe Gu, Wenwei Zhang, Derek F. Wong, Songyang Zhang, Kai Chen
| Challenge: | Existing approaches lack robustness to handle complex edge cases and generalizability across different domains. |
| Approach: | They develop an accurate and lightweight verifier model for evaluation and outcome reward that matches unstructured outputs against standard answers. |
| Outcome: | The proposed model can process multiple answer types including multi-subproblems, formulas, and sequence answers while identifying abnormal/invalid responses. |
Idea First, Code Later: Disentangling Problem Solving from Code Generation in Evaluating LLMs for Competitive Programming (2026.findings-acl)
Copied to clipboard
Sama Hadhoud, Alaa Elsetohy, Frederikus Hudi, Jan Christian Blaise Cruz, Steven Halim, Alham Fikri Aji
| Challenge: | Existing evaluations conflate algorithmic reasoning with code-level implementation. |
| Approach: | They propose to center editorials in both solution generation and evaluation . they propose to compare editorials to gold standards and validate an LLM-as-a-judge protocol . |
| Outcome: | The proposed approach improves solve rates on some LLMs with gold editorials . but the gap between gold and generated editorials shows bottlenecks in implementation . |
AI Agents for the Science of Science: A Survey of Tasks, Architectures, Evaluations, and Challenges (2026.findings-acl)
Copied to clipboard
| Challenge: | The Science of Science (SciSc) examines how scientific knowledge is produced, evaluated, and transformed by utilizing large-scale scholarly and bibliometric data. |
| Approach: | They propose a task-centered taxonomy for AI agents that model citations, collaborations, and community dynamics. |
| Outcome: | The proposed taxonomy distinguishes agents as simulations from tools for empirical analysis and scientific workflows. |
Can Large Language Models Effectively Support Decision-Making in Sudden Emergencies? (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing research has focused on the earlier stages of emergency response . lack of suitable datasets for reliable and compliance-aware decision-oriented modeling and evaluation is limiting current research . |
| Approach: | They propose a first real-world emergency decision-making dataset EDM-Bench . they propose 'rule-enhanced reasoning framework' that integrates external regulatory knowledge with constrained inference mechanisms to improve both decision safety and interpretability. |
| Outcome: | The proposed framework improves decision safety and interpretability by integrating regulatory knowledge with constrained inference mechanisms. |
Simulated Students in Tutoring Dialogues: Substance or Illusion? (2026.acl-long)
Copied to clipboard
| Challenge: | evaluating the effectiveness of new technology requires real students, which is time-consuming and hard to scale up. |
| Approach: | They propose to define the student simulation task and benchmark a wide range of student simulation methods on these metrics. |
| Outcome: | The proposed evaluation metrics show that prompting strategies perform poorly on a real-world tutoring dialogue dataset. |
SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models (2026.acl-long)
Copied to clipboard
Yiyang Gu, Junwei Yang, Junyu Luo, Ye Yuan, Bin Feng, Yingce Xia, Shufang Xie, Kaili Liu, Bohan Wu, Qi Shi, Haoran Li, Beier Xiao, Zhiping Xiao, Xiao Luo, Weizhi Zhang, Philip S. Yu, Zequn Liu, Ming Zhang
| Challenge: | Existing evaluations of large language models fail to reflect fine-grained capabilities . existing benchmarks are manually curated or domain-generic, limiting scalability and alignment with real use cases. |
| Approach: | They propose a framework that allows custom construction of benchmarks from large-scale scientific data to evaluate application-specific scientific capabilities in LLMs. |
| Outcome: | The proposed framework reveals fine-grained differences in scientific capabilities that standard benchmarks overlook . it allows custom construction of benchmarks from large-scale scientific data to evaluate application-specific capabilities in LLMs. |
UniSRM: A Unified Speech Reward Model for Reasoning-Based Fine-grained Assessment (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for speech generation rely on subjective, expensive judgments . Existing models only cover a narrow set of scenarios and only provide limited coverage . |
| Approach: | They propose a unified speech reward model that can support multi-dimensional, interpretable reward signals with reliable reasoning. |
| Outcome: | The proposed model can support multi-dimensional, interpretable reward signals with reliable reasoning. |
From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP (2026.acl-long)
Copied to clipboard
Adithya V Ganesan, Vasudha Varadarajan, Oscar Kjell, Whitney Ringwald, Scott M. Feltman, Benjamin J. Luft, Roman Kotov, Ryan L. Boyd, H. Andrew Schwartz
| Challenge: | a longitudinal model for NLP relies on document-level evaluation to map isolated instances of language to an outcome. |
| Approach: | They propose a longitudinal model that aligns evaluation splits to generalization over people and time . they propose integrating a sequence inputs to incorporate history by default . |
| Outcome: | The proposed model improves on a dataset of 17k daily diary transcripts paired with PTSD symptom severity from 238 participants. |