Papers with GPT-4
ActiveLLM: Large Language Model-Based Active Learning for Textual Few-Shot Scenarios (2026.tacl-1)
Copied to clipboard
| Challenge: | Active learning strategies struggle with a ‘cold-start’ problem, needing substantial initial data to be effective. |
| Approach: | They propose an active learning approach that leverages Large Language Models such as GPT-4, o1, Llama 3, or Mistral Large for selecting instances. |
| Outcome: | The proposed approach outperforms existing methods ADAPET, PERFECT, and SetFit in few-shot scenarios and can be extended to non-few scenarios. |
Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translation (2025.naacl-srw)
Copied to clipboard
| Challenge: | Large language models excel in machine translation, but most studies focus on sentence-level translation. |
| Approach: | They propose to use LLMs as a judge paradigm to evaluate document-level translations by directly prompting them to translate entire documents in a single pass. |
| Outcome: | The proposed method improves translation quality even without document-level fine-tuning compared to translating sentences separately . |
FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models (2024.acl-short)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models focus on intradocument dependencies or dependencies between a small number of documents. |
| Approach: | They propose to use a dataset of fan-out question-answer pairs and human-annotated decompositions with English Wikipedia as the knowledge base to evaluate models' reasoning. |
| Outcome: | The proposed dataset shows that models still have room to improve reasoning over inter-document dependencies in a long context. |
Struc-Bench: Are Large Language Models Good at Generating Complex Structured Tabular Data? (2024.naacl-short)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have advanced capabilities but produce complex structured data. |
| Approach: | They propose a structure-aware fine-tuning method to bolster LLMs' performance by crafting format-specific instructions from the intended outputs. |
| Outcome: | The proposed method outperforms LLMs on all three formats and spans text tables, HTML, and LaTeX formats. |
CogLM: Tracking Cognitive Development of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have recently shown remarkable abilities across a wide variety of tasks, but few studies have explored the reasons behind the evolutionary relationship among various abilities. |
| Approach: | They construct a benchmark CogLM based on Piaget's Theory of Cognitive Development (PTC) which measures the cognitive levels of Large Language Models (LLMs) using 1,220 questions spanning 10 cognitive abilities crafted by more than 20 human experts. |
| Outcome: | The proposed framework provides a comprehensive testbed for the cognitive levels of LLMs. |
SEAVER: Attention Reallocation for Mitigating Distractions in Language Models for Conditional Semantic Textual Similarity Measurement (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Conditional Semantic Textual Similarity (C-STS) introduces specific limiting conditions to the traditional Semantics task. |
| Approach: | They propose a conditional semantic textual similarity (C-STS) task that introduces specific limiting conditions to the traditional Semantic Textual Similarity task. |
| Outcome: | The proposed model outperforms existing models on the C-STS-2023 test set and consistently improves on million-scale fine-tuning baseline models (up to 3 points). |
Predicting Entity Salience in Extremely Short Documents (2024.emnlp-industry)
Copied to clipboard
| Challenge: | False positive: ES is a natural language understanding task that determines which entities are most salient to a passage . Falsity: Popsicle, Frank Epperson and San Francisco are salient entities . |
| Approach: | They propose a lightweight and data-efficient approach for entity salience detection on short documents . they propose he use of a human-labeled dataset to evaluate entity salient on short questions . |
| Outcome: | The proposed approach achieves competitive performance over state-of-the-art models at significant cost and latency advantages. |
EasyJudge: an Easy-to-use Tool for Comprehensive Response Evaluation of LLMs (2025.coling-demos)
Copied to clipboard
| Challenge: | Existing open-source evaluation models lack a user-friendly visualization tool and are not optimized for accelerated model inference. |
| Approach: | They propose to use open-source evaluation models to evaluate language model responses. |
| Outcome: | The proposed model is lightweight, precise, efficient, and user-friendly, with an intuitive visualization interface for ease of deployment and use. |
Exploring Factual Entailment with NLI: A News Media Study (2024.starsem-1)
Copied to clipboard
| Challenge: | Recent studies have focused on the relationship between factuality and Natural Language Inference (NLI). |
| Approach: | They propose a novel annotation scheme that models factual rather than textual entailment and use it to annotate a dataset of naturally occurring sentences from news articles. |
| Outcome: | The proposed annotation scheme can be used to model factual relationships on a dataset of naturally occurring sentences from news articles. |
Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work on LLMs that only enhance reasoning abilities, but which lack factual hallucination and slow-thinking capabilities, argues that SPP is a cognitive synergist. |
| Approach: | They propose a Solo Performance Prompting (SPP) that transforms a single LLM into a cognitive synergist by engaging in multi-turn self-collaboration with multiple personas. |
| Outcome: | The proposed model reduces factual hallucination and maintains strong reasoning abilities on three challenging tasks . |
Unleashing the Unseen: Harnessing Benign Datasets for Jailbreaking Large Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Despite significant efforts in safety alignment, large language models (LLMs) such as GPT-4 and LLaMA 3 remain vulnerable to jailbreak attacks that can induce harmful behaviors. |
| Approach: | They propose a feature extraction method to extract sample-agnostic features from benign datasets in the form of adversarial suffixes and propose 'suffix maybe features' they show that adversarials generated from jailbreak attacks may contain meaningful features, i.e. appending the same suffix to different prompts results in responses exhibiting specific characteristics. |
| Outcome: | The proposed method extracts sample-agnostic features from benign datasets and shows that they may contain meaningful features. |
Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated state-of-the-art (SOTA) performance across a wide spectrum of tasks. |
| Approach: | They propose a framework that leverages three signal types to improve efficiency within resource-constrained, imperfect teacher scenarios. |
| Outcome: | The proposed framework improves on four complex reasoning tasks by 20.79% compared to fine-tuning without any signals across datasets. |
Small Models are Valuable Plug-ins for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models are difficult to fine-tune due to their huge weights and limited context length. |
| Approach: | They propose an approach which allows black-box LLMs to work with locally fine-tuned smaller models, resulting in superior performance on supervised tasks. |
| Outcome: | The proposed approach overcomes the challenges of poor performance and instability of In-Context Learning (ICL) while reducing the complexity of in-context learning. |
Can Uniform Meaning Representation Help GPT-4 Translate from Indigenous Languages? (2025.acl-short)
Copied to clipboard
| Challenge: | In this study, we examine the downstream utility of Uniform Meaning Representation (UMR) for low-resource languages. |
| Approach: | They explore the utility of Uniform Meaning Representation (UMR) for low-resource languages by incorporating it into GPT-4 prompts. |
| Outcome: | The proposed model performs better than existing models in Navajo, Arápaho, and Kukama with and without demonstrations and annotations. |
Translate With Care: Addressing Gender Bias, Neutrality, and Reasoning in Large Language Model Translations (2025.findings-acl)
Copied to clipboard
| Challenge: | addressing gender bias and maintaining logical coherence in machine translation remains challenging, especially when translating between natural gender languages, like English, and genderless languages, such as Persian, Indonesian, and Finnish. |
| Approach: | They propose a dataset to assess translation systems' performance in six low- to mid-resource languages and a translation dataset to examine gender bias and logical coherence. |
| Outcome: | The Translate-with-Care dataset, comprising 3,950 challenging scenarios across six low- to mid-resource languages, reveals a universal struggle in translating genderless content, resulting in gender stereotyping and reasoning errors. |
TransLLaMa: LLM-based Simultaneous Translation System (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Decoder-only large language models have limited applications in simultaneous machine translation . naively translating each source word immediately results in compromised target quality . |
| Approach: | a study shows that a pre-trained open-source LLM can control input segmentation directly by generating a special "wait" token. |
| Outcome: | a new open-source model can control input segmentation directly by generating a special "wait" token. |
BookSQL: A Large Scale Text-to-SQL Dataset for Accounting Domain (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing models for accounting databases that can be queried using natural language are lacking in some domains. |
| Approach: | They propose a large-scale text-to-SQL dataset for accounting and financial domains . they propose 'bookSQl' to be used to query accounting databases using natural language . |
| Outcome: | The proposed model performs poorly on the existing model, pointing towards a more focused model for this domain. |
Can LLMs Augment Low-Resource Reading Comprehension Datasets? Opportunities and Challenges (2024.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive zero-shot performance on a wide range of NLP tasks. |
| Approach: | They propose to use large language models to augment extractive reading comprehension datasets by fine-tuning their annotations and comparing their performance to human annotators. |
| Outcome: | The proposed model can be used to augment extractive reading comprehension datasets. |
MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL (2025.coling-main)
Copied to clipboard
Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, LinZheng Chai, Zhao Yan, Qian-Wen Zhang, Di Yin, Xing Sun, Zhoujun Li
| Challenge: | Recent LLM-based Text-to-SQL methods suffer from performance degradation on “huge” databases and complex user questions that require multi-step reasoning. |
| Approach: | They propose a framework that integrates a decomposer agent and auxiliary agents to generate SQL queries from natural language text. |
| Outcome: | The proposed framework achieves comparable execution accuracy on SQL-Llama tasks compared to the baseline model. |
Large Language Models can Share Images, Too! (2024.findings-acl)
Copied to clipboard
| Challenge: | Using a zero-shot prompting, large language models can be used to share images in a multi-tasking environment. |
| Approach: | They introduce a dataset that includes enriched annotations and a framework to evaluate LLMs. |
| Outcome: | The proposed framework unlocks image-sharing capability of LLMs in zero-shot prompting, with ChatGPT achieving the best performance. |
Evaluating Credibility and Political Bias in LLMs for News Outlets in Bangladesh (2025.acl-srw)
Copied to clipboard
| Challenge: | Large language models (LLMs) are widely used in search engines to provide direct an-swers, while AI chatbots retrieve updated infor-mation from the web. |
| Approach: | They audit nine Large Language Models from OpenAI, Google, and Meta to assess their ability to eval-uate the credibility and political bias of the top20 most popular news outlets in Bangladesh. |
| Outcome: | The proposed models show internal consistency in credibil-ity ratings, but misalignment with human experts. |
SeqAR: Jailbreak LLMs with Sequential Auto-Generated Characters (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on the potential misuse of large language models (LLMs) however, the ability to align LLMs with human values is still vulnerable to malicious attacks. |
| Approach: | They propose a red-teaming strategy to enhance LLM safety by using a framework to design jailbreak prompts automatically. |
| Outcome: | The proposed framework achieves attack success rates of 88% and 60% in cold-start scenarios. |
Streamlining LLMs: Adaptive Knowledge Distillation for Tailored Language Models (2025.naacl-srw)
Copied to clipboard
| Challenge: | Large language models (LLMs) have transformative potential across industries, e.g., enhancing customer service, revolutionizing medical diagnostics, or identifying crises in news articles. |
| Approach: | They propose to distill compact, parameter-efficient tailored language models from LLMs for domain-specific tasks with comparable performance. |
| Outcome: | The proposed framework outperforms knowledge distillation frameworks in the crisis domain, where labeled data is scarce. |
A Continued Pretrained LLM Approach for Automatic Medical Note Generation (2024.naacl-short)
Copied to clipboard
Dong Yuan, Eti Rastogi, Gautam Naik, Sree Prasanna Rajagopal, Sagar Goyal, Fen Zhao, Bharath Chintagunta, Jeffrey Ward
| Challenge: | HEAL is the first continuously trained LLaMA2-based LLM for medical conversations . despite the success of LLMs in general capabilities, they often fall short in niche domains like healthcare . |
| Approach: | They propose a 13B LLaMA2-based LLM that is purpose-built for medical conversations and measured on automated scribing. |
| Outcome: | The HEAL LLM outperforms GPT-4 and PMC-LLaMA in PubMedQA with 78.4% accuracy and parity with GPT-LLAMA in generating medical notes. |
KorSmishing Explainer: A Korean-centric LLM-based Framework for Smishing Detection and Explanation Generation (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing models that detect smishing have a high accuracy but lack interpretability, which undermines user trust and practical applicability. |
| Approach: | They propose an explainable smishing detection framework that adapts to a Korean-centric large language model for sys-phishing detection. |
| Outcome: | The proposed framework achieves 15% improvement in accuracy over existing models and produces high-quality explanatory text. |
PsyDT: Using LLMs to Construct the Digital Twin of Psychological Counselor with Personalized Counseling Style for Psychological Counseling (2025.acl-long)
Copied to clipboard
| Challenge: | Existing mental health LLMs do not consider the fact that different psychological counselors exhibit different personal styles. |
| Approach: | They propose a framework that uses LLMs to construct the digital twin of psychological counselor with personalized counseling style. |
| Outcome: | The proposed framework can synthesize multi-turn dialogues that closely resemble real-world counseling cases and demonstrate better performance compared to baselines. |
BIPro: Zero-shot Chinese Poem Generation via Block Inverse Prompting Constrained Generation Framework (2025.acl-long)
Copied to clipboard
| Challenge: | generative pre-trained models face challenges on constrained writing tasks like poem generation . brian mccartney: BIPro improves the zero-shot generation quality on constricted writing tasks . |
| Approach: | They propose a framework that leverages two block inverse prompting methods to improve the quality of constrained writing tasks. |
| Outcome: | BIPro significantly improves the quality of Chinese poem generation without priming or training. |
Evaluating Morphological Compositional Generalization in Large Language Models (2025.naacl-long)
Copied to clipboard
Mete Ismayilzada, Defne Circi, Jonne Sälevä, Hale Sirin, Abdullatif Köksal, Bhuwan Dhingra, Antoine Bosselut, Duygu Ataman, Lonneke Van Der Plas
| Challenge: | Large language models (LLMs) have demonstrated significant progress in various natural language generation and understanding tasks. |
| Approach: | They define morphemes as compositional primitives and design a suite of generative and discriminative tasks to assess morphological productivity and systematicity. |
| Outcome: | The proposed models can identify individual morphological combinations better than chance, but their performance lacks systematicity, leading to significant accuracy gaps compared to humans. |
LangNav: Language as a Perceptual Representation for Navigation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to vision-and-language navigation use visual features as the perceptual representation of a visual representation of an agent's egocentric panoramic view. |
| Approach: | They propose to use off-the-shelf vision systems to convert an agent’s egocentric panoramic view into natural language descriptions. |
| Outcome: | The proposed approach improves on the R2R VLN benchmark by using synthetic trajectories from a prompted language model and domain transfer where a policy learned on one simulated environment (ALFRED) is transferred to another (more realistic) environment and combining both vision- and language-based representations. |
Do Language Models Know When They’re Hallucinating References? (2024.findings-eacl)
Copied to clipboard
| Challenge: | State-of-the-art language models (LMs) are notoriously susceptible to generating hallucinated information. |
| Approach: | They propose to use hallucinated book and article references as "model organism" of hallucinism research . authors propose queries to the language model to identify hallucinous references . |
| Outcome: | The authors show that language models can identify hallucinated references without external resources . they show that LMs often produce inconsistent author lists for hallucinos, but also accurately recall the authors of real references . |
Little Giants: Synthesizing High-Quality Embedding Data at Scale (2025.naacl-long)
Copied to clipboard
| Challenge: | Synthetic data generation is an increasingly popular way of training models without the need for large, manually labeled datasets. |
| Approach: | They propose a framework that aligns open-source small models to efficiently generate large-scale embedding data. |
| Outcome: | The proposed framework outperforms state-of-the-art embedding models by using only 1/10 of the GPT API calls. |
SusGen-GPT: A Data-Centric LLM for Financial NLP and Sustainability Report Generation (2025.findings-naacl)
Copied to clipboard
Qilong Wu, Xiaoneng Xiang, Huang Hejia, Xuan Wang, Yeo Wei Jie, Ranjan Satapathy, Ricardo Shirota Filho, Bharadwaj Veeravalli
| Challenge: | Existing tools for financial reporting and ESG analysis are lacking . large language models are not proficient across general finance and ESE domains . |
| Approach: | They propose a dataset that includes seven financial NLP tasks and a benchmark to improve sustainability report generation. |
| Outcome: | SusGen-30k, a high-quality dataset, shows state-of-the-art performance . it surpasses all other models except GPT-4 in six adapted tasks and two off-the shelf tasks . |
E5: Zero-shot Hierarchical Table Analysis using Augmented LLMs via Explain, Extract, Execute, Exhibit and Extrapolate (2024.naacl-long)
Copied to clipboard
| Challenge: | Hierarchical tables with multi-level headers are confusing for models due to their complex structure, implicit semantics, and calculation relationships. |
| Approach: | They propose a code-augmented framework for zero-shot hierarchical table question answering that combines self-explaining the table's hierarchic structures with external code execution to prevent hallucinations. |
| Outcome: | The proposed framework outperforms state-of-the-art fine-tuning methods with a 44.38 Exact Match improvement. |
Teaching-Inspired Integrated Prompting Framework: A Novel Approach for Enhancing Reasoning in Large Language Models (2025.coling-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit impressive performance across various domains but struggle with arithmetic reasoning tasks. |
| Approach: | They propose a Teaching-Inspired Integrated Prompting Framework which emulates the instructional process of a teacher guiding students. |
| Outcome: | The proposed framework improves reasoning accuracy on nine benchmarks. |
Generation, Distillation and Evaluation of Motivational Interviewing-Style Reflections with a Foundational Language Model (2024.eacl-long)
Copied to clipboard
| Challenge: | Motivational Interviewing (MI) is a counselling technique used to guide people towards behaviour change. |
| Approach: | They propose a method for distilling reflections from a foundational language model into smaller models that can be owned and controlled. |
| Outcome: | The proposed method achieves 100% success rate on hold-out test set and 90% on the GPT-2 XL. |
IndiFoodVQA: Advancing Visual Question Answering and Reasoning with a Knowledge-Infused Synthetic Data Generation Pipeline (2024.findings-eacl)
Copied to clipboard
| Challenge: | Large Vision Language Models lack domain-specific data for reasoning on complex problems. |
| Approach: | They propose to use explicit knowledge-infused questions, answers, and reasons to answer and reason upon the questions. |
| Outcome: | The proposed model improves by 25% over the baseline model. |
Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency (2024.acl-long)
Copied to clipboard
| Challenge: | Existing work utilizes verification properties to verify and re-rank solutions in a majority voting manner, but this assumption may not hold. |
| Approach: | They propose a multi-perspective self-consistency framework that incorporates both inter- and intra-consistency across outputs from multiple perspectives. |
| Outcome: | The proposed framework significantly boosts performance of foundation models on various benchmarks, including HumanEval (+15.91%), MBPP (+6.43%) and CodeContests (+9.37%). |
Training Language Models to Critique With Multi-agent Feedback (2025.findings-emnlp)
Copied to clipboard
Tian Lan, Wenwei Zhang, Chengqi Lyu, Shuaibin Li, Chen Xu, Heyan Huang, Dahua Lin, Xian-Ling Mao, Kai Chen
| Challenge: | utilizing human annotations can enhance critique ability, but model-generated critiques suffer from inherent flaws due to complexity of critique . a new framework that leverages multi-agent feedback improves critique ability . |
| Approach: | They propose a framework that leverages multi-agent feedback to improve critique ability . they propose to use supervised fine-tuning and reinforcement learning to improve this capability . |
| Outcome: | The proposed framework improves critique ability in both supervised fine-tuning and reinforcement learning stages. |
Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are effective at answering clear questions but when faced with ambiguous queries they act unpredictably and produce incorrect outputs. |
| Approach: | They propose to use a surrogate problem to assess an LLMs’s ability to deduce an entity unknown to itself, but revealed to a judge, by asking the judge a series of queries. |
| Outcome: | The proposed model outperforms human players on the entity-deducing task by a large margin. |
Can ChatGPT Assess Human Personalities? A General Evaluation Framework (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies study the virtual personalities of LLMs but rarely explore the possibility of analyzing human personalities via LLM. |
| Approach: | They propose to use Myers–Briggs Type Indicator (MBTI) tests to generate unbiased prompts and replace the subject in question statements to enable flexible queries and assessments. |
| Outcome: | The proposed framework enables LLMs to flexibly assess personalities of different groups of people. |
ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing static benchmarks do not guarantee that models can use the provided evidence for answering, which is essential to avoid hallucination when the required knowledge is new or private. |
| Approach: | They propose to automatically perturb existing static one for dynamic evaluation by using a chatGPT framework and a set of open-domain QA datasets. |
| Outcome: | The proposed framework generates new test cases on two open-domain QA datasets and is human-readable and useful to trigger hallucination in LLMs. |
Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human? (2025.acl-short)
Copied to clipboard
| Challenge: | Existing automatic evaluation metrics are based on procedures that diverge from human evaluation. |
| Approach: | They propose to aggregate automatic evaluation metrics to bridge this gap . they propose to use edit-based metrics, -gram based metrics and sentence-level metrics to find the best ranking system. |
| Outcome: | The proposed method outperforms existing metrics on the SEEDA benchmark and improves edit-based metrics, -gram based metrics and sentence-level metrics. |
AHP-Powered LLM Reasoning for Multi-Criteria Evaluation of Open-Ended Responses (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Question answering (QA) tasks have been extensively studied in the field of natural language processing. |
| Approach: | They propose a method that leverages large language models and the analytic hierarchy process to assess open-ended questions. |
| Outcome: | The proposed method more closely aligns with human judgment compared to baselines on four datasets. |
Event Causality Identification with Synthetic Control (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to event causality identification have primarily utilized linguistic patterns and multi-hop relational inference, risking false causality . |
| Approach: | They propose to use the Rubin Causal Model to identify event causality by generating a twin from existing corpora. |
| Outcome: | The proposed method can identify causal relations more robustly than previous methods, including GPT-4, which is demonstrated on a causality benchmark, COPES-hard. |
Alleviating Hallucinations in Large Language Models via Truthfulness-driven Rank-adaptive LoRA (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to improve truthfulness are training-free without modifying the LLM itself. |
| Approach: | They propose a rank-adaptive LoRA method to improve LLM truthfulness that allocates ranks according to truthfulness correlations of LLM modules. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the LLM family and makes the performance of 7B LLMs exceed GPT-4. |
AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension (2024.acl-long)
Copied to clipboard
Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, Jingren Zhou
| Challenge: | Existing benchmarks for audio-centric interaction have impeded advancements in this field . AIR-Bench evaluates LALMs' ability to understand audio signals and interact with humans . |
| Approach: | They propose a benchmark to evaluate the ability of large audio-language models to understand audio signals . they use 19 tasks with approximately 19k single-choice questions to examine single-task ability . |
| Outcome: | The proposed framework evaluates the ability of large audio-language models to understand audio signals and interact with humans in the textual format. |
CausalGraph2LLM: Evaluating LLMs for Causal Queries (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have opened up new avenues for their use beyond standard Natural Language Processing tasks. |
| Approach: | They propose a benchmark to evaluate the capabilities of Large Language Models (LLMs) they use over 700k queries to compare their encoding capabilities. |
| Outcome: | The proposed benchmark compared LLMs on graph-level and node-level queries and open-sourced and closed models. |
Multi-dimensional Evaluation of Empathetic Dialogue Responses (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prior efforts to measure conversational empathy focus on expressed communicative intents, but ignore the fact that conversation is also a collaboration involving both speakers and listeners. |
| Approach: | They propose a multi-dimensional empathy evaluation framework to measure both expressed intents from the speaker’s perspective and perceived empathy from the listener’s viewpoint. |
| Outcome: | The proposed framework measures both expressed intents from the speaker’s perspective and perceived empathy from the listener’s viewpoint. |
A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for generating 'jailbreaks' suffer from manual design or require optimization on other white-box models, which compromises either generalization or efficiency. |
| Approach: | They propose a framework that leverages LLMs to generate effective jailbreak prompts and a generalized framework that can be used to generate prompts. |
| Outcome: | The proposed framework improves the attack success rate while reducing the time cost compared to baselines. |
A Paradigm Shift: The Future of Machine Translation Lies with Large Language Models (2024.lrec-main)
Copied to clipboard
Chenyang Lyu, Zefeng Du, Jitao Xu, Yitao Duan, Minghao Wu, Teresa Lynn, Alham Fikri Aji, Derek F. Wong, Longyue Wang
| Challenge: | Large Language Models (LLMs) are introducing a new phase in machine translation . despite advances in MT, there are still many challenges to overcome . |
| Approach: | They propose to highlight several new directions for MT that are influenced by Large Language Models like GPT-4 and ChatGPT. |
| Outcome: | The proposed models offer vast linguistic understandings and bring innovative methodologies, such as prompt-based techniques, that have the potential to further elevate MT. |
In-Contextual Gender Bias Suppression for Large Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Prior work has proposed debiasing methods that require human labelled examples, data augmentation and fine-tuning of LLMs, which are computationally expensive. |
| Approach: | They propose to suppress gender biases by providing textual preambles from manually designed templates and real-world statistics without accessing model parameters. |
| Outcome: | The proposed methods suppress gender biases in English LLMs using a CrowsPairs dataset without accessing model parameters. |
VariErr NLI: Separating Annotation Error from Human Label Variation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing work on label variation and annotation errors has focused on them in isolation. |
| Approach: | They propose a 2-round annotation procedure to separate human label variation from annotation errors by pairing valid explanations with annotators' validations. |
| Outcome: | The proposed procedure is based on the NLI task in English and contains 7,732 valid judgements on 1,933 explanations for 500 re-annotated items. |
LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models (2024.emnlp-main)
Copied to clipboard
Yuxuan Wan, Wenxuan Wang, Yiliu Yang, Youliang Yuan, Jen-tse Huang, Pinjia He, Wenxiang Jiao, Michael Lyu
| Challenge: | LogicAsker examines and improves the reasoning abilities of large language models such as ChatGPT and GPT-4. |
| Approach: | They propose a set of atomic reasoning skills grounded in propositional and predicate logic to examine and improve the reasoning abilities of large language models such as ChatGPT and GPT-4. |
| Outcome: | The proposed approach improves reasoning abilities in large language models such as ChatGPT and GPT-4 by up to 5%. |
Counter Turing Test (CT2): AI-Generated Text Detection is Not as Easy as You May Think - Introducing AI Detectability Index (ADI) (2023.emnlp-main)
Copied to clipboard
Megha Chakraborty, S.M Towhidul Islam Tonmoy, S M Mehedi Zaman, Shreya Gautam, Tanay Kumar, Krish Sharma, Niyar Barman, Chandan Gupta, Vinija Jain, Aman Chadha, Amit Sheth, Amitava Das
| Challenge: | a number of issues have arisen regarding the risk and consequences of AI-generated text detection. |
| Approach: | They propose a counter-turing test to evaluate the robustness of existing AGTD methods . they propose ADI, a quantifiable spectrum to assess detectability of LLMs . |
| Outcome: | The proposed method evaluates the robustness of existing AGTD methods . it shows that larger LLMs tend to have lower ADI, indicating they are less detectable . |
Enhancing Perception: Refining Explanations of News Claims with LLM Conversations (2024.findings-naacl)
Copied to clipboard
| Challenge: | a new framework for Large Language Models (LLMs) streamlines the task of crafting explanations for fake news . a study compared refinement conversations between human and LLMs to enhance the effectiveness of LLM explanations . |
| Approach: | They propose a framework for Large Language Models to streamline the task of crafting fake news explanations. |
| Outcome: | The proposed framework enhances the process of crafting explanations for fake news claims through conversational refinement. |
MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks (2024.naacl-long)
Copied to clipboard
Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram
| Challenge: | Several new LLMs have been introduced necessitating their evaluation on non-English languages. |
| Approach: | They perform a thorough evaluation of the non-English capabilities of SoTA LLMs by comparing them on the same set of multilingual datasets. |
| Outcome: | The proposed model outperforms models on multilingual datasets on 22 languages including low-resource African languages. |
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2024.findings-naacl)
Copied to clipboard
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, Nan Duan
| Challenge: | Traditional benchmarks for evaluating foundation models often fail to accurately represent their general abilities for human-centric tasks. |
| Approach: | They propose a bilingual benchmark to assess foundation models in the context of human-centric standardized exams such as college entrance exams, law school admission tests, and math competitions. |
| Outcome: | The proposed benchmark exceeds the average human performance on SAT, LSAT, and math competitions with 95% accuracy and 92.5% on the Chinese college entrance English exam. |
MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing models have demonstrated outstanding capabilities in mathematical reasoning, but there is a performance gap between open-source models and closed-source ones. |
| Approach: | They propose a method for generating diverse and reliable math problems by leveraging the ground-truth solutions of the seed data. |
| Outcome: | The proposed model outperforms open-source models across five representative mathematical reasoning datasets. |
A RelEntLess Benchmark for Modelling Graded Relations between Named Entities (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing Knowledge Graphs do not cover graded relations, yet they are difficult to draw a line between those that satisfy them and those that do not. |
| Approach: | They propose a benchmark in which entity pairs have to be ranked according to how much they satisfy a given graded relation. |
| Outcome: | The proposed model outperforms several publicly available LLMs and closed conversational models. |
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment (2023.emnlp-main)
Copied to clipboard
| Challenge: | Conventional reference-based metrics have low correlation with human judgments, especially for open-ended generation tasks. |
| Approach: | They propose to use large language models as reference-free NLG evaluators to assess the quality of NLG outputs. |
| Outcome: | The proposed framework outperforms all previous methods in two generation tasks, and has a Spearman correlation of 0.514 with human on summarization task, and a large variance in human judgments. |
WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new few-shot LLM-based chatbot is able to provide factual and engaging responses . a novel hybrid human-and-LLM evaluation methodology is used to evaluate the system . |
| Approach: | They propose a few-shot LLM-based chatbot that almost never hallucinates . they distill WikiChat into a 7B-parameter LLaMA model with minimal loss of quality . |
| Outcome: | The proposed system outperforms retrieval-based and LLM-based systems on the Wikipedia corpus. |
Superlatives in Context: Modeling the Implicit Semantics of Superlatives (2025.naacl-long)
Copied to clipboard
| Challenge: | a study of superlatives shows that the semantics of superlations in context can be challenging for contemporary models. |
| Approach: | They propose a unified account of superlative semantics which allows for a broad-coverage annotation schema. |
| Outcome: | The proposed schema allows for interpreting superlative expressions and their semantic interpretations. |
Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots (2025.findings-naacl)
Copied to clipboard
| Challenge: | Multi-modal Large Language Models have shown remarkable progress in visual contexts, yet their ability to convert visual figures into executable code remains underexplored. |
| Approach: | They propose to use a set of visual coding metrics to assess MLLMs' visual . pass rate, text-match ratio, and GPT-4V rating judgement to assess the quality of generated code and rendered images. |
| Outcome: | The proposed benchmark includes 132 high-quality matplotlib plots across six plot types, as well as 150 and 86 plots from Python’s and R’s plotly libraries respectively, totaling 368 plots. |
ChatGPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Trigger (2024.naacl-long)
Copied to clipboard
| Challenge: | Textual backdoor attacks are increasingly challenging to detect due to the use of advanced generative models such as GPT-4. |
| Approach: | They propose a framework that harnesses advanced generative models to execute stealthier backdoor attacks on text classifiers. |
| Outcome: | The proposed framework achieves state-of-the-art attack success rate of 97.35% over four sentiment classification tasks and four human cognition stealthiness tests. |
Black-Box Prompt Optimization: Aligning Large Language Models without Model Training (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models are often not well aligned with human intents, which requires additional training. |
| Approach: | They propose to use Black-Box Prompt Optimization (BPO) to perform alignments on large language models that are not well aligned with human intents. |
| Outcome: | The proposed model outperforms existing models and is model-agnostic. |
Enhancing the General Agent Capabilities of Low-Paramter LLMs through Tuning and Multi-Branch Reasoning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Open-source pre-trained Large Language Models exhibit strong language understanding and generation capabilities, making them highly successful in a variety of tasks. |
| Approach: | They propose a method to construct agent-specific data using GPT-4 and supervised fine-tuning . they find that supervised tunning can significantly reduce hallucination outputs and formatting errors in agent tasks . |
| Outcome: | The proposed method improves on five agent tasks of AgentBench. |
BIPED: Pedagogically Informed Tutoring System for ESL Education (2024.acl-long)
Copied to clipboard
| Challenge: | Existing Large Language Models (LLMs) are limited in scope and lack pedagogical depth. |
| Approach: | They construct a BIlingual PEDagogically-informed Tutoring Dataset of one-on-one, human-to-human tutoring interactions using a post-hoc analysis. |
| Outcome: | The proposed models replicate the style of human teachers and employ diverse and contextually appropriate pedagogical strategies. |
LogiCoT: Logical Chain-of-Thought Instruction Tuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on self-instruction tuning has focused on enhancing the general proficiency of models. |
| Approach: | They propose a new instruction-tuning dataset for Logical Chain-of-Thought reasoning with GPT-4 that harvests instructions for prompting GPT to generate chain-of thought rationales. |
| Outcome: | The proposed dataset enables the model to generate chain-of-thought rationales with GPT-4. |
Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are known to perform tasks by simply observing few exemplars, but performance among under-represented languages falls behind due to pre-training data imbalance. |
| Approach: | They propose to assemble synthetic exemplars from high-resource languages to prompt LLMs to translate from any language into English and use them to create intra-lingual exemplar models to perform tasks in target languages. |
| Outcome: | The proposed method outperforms supervised few-shot learning in LLMs of different sizes for translations between English and 13 Indic and 21 African low-resource languages. |
Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT (2024.lrec-main)
Copied to clipboard
Amirhossein Abaskohi, Sara Baruni, Mostafa Masoudi, Nesa Abbasi, Mohammad Hadi Babalou, Ali Edalat, Sepehr Kamahi, Samin Mahdizadeh Sani, Nikoo Naghavian, Danial Namazifard, Pouya Sadeghi, Yadollah Yaghoobzadeh
| Challenge: | a new study examines the efficacy of large language models (LLMs) for Persian . ChatGPT and LLMs have shown remarkable performance in English, but their efficiency for low-resource languages remains an open question. |
| Approach: | They present a benchmarking study of large language models (LLMs) for Persian . they focus on GPT-3.5-turbo, but also GPT-4 and OpenChat-3.5 . |
| Outcome: | The proposed model performs better in Persian than other low-resource languages . the study is the first comprehensive benchmarking of large language models . |
DARA: Decomposition-Alignment-Reasoning Autonomous Language Agent for Question Answering over Knowledge Graphs (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to answer questions over Knowledge Graphs (KGQA) are not available for KGQA. |
| Approach: | They propose a framework to improve the neural-symbolic reasoning capabilities of language agents powered by Large Language Models (LLMs) they show that DARA can be efficiently trained with a small number of high-quality reasoning trajectories. |
| Outcome: | The proposed framework outperforms in-context learning-based agents with GPT-4 and alternative fine-tuned agents across different benchmarks. |
Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) prompting has been shown to enhance the multi-step reasoning capabilities of Large Language Models (LLMs). |
| Approach: | They propose to use CoT prompting to analyze a symbolic reasoning task where letters are shifted forward some number of steps in the alphabet. |
| Outcome: | The proposed model performs well on a symbolic reasoning task, with three LLMs performing the task using CoT prompts. |
CONSCENDI: A Contrastive and Scenario-Guided Distillation Approach to Guardrail Models for Virtual Assistants (2024.naacl-long)
Copied to clipboard
| Challenge: | A major challenge in deploying LLM-based virtual conversational assistants in real world settings is ensuring they operate within what is admissible for the task. |
| Approach: | They propose to use large language models (LLMs) to generate training data with two key LLM components: scenario-augmented generation and contrastive training examples. |
| Outcome: | The proposed model improves over baselines in multiple dialogue domains. |
Controlling Out-of-Domain Gaps in LLMs for Genre Classification and Generated Text Detection (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have pushed the boundaries of natural language processing, but their consistency is often limited when applied to unfamiliar domains. |
| Approach: | They propose a method that controls which predictive indicators are used and which are excluded during classification. |
| Outcome: | The proposed method reduces the OOD gap by up to 20 percentage points in a few-shot setup. |
Mitigating Paraphrase Attacks on Machine-Text Detection via Paraphrase Inversion (2025.findings-acl)
Copied to clipboard
| Challenge: | Paraphrases applied to machine-generated texts can degrade performance of machine-text detectors. |
| Approach: | They propose an approach which frames the problem as translation from paraphrased text back to the original text. |
| Outcome: | The proposed approach yields an average improvement of +22% AUROC across seven detectors and three different domains. |
FeRG-LLM : Feature Engineering by Reason Generation Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | FeRG-LLM is a large language model that performs feature engineering at an 8billion-parameter scale. |
| Approach: | They propose a framework to perform feature engineering at an 8billion-parameter scale using conversational dialogues. |
| Outcome: | The proposed framework outperforms Llama 3.1 70B and Llma 3.2 on most datasets while using fewer resources and achieving reduced inference time. |
Diversity Helps Jailbreak Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for jailbreaking large language models rely on laborious human engineering and whitebox access to model internals. |
| Approach: | They propose a method that instructs large language models to deviate from prior context and generate harmful outputs by instructing them to deviat from previous attacks. |
| Outcome: | The proposed method achieves a 62.83% higher success rate in compromising ten leading chatbots, while using only 12.9% of the queries. |
SMART: Self-Aware Agent for Tool Overuse Mitigation (2025.findings-acl)
Copied to clipboard
Cheng Qian, Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-Tür, Gokhan Tur, Heng Ji
| Challenge: | Current Large Language Models (LLMs) lack self-awareness to balance reasoning and tool use, increasing computational overhead. |
| Approach: | They propose a paradigm that enhances an agent’s self-awareness to optimize task handling and reduce tool overuse. |
| Outcome: | The proposed model reduces tool use by 24% while improving performance by over 37%. |
STOP! Benchmarking Large Language Models with Sensitivity Testing on Offensive Progressions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that assess explicit and implicit biases are based on a single scenario . a dataset of 450 offensive progressions contains 2,700 sentences of varying severity . |
| Approach: | They evaluate a dataset of offensive progressions that contain 2,700 sentences . they find that even the best-performing models detect bias inconsistently . |
| Outcome: | The proposed dataset shows that even the best-performing models detect bias inconsistently . aligning models with human judgments on STOP can improve answer rates on sensitive tasks by 191% . |
Hidden Persuaders: LLMs’ Political Leaning and Their Influence on Voters (2024.emnlp-main)
Copied to clipboard
| Challenge: | This paper examines the political leanings of large language models (LLMs) in the 2024 election. |
| Approach: | They propose to use large language models to examine users' political leanings in the 2024 presidential election to determine their political preference. |
| Outcome: | The proposed models show that they have a political leaning and can influence political views in the 2024 presidential election. |
Can GPT-4 Identify Propaganda? Annotation and Detection of Propaganda Spans in News Articles (2024.lrec-main)
Copied to clipboard
| Challenge: | Using large language models (LLMs) to detect propaganda from text is a challenge for the development of sophisticated models. |
| Approach: | They propose to use a large propaganda dataset to identify propagandistic content in text, visual, or multimodal languages to improve their models. |
| Outcome: | The proposed model performs better on a large propaganda dataset than the existing models on skewed datasets. |
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024.emnlp-main)
Copied to clipboard
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo
| Challenge: | Existing open-source evaluation paradigms lack flexibility and performance . language model-based evaluation is cheap and scalable, but it is difficult to evaluate . |
| Approach: | They propose a language model-based evaluation paradigm that uses a scalar indicator of quality to assess LM outputs. |
| Outcome: | The proposed language model-based evaluation model is more powerful than its predecessor. |
TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization (2024.naacl-long)
Copied to clipboard
Liyan Tang, Igor Shalyminov, Amy Wong, Jon Burnsky, Jake Vincent, Yu’an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, Kathleen McKeown
| Challenge: | Existing LLMs hallucinate significant amounts of factual errors in the dialogue domain, regardless of the model’s size. |
| Approach: | They propose to evaluate topic-focused dialogue summarization by using large language models (LLMs) they use human annotations to evaluate factual consistency and explain factually inconsistent sentences. |
| Outcome: | The proposed evaluation benchmark on topic-focused dialogue summarization shows that existing LLMs hallucinate significant amounts of factual errors regardless of the model’s size. |
ARES: Alternating Reinforcement Learning and Supervised Fine-Tuning for Enhanced Multi-Modal Chain-of-Thought Reasoning Through Diverse AI Feedback (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Multimodal Models excel at comprehending human instructions and demonstrate remarkable results across a broad spectrum of tasks. |
| Approach: | They propose an algorithm that alters REinforcement Learning and Supervised Fine-Tuning to refine large multimodal models with specific preferences. |
| Outcome: | The proposed algorithm achieves 70% win rate compared to baseline models judged by GPT-4o. |
Language Models can be Deductive Solvers (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances have enabled Large Language Models (LLMs) to potentially exhibit reasoning capabilities, but complex logical reasoning remains a challenge. |
| Approach: | They propose a novel language model that internalizes and emulates the reasoning processes of logical solvers and avoids parsing errors by learning strict adherence to solver syntax and grammar. |
| Outcome: | The proposed model outperforms state-of-the-art solver-augmented language models and few-shot prompting methods on public deductive reasoning benchmarks. |
Lost in the Distance: Large Language Models Struggle to Capture Long-Distance Relational Knowledge (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent large language models have demonstrated impressive capabilities in handling long contexts . however, as context length increases, LLMs struggle more with filtering out irrelevant information . |
| Approach: | They propose to use unrelated sentences to capture relational knowledge over long contexts . they find that LLMs can handle edge noise with little impact, but can reason about distant relationships . |
| Outcome: | The proposed model can handle edge noise with little impact, but its ability to reason about distant relationships declines as the noise grows. |
InstructExcel: A Benchmark for Natural Language Instruction in Excel (2023.findings-emnlp)
Copied to clipboard
Justin Payan, Swaroop Mishra, Mukul Singh, Carina Negreanu, Christian Poelitz, Chitta Baral, Subhro Roy, Rasika Chakravarthy, Benjamin Van Durme, Elnaz Nouri
| Challenge: | Large Language Models (LLMs) can solve increasingly complex NLP tasks such as Excel specific tasks. |
| Approach: | They propose a large-scale benchmark to test whether Large Language Models can generate code that solves Excel specific tasks provided via natural language user instructions. |
| Outcome: | The proposed model outperforms existing models and provides a hard benchmark for state of the art models like GPT-4. |
Using Language Models to Disambiguate Lexical Choices in Translation (2024.emnlp-main)
Copied to clipboard
| Challenge: | In translation, a concept represented by a single word can have multiple variations in a target language. |
| Approach: | They evaluate language models that can be used to generate English rules for lexical selection . they find weaker models with high-quality lexicals improve accuracy . |
| Outcome: | The proposed model outperforms existing models on the lexical selection task in English and with native speakers. |
NeuroComparatives: Neuro-Symbolic Distillation of Comparative Knowledge (2024.findings-naacl)
Copied to clipboard
| Challenge: | Comparative knowledge is an essential component of our world knowledge, yet understudied in prior literature. |
| Approach: | They propose a framework for comparative knowledge distillation overgenerated from language models . they use a corpus of 8.8M comparisons over 1.74M entity pairs to acquire comparative information . |
| Outcome: | The proposed framework acquires comparative knowledge between everyday objects . human evaluations show that it outperforms existing resources in terms of validity . |
Language Models can Evaluate Themselves via Probability Discrepancy (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation frameworks focus on superficial text differences and fail to align with human judgment. |
| Approach: | They propose a new method to evaluate the performance of Large Language Models (LLMs) by calculating probability discrepancies between original response generation and revised versions of LLMs. |
| Outcome: | The proposed method eliminates the need for training an additional evaluation model or relying on external proprietary models such as GPT-4 as a judger. |
X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agents (2025.acl-long)
Copied to clipboard
| Challenge: | Traditional Turing test limits each participant to one message at a time and requires constant human participation. |
| Approach: | They propose to enhance the original Turing test with a burst dialogue pattern, allowing more dynamic exchanges using consecutive messages. |
| Outcome: | The proposed test improves the original test with a burst dialogue pattern, allowing more dynamic exchanges using consecutive messages. |
MLeVLM: Improve Multi-level Progressive Capabilities based on Multimodal Large Language Model for Medical Visual Question Answering (2024.findings-acl)
Copied to clipboard
Dexuan Xu, Yanyuan Chen, Jieyi Wang, Yue Huang, Hanpin Wang, Zhi Jin, Hongxing Wang, Weihua Yue, Jing He, Hang Li, Yu Huang
| Challenge: | Existing MVQA models ignore multi-level progressive capabilities due to unspecific data and plain architecture. |
| Approach: | They propose a multi-level visual language model for medical visual question answering (MVQA) which covers multi- level questions and answers as well as reasoning processes from visual clues to semantic cognition. |
| Outcome: | The proposed model outperforms existing medical multimodal large language models on a multi-level instruction dataset and a feature alignment module. |
Reformulating Domain Adaptation of Large Language Models as Adapt-Retrieve-Revise: A Case Study on Chinese Legal Domain (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent large language models like GPT-4 have demonstrated astonishing zero-shot capabilities in general domain tasks, but they often generate content with hallucinations in specific domains such as Chinese law. |
| Approach: | They propose a framework for adapting large language models (LLMs) to Chinese legal domains by reformulating generation as an adapt-retrieve-revise process. |
| Outcome: | The proposed framework outperforms existing models in the Chinese legal domain by +33.6 points in the zero-shot setting. |
GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering (2025.coling-main)
Copied to clipboard
| Challenge: | Existing automated RAG evaluation frameworks overlook important failure modes when using GPT-4 as a judge. |
| Approach: | They propose a novel pipeline to assess the calibration and discrimination capabilities of judge models by using a meta-evaluation benchmark of 144 unit tests to identify key failure modes. |
| Outcome: | The proposed pipeline improves on existing frameworks, while state-of-the-art open-source judges do not generalize to their proposed criteria. |
Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes (2024.findings-acl)
Copied to clipboard
Sunjun Kweon, Junu Kim, Jiyoun Kim, Sujeong Im, Eunbyeol Cho, Seongsu Bae, Jungwoo Oh, Gyubok Lee, Jong Hak Moon, Seng Chan You, Seungjin Baek, Chang Hoon Han, Yoon Bin Jung, Yohan Jo, Edward Choi
| Challenge: | Clinical notes are an extensive repository of information specific to individual patients. |
| Approach: | They create synthetic large-scale clinical notes using publicly available case reports extracted from biomedical literature and train a clinical large language model, Asclepius. |
| Outcome: | The proposed model outperforms several other models and is supported by detailed evaluations conducted by GPT-4 and medical professionals. |
An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4 (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have fine-tuned judge models based on open-source LLMs to evaluate the quality of other LLM. |
| Approach: | They propose to use open-source LLMs to evaluate Large Language Models (LLMs) their empirical results show that the models underperform GPT-4 in several dimensions . |
| Outcome: | The proposed models outperform GPT-4 on several dimensions including generalizability, fairness and adaptability. |
LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers (2023.emnlp-main)
Copied to clipboard
Theo Olausson, Alex Gu, Ben Lipkin, Cedegao Zhang, Armando Solar-Lezama, Joshua Tenenbaum, Roger Levy
| Challenge: | Logical reasoning is an important task for artificial intelligence, says a new study . many prompting-based strategies to enable large language models fail in subtle and unpredictable ways. |
| Approach: | They propose to reformulate logical reasoning tasks by leveraging large language models . they use a modular neurosymbolic programming approach to translate premises and conclusions from natural language to logic . |
| Outcome: | The proposed approach outperforms open-source models on FOLIO and ProofWriter while showing distinct failure modes. |
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023.emnlp-main)
Copied to clipboard
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Lee
| Challenge: | Large language models (LLMs) have shown unprecedented performance across various tasks. |
| Approach: | They propose an easy-to-use framework that integrates adapters into LLMs . they evaluate adapters on 14 datasets from two different reasoning tasks . |
| Outcome: | The proposed framework can be used to fine-tune open-access language models with task-specific data and instruction data. |
CONAN-MT-SP: A Spanish Corpus for Counternarrative Using GPT Models (2024.lrec-main)
Copied to clipboard
María Estrella Vallecillo Rodríguez, Maria Victoria Cantero Romero, Isabel Cabrera De Castro, Arturo Montejo Ráez, María Teresa Martín Valdivia
| Challenge: | a new study evaluates the performance of GPT-based models to generate CNs for hate speech in Spanish . a growing number of social interactions through digital platforms have led to inappropriate behavior . |
| Approach: | They propose to use GPT-based models to generate CNs for Hate Speech in Spanish . they use the DeepL API to automatically translate the HS segment into Spanish based on the original CN pairs translated into spanish . |
| Outcome: | The proposed models outperform human models in most instances, the authors say . the results will be made available to the research community . |
JRE-L: Journalist, Reader, and Editor LLMs in the Loop for Science Journalism for the General Audience (2025.naacl-long)
Copied to clipboard
| Challenge: | Science journalism reports current scientific discoveries to non-specialists, aiming to enable public comprehension of the state of the art. |
| Approach: | They propose a framework that integrates three LLMs mimicking the writing-reading-feedback-revision loop. |
| Outcome: | The proposed framework generates articles that are more accessible than existing methods, including prompting single advanced models such as GPT-4 and other LLM-collaboration strategies. |
Benchmarking Large Language Models on CFLUE - A Chinese Financial Language Understanding Evaluation Dataset (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have revolutionized natural language processing (NLP) there is an urgent need for new benchmarks to keep pace with the development of LLMs. |
| Approach: | They propose a benchmark to assess the capability of large language models (LLMs) they use a dataset to provide both knowledge assessment and application assessment . |
| Outcome: | The proposed benchmark provides datasets tailored for knowledge assessment and application assessment. |
Fine-Grained Detection of Solidarity for Women and Migrants in 155 Years of German Parliamentary Debates (2024.emnlp-main)
Copied to clipboard
| Challenge: | Traditionally, solidarity relied on common identity and reciprocity, potentially excluding out-groups like migrants. |
| Approach: | They examine the frequency of (anti-)solidarity towards women and migrants in German parliamentary debates between 1867 and 2022. |
| Outcome: | The proposed model outperforms other models in the analysis of 2,864 text snippets and finds that solidarity with migrants outweighs anti-solidarity but frequencies and solidarity types shift over time. |
Phased Instruction Fine-Tuning for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to enhance pre-trained language models' ability to follow instructions are limited due to the simultaneous handling of varying instruction complexities. |
| Approach: | They propose a phased instruction fine-tuning method that posits that the transition of a pre-trained language model from simple next-word prediction to sophisticated instruction following is a gradual learning process. |
| Outcome: | The proposed method surpasses the one-off instruction fine-tuning method in win rate and validates the hypothesis of progressive alignment. |
TOREE: Evaluating Topic Relevance of Student Essays for Chinese Primary and Middle School Education (2024.findings-acl)
Copied to clipboard
Xinlin Zhuang, Hongyi Wu, Xinshu Shen, Peimin Yu, Gaowei Yi, Xinhao Chen, Tu Hu, Yang Chen, Yupei Ren, Yadong Zhang, Youqi Song, Binxuan Liu, Man Lan
| Challenge: | Existing research on Automatic Essay Scoring (AES) for Chinese essays has overlooked topic relevance and lacks detailed feedback. |
| Approach: | They propose to use TOREE to assess topic relevance in Chinese primary and middle school students’ essays to improve automatic and human evaluations. |
| Outcome: | The proposed method significantly improves both automatic and human evaluations across four diverse LLMs. |
C-STS: Conditional Semantic Textual Similarity (2023.emnlp-main)
Copied to clipboard
Ameet Deshpande, Carlos Jimenez, Howard Chen, Vishvak Murahari, Victoria Graf, Tanmay Rajpurohit, Ashwin Kalyan, Danqi Chen, Karthik Narasimhan
| Challenge: | Semantic textual similarity (STS) is a cornerstone task in natural language processing, but it is inherently ambiguous. |
| Approach: | They propose a task called conditional STS which measures similarity conditioned on an aspect elucidated in natural language. |
| Outcome: | The proposed task reduces subjectivity and ambiguity and enables fine-grained similarity evaluation using diverse conditions. |
SongComposer: A Large Language Model for Lyric and Melody Generation in Song Composition (2025.acl-long)
Copied to clipboard
Shuangrui Ding, Zihan Liu, Xiaoyi Dong, Pan Zhang, Rui Qian, Junhao Huang, Conghui He, Dahua Lin, Jiaqi Wang
| Challenge: | Creating lyrics and melodies in symbolic format requires expert knowledge of melody and an advanced understanding of lyrics. |
| Approach: | They introduce SongComposer, a music-specialized large language model that can create symbolic lyrics and melodies following instructions. |
| Outcome: | The proposed model outperforms existing models in symbolic song composition tasks. |
GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing algorithms that generate captions for scientific figures are costly and dependent on author-written captions. |
| Approach: | They constructed a human evaluation dataset that contains human judgments for 3,600 scientific figure captions for 600 arXiv figures. |
| Outcome: | The proposed model outperforms all other models and outperformed undergraduates in achieving a Kendall correlation score of 0.401 with Ph.D. students’ rankings. |
Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotations (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent MT metrics like xCOMET, Met-ricX, and Remedy have strong correlations with human preferences, but they are black boxes, revealing little insight into why a translation is good or bad. |
| Approach: | They propose a reasoning-driven generative MT metric trained with reinforcement learning from pairwise translation preferences without requiring error-span annotations or distillation from closed LLMs. |
| Outcome: | The proposed reasoning-driven generative MT metric produces step-by-step analyses of accuracy, fluency, and completeness, enabling more interpretable assessments. |
INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate the quality of language generation do not provide explicit explanation of their verdicts. |
| Approach: | They propose a fine-grained explainable evaluation metric for text generation that harnesses human instruction and implicit knowledge of GPT-4 to fine-tune it. |
| Outcome: | The proposed model outperforms all other unsupervised metrics on translation, captioning, data-to-text, and commonsense generation tasks. |
How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on instruction-tuned Large Language Models has used automatic methods based on text overlap and LLM judgments as cost-effective alternatives to human evaluation. |
| Approach: | They perform a meta-evaluation of automatic methods and assess their reliability across a broad range of tasks. |
| Outcome: | The proposed method is unreliable in free-form generation tasks and cross-lingual scenarios. |
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing web agents only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. |
| Approach: | They propose a large multimodal model-powered web agent that can complete user instructions end-to-end by interacting with real-world websites. |
| Outcome: | The proposed agent achieves 59.1% task success rate, surpassing both GPT-4 and WebVoyager setups. |
Instances Need More Care: Rewriting Prompts for Instances with LLMs in the Loop Yields Better Zero-Shot Performance (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized zero-shot task performance, mitigating the need for task-specific annotations while enhancing task generalizability. |
| Approach: | They propose an approach that optimizes the zero-shot prompts for individual task instances following an innovative manner of "LLMs in the loop" their results show that PRomPTed outperforms naive zero- shot approaches and a strong baseline which refines the task output instead of the input prompt. |
| Outcome: | The proposed approach outperforms naive approaches and a strong baseline which refines the task output instead of the input prompt. |
AutoBreach: Universal and Adaptive Jailbreaking with Efficient Wordplay-Guided Optimization via Multi-LLMs (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing jailbreak research exhibits limitations in universality, validity, and efficiency . Existing methods for jailbreaking LLMs have limited validity and effectiveness . |
| Approach: | They propose a black-box approach that uses wordplay-guided mapping rule sampling to create universal adversarial prompts. |
| Outcome: | The proposed method efficiently identifies security vulnerabilities across various LLMs, achieving an average success rate of over 80% with fewer than 10 queries. |
NAT: Enhancing Agent Tuning with Negative Samples (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning and reinforcement learning use only positive examples, limiting their efficiency in low-resource scenarios. |
| Approach: | They propose a method that leverages both successful and failed trajectories for fine-tuning, maximizing the utility of limited resources. |
| Outcome: | The proposed method surpasses existing methods, including SFT, DPO, and PPO, across various tasks. |
Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can help in citation generation but can also amplify existing biases, such as the Matthew effect, and introduce new ones, potentially skewing scientific knowledge dissemination. |
| Approach: | They propose to use large language models to generate scholarly references for in-text citations in papers published after GPT-4's knowledge cut-off date. |
| Outcome: | The proposed model can generate scholarly references for in-text citations, but without the aid of web browsing or a search engine, the results show a similarity between human and LLM citation patterns, but with a more pronounced high citation bias. |
TRAM: Benchmarking Temporal Reasoning for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | despite advances in natural language processing, temporal reasoning is still a challenge . despite advancements in NLP, current language models have yet to reach human level in this domain . |
| Approach: | They propose a temporal reasoning benchmark that measures time-related temporal aspects of events . they evaluate popular LLMs like GPT-4 and Llama2 in zero-shot and few-shot scenarios . |
| Outcome: | The proposed model outperforms human models in a few-shot and zero-shot scenarios . the best-performing model lags significantly behind human models, the authors say . |
Translation and Fusion Improves Cross-lingual Information Extraction (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown significant progress in information extraction tasks due to lack of labeled data for fine-tuning and unlabeled text for pre-training. |
| Approach: | They propose a framework in which large language models are fine-tuned to use English translations of low-resource language data. |
| Outcome: | The proposed model improves cross-lingual transfer over the base model on 12 multilingual IE datasets spanning 50 languages. |
Forget for Get: A Lightweight Two-phase Gradient Method for Knowledge Editing in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge editing methodologies often encounter parameter conflict during knowledge overwriting and excessive computational overhead. |
| Approach: | They propose a method that erases outdated knowledge and inserts new knowledge at the location that corresponds to the target knowledge. |
| Outcome: | The proposed method achieves more effective knowledge editing at a lower cost compared to previous methods across various base models. |
Leveraging pre-trained language models for linguistic analysis: A case of argument structure constructions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Argument structure constructions (ASCs) are lexicogrammatical patterns at the clausal level. |
| Approach: | They evaluate the effectiveness of pre-trained language models in identifying argument structure constructions . they use supervised training with RoBERTa and prompt-guided annotation with GPT-4 . |
| Outcome: | The proposed model outperforms the gold-standard model on three methods . the results show that the model performs better on gold-standardized data . |
Uncertainty Quantification for Clinical Outcome Predictions with (Large) Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Language models (LMs) have significant potential for clinical prediction tasks . however, unreliable decisions can result in significant costs due to compromised patient safety and ethical concerns . |
| Approach: | They propose to combine ensembling and multi-tasking approaches to reduce uncertainty in EHRs by using multi-tapping methods. |
| Outcome: | The proposed framework reduces model uncertainty in white-box and black-box settings, and improves model transparency in both settings. |
Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies decompose complex tasks into intermediate steps by prompting strategies, such as Chain-of-Thought and its variants. |
| Approach: | They propose to use code comments as natural logic pivot between natural language and code language to boost the code generation ability of code LLMs. |
| Outcome: | The proposed method significantly improves the code pass rate on humanEval and MBPP, while the robustness of the logical comment decoding strategy is higher than the Chain-of-thoughts prompting. |
SoFA: Shielded On-the-fly Alignment via Priority Rule Following (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing alignment methods fail to adapt to the diversity of preferences and regulatory standards. |
| Approach: | They propose a method for prioritizing rules over user instructions to minimize misalignments in Large Language Models. |
| Outcome: | The proposed approach minimizes misalignments and adapts smoothly to various unseen rules, ensuring they are shielded from hijacking and that the model responds appropriately. |
Evaluating the Knowledge Base Completion Potential of GPT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) have been proposed for unsupervised knowledge base completion (KBC) however, their ability to do this at scale and with high accuracy remains an open question. |
| Approach: | They propose to use language models to complete a large public KB, Wikidata, with 90% precision. |
| Outcome: | The proposed models can extend Wikidata by 27M facts at 90% precision. |
THREAD: Thinking Deeper with Recursive Spawning (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown impressive capabilities across diverse settings, but their performance degrades as context length and complexity increases. |
| Approach: | They propose to frame model generation as a thread of execution that, based on the context, can run to completion or dynamically spawn new threads. |
| Outcome: | The proposed model outperforms existing frameworks by 10% to 50% on diverse benchmarks. |
Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks or datasets require only a few steps of reasoning, making it difficult to analyse AI’s behaviour with reference to different problems within a specific topic in detail. |
| Approach: | They propose a conic10K math problem dataset that requires only a few steps of reasoning to be analysed. |
| Outcome: | The proposed dataset shows that existing language models exhibit weak performance on complex reasoning. |
The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis (2024.findings-acl)
Copied to clipboard
Miaoran Zhang, Vagrant Gautam, Mingyang Wang, Jesujoba Alabi, Xiaoyu Shen, Dietrich Klakow, Marius Mosbach
| Challenge: | In-context learning is a popular inference strategy where large language models solve a task using only a few labeled demonstrations without updating the model parameters. |
| Approach: | They conduct multidimensional analysis of multilingual in-context learning using 5 models from different model families and 9 datasets covering classification and generation tasks. |
| Outcome: | The results show that demonstrations vary significantly across models, tasks, and languages. |
ECON: On the Detection and Resolution of Evidence Conflicts (2024.emnlp-main)
Copied to clipboard
Cheng Jiayang, Chunkit Chan, Qianqian Zhuang, Lin Qiu, Tianhang Zhang, Tengxiao Liu, Yangqiu Song, Yue Zhang, Pengfei Liu, Zheng Zhang
| Challenge: | Recent studies have shown that AI generated content is more likely to dominate search results, making it difficult to detect when compared to human-produced content. |
| Approach: | They propose a method for generating diverse, validated evidence conflicts to simulate real-world misinformation scenarios. |
| Outcome: | The proposed method enables the detection of conflicting information in real-world scenarios and shows that weaker models struggle with similar answer conflicts while stronger models show robust performance. |
GraphEval36K: Benchmarking Coding and Reasoning Capabilities of Large Language Models on Graph Datasets (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated significant capabilities in processing and understanding text data. |
| Approach: | They propose a structure-based instruction-based method to enhance LLM performance on complex graph tasks. |
| Outcome: | The proposed framework outperforms open-source models on graph problem-solving, but the gap is narrowing. |
Branch-Solve-Merge Improves Large Language Model Evaluation and Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are often used for multi-faceted language generation and evaluation tasks that require complex user constraints or taking into account multiple aspects and criteria. |
| Approach: | They propose a Large Language Model program that consists of branch, solve, and merge modules that are parameterized with specific prompts to the base LLM. |
| Outcome: | The proposed program improves the evaluation correctness and consistency for each LLM by up to 26%, reducing length and pairwise position biases by up 50%, and allowing LLaMA-2-chat to match or outperform GPT-4 on most domains. |
Towards Benchmarking Situational Awareness of Large Language Models:Comprehensive Benchmark, Evaluation and Analysis (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Situational awareness is crucial for decision-making, anticipating potential issues, and adapting to dynamic circumstances. |
| Approach: | They propose a benchmark that covers three tiers of situational awareness capabilities . they conduct extensive experiments on advanced LLMs including GPT-4, LLaMA3, Qwen1.5 . |
| Outcome: | The proposed benchmark covers environment perception, situation comprehension and future projection. |
Close or Cloze? Assessing the Robustness of Large Language Models to Adversarial Perturbations via Word Recovery (2025.coling-main)
Copied to clipboard
| Challenge: | Existing models implicitly recover the original text, but it is unclear when they rely on context and when they implicitly do so. |
| Approach: | They propose to use a dictionary to recover adversarial words by using a phonetic, typo, and visual attack to study word recovery performance. |
| Outcome: | The proposed model outperforms open-source models on hateful, offensive, and toxic classification tasks. |
NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Queries (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) generate code for productive activities, but current benchmarks for code synthesis are oriented towards introductory tasks on algorithm and data science. |
| Approach: | They propose a code benchmark to mirror the complexity and variety of scenarios in real-world coding tasks. |
| Outcome: | The proposed benchmark improves on 39 large language models with close HumanEval scores and achieves an efficiency increase of more than 4 times. |
X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects (2024.naacl-long)
Copied to clipboard
| Challenge: | X-Eval is a two-stage instruction tuning framework to evaluate text in both seen and unseen aspects customized by end users. |
| Approach: | They introduce a two-stage instruction tuning framework to evaluate text in both seen and unseen aspects customized by end users. |
| Outcome: | The proposed framework improves the model’s ability to follow evaluation instructions and enhances the learning stage to better assess text quality. |
SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current scientific fact-checking benchmarks exhibit several shortcomings, such as biases arising from crowd-sourced claims and an over-reliance on text-based evidence. |
| Approach: | They present a dataset of 1.2K expert-verified scientific claims that require compositional reasoning for verification. |
| Outcome: | The proposed model outperforms existing models in table-based pretraining models and large language models. |
TheoremQA: A Theorem-driven Question Answering Dataset (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent LLMs like GPT-4 and PaLM-2 have made tremendous progress in solving fundamental math problems like GSM8K by achieving over 90% accuracy. |
| Approach: | They propose to use theorem-driven question-answering dataset to evaluate AI models' ability to apply theoretic concepts to solving challenging science problems. |
| Outcome: | TheoremQA is curated by domain experts and contains 800 high-quality questions covering 350 theoremics from Math, Physics, EE&CS, and Finance. |
Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework (2025.findings-acl)
Copied to clipboard
Kaishuai Xu, Tiezheng Yu, Yi Cheng, Wenjun Hou, Liangyou Li, Xin Jiang, Lifeng Shang, Qun Liu, Wenjie Li
| Challenge: | Existing methods for fine-tuning open-source LLMs are limited to text-based analysis under predefined general criteria. |
| Approach: | They propose a framework that fine-tunes LLMs to replicate the evaluation explanations and judgments of proprietary models. |
| Outcome: | The proposed evaluation framework outperforms existing fine-tuned evaluation methods in effectiveness and robustness. |
MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current methods for fact-checking are based on verifying each piece of a model against potential evidence using an LLM. |
| Approach: | They propose a method that builds small fact-checking models that have GPT-4-level performance but 400x lower cost. |
| Outcome: | The proposed model outperforms other models and reaches GPT-4 accuracy. |
Perspective Transition of Large Language Models for Solving Subjective Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized the field of natural language processing . performance of LLMs on subjective tasks is limited, authors say . |
| Approach: | They propose a method that allows LLMs to select between direct, role, and third-person perspectives for best way to solve corresponding subjective problem. |
| Outcome: | The proposed method outperforms widely used single fixed perspective based methods on 12 subjective tasks. |
Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study (2025.coling-main)
Copied to clipboard
Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng, Kai Chen
| Challenge: | Existing benchmarks focused on simplified or isolated aspects of coding, ignoring the full spectrum of programming challenges. |
| Approach: | They propose a case study that examines the performance of large language models across the entire software development lifecycle with four programming languages, multiple domains, and carefully designed and verified metrics for each task. |
| Outcome: | The proposed model performs across the entire software development lifecycle, including design, environment setup, implementation, acceptance testing, and unit testing. |
GATE X-E : A Challenge Set for Gender-Fair Translations from Weakly-Gendered Languages (2024.findings-acl)
Copied to clipboard
| Challenge: | despite improvements in NMT quality and adoption, perpetuation of gender bias remains a concern. |
| Approach: | They propose a translation gender rewriting solution built with GPT-4 and use it to evaluate it. |
| Outcome: | The proposed translation gender rewriting solution is open source and can be used in English. |
Assistant-Guided Mitigation of Teacher Preference Bias in LLM-as-a-Judge (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LLM-as-a-Judge uses large language models to evaluate the quality of LLM generated responses, but training proxy judge models using evaluation data generated by powerful teacher models introduces a critical yet previously overlooked issue: teacher preference bias. |
| Approach: | They propose a new setting that incorporates an additional assistant model, which is not biased toward the teacher model’s responses, to complement the training data. |
| Outcome: | The proposed model reduces teacher preference bias while maintaining strong performance across six evaluation benchmarks. |
Annotation alignment: Comparing LLM and human annotations of conversational safety (2024.emnlp-main)
Copied to clipboard
| Challenge: | We examine whether LLMs and humans agree when annotating the safety of user-chatbot conversations. |
| Approach: | They leverage a recent DICES dataset in which 350 conversations are each rated for safety by 112 annotators spanning 10 race-gender groups. |
| Outcome: | The LLMs annotators are compared to human annotator demographic groups and can predict when one group finds a conversation unsafe . |
MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation (2024.acl-long)
Copied to clipboard
| Challenge: | Embodied agents equipped with GPT as their brains have extraordinary decision-making and generalization abilities across various tasks. |
| Approach: | They propose a map-based agent that introduces an online linguistic-formed map to encourage global exploration. |
| Outcome: | The proposed agent achieves state-of-the-art zero-shot performance on R2R and REVERIE simultaneously. |
Large Language Models Can Learn Representation in Natural Language (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are unable to complete complex tasks involving multiple entities, such as tool APIs. |
| Approach: | They propose a method which uses natural language representations to refine entity descriptions for improved retrieval and LLM utilization. |
| Outcome: | The proposed method improves GPT-4's performance on classification tasks and API call tasks. |
MixLLM: Dynamic Routing in Mixed Large Language Models (2025.naacl-long)
Copied to clipboard
Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, Haifeng Chen
| Challenge: | Large Language Models (LLMs) exhibit potential artificial generic intelligence, however, their usage is costly with high response latency. |
| Approach: | They develop a dynamic contextual-bandit-based routing system for query-LLM assignment that leverages query tags to enhance query embeddings. |
| Outcome: | The proposed model maximizes response quality and minimizes cost and latency. |
Can Many-Shot In-Context Learning Help LLMs as Evaluators? A Preliminary Empirical Study (2025.coling-main)
Copied to clipboard
| Challenge: | Existing evaluation approaches to evaluate Large Language Models are affected by potential biases within LLMs. |
| Approach: | They propose two many-shot In-Context Learning (ICL) prompt templates to help LLM evaluators mitigate potential biases. |
| Outcome: | The proposed templates reduce biases by using in-context examples with model-generated rationales as references. |
Perceive the Passage of Time: A Systematic Evaluation of Large Language Model in Temporal Relativity (2025.coling-main)
Copied to clipboard
| Challenge: | Temporal perception is crucial for Large Language Models to understand the world. |
| Approach: | They propose a temporal-relative ability benchmark to evaluate LLMs' temporal perception . they conduct extensive experiments on popular LLM GPT-4 scenarios . |
| Outcome: | The proposed benchmarks show a significant performance gap between LLMs and humans in temporal-relative capability. |
An Empirical Study of Many-to-Many Summarization with Large Language Models (2025.acl-long)
Copied to clipboard
Jiaan Wang, Fandong Meng, Zengkui Sun, Yunlong Liang, Yuxuan Cao, Jiarong Xu, Haoxiang Shi, Jie Zhou
| Challenge: | Recent studies have shown that large language models (LLMs) have strong multilingual abilities, giving them the potential to perform M2MS in real applications. |
| Approach: | They propose to use many-to-many summarization (M2MS) to generate a brief summary in any language given a document also in any other language. |
| Outcome: | The proposed model outperforms zero-shot LLMs in terms of automatic evaluations. |
Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language We Prompt Them in (2024.lrec-main)
Copied to clipboard
| Challenge: | Ethical reasoning is a crucial skill for Large Language Models (LLMs). However, moral values are not universal, but rather influenced by language and culture. |
| Approach: | They extend the study of ethical reasoning of LLMs by (CITATION) to a multilingual setup using six languages: English, Spanish, Russian, Chinese, Hindi, and Swahili. |
| Outcome: | The proposed model is based on a multilingual setup in English, Spanish, Russian, Chinese, Hindi, and Swahili. |
Measuring What Matters: Evaluating Ensemble LLMs with Label Refinement in Inductive Coding (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are prone to inconsistencies and individual biases, limiting their reliability. |
| Approach: | They propose a framework that combines ensemble methods with code refinement methodology to address these challenges. |
| Outcome: | The proposed framework outperforms large language models and LLMs with a low-rank averaging and a moderator-based mechanism to simulate human consensus. |
Evaluating Code-Switching Translation with Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown they can match or surpass finetuned models on many natural language processing tasks. |
| Approach: | They propose to use in-context learning and pivot translation to improve code-switching translation. |
| Outcome: | The proposed models show strong ability for cross-lingual understanding in a code-switching setting. |
LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error (2024.acl-long)
Copied to clipboard
| Challenge: | Existing work on tool-augmented LLMs focuses on the broad coverage of tools and the flexibility of adding new tools. |
| Approach: | They propose a biologically inspired method for tool-augmented LLMs that orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory. |
| Outcome: | The proposed method improves tool learning for LLMs under both in-context learning and fine-tuning settings, bringing a boost of 46.7% to Mistral-Instruct-7B and outperforms GPT-4. |
Visual Hallucinations of Multi-modal Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies find VH instances only in existing image datasets, which results in biased understanding of MLLMs’ performance under VH. |
| Approach: | They propose a tool called VHTest to generate a diverse set of VH instances from existing image datasets and a text-to-image generative model to generate VH images based on the text descriptions. |
| Outcome: | The proposed tool finds VH instances in existing image datasets and generates images based on the text descriptions. |
Learning to Substitute Words with Model-based Score Ranking (2025.naacl-long)
Copied to clipboard
| Challenge: | Experimental results show that the proposed approach outperforms both masked language models and large language models. |
| Approach: | They propose a model-based scoring approach to quantify sentence quality . they propose 'loss function' that optimizes alignment between model predictions and sentence scores . |
| Outcome: | The proposed approach outperforms masked language models and large language models in the quantitative analysis of word substitutions. |
xLAM: A Family of Large Action Models to Empower AI Agent Systems (2025.naacl-long)
Copied to clipboard
Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Quoc Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao Feng, Tulika Manoj Awalgaonkar, Rithesh R N, Zeyuan Chen, Ran Xu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, Caiming Xiong
| Challenge: | Autonomous agents powered by large language models (LLMs) have attracted significant research interest, but there are few standards for developing specialized models for agent tasks. |
| Approach: | They propose a series of large action models with dense and mixture-of-expert architectures that unifies, augments, and synthesizes diverse datasets to enhance agent generalizability and performance. |
| Outcome: | The proposed models outperform GPT-4, Claude-3, and many other models in terms of tool use and outperformed GPT-based models on multiple agent ability benchmarks. |
RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) is a main technique for alleviating hallucinations in large language models. |
| Approach: | They propose to integrate RAG into large language models to analyze word-level hallucinations using a corpus of 18,000 naturally generated responses from diverse LLMs. |
| Outcome: | The proposed model can fine tune a relatively small LLM and achieve a competitive hallucination detection performance when compared to the existing prompt-based approaches. |
Large Language Models are biased to overestimate profoundness (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have been suggested to approach AI . however, it is still unclear whether LLMs possess similar reasoning abilities to humans . |
| Approach: | They evaluate GPT-4 and other LLMs in judging the profoundness of mundane statements . they find a significant correlation between the LLM and humans . |
| Outcome: | The proposed model overestimates the profoundness of nonsensical statements . the model overstates the profound nature of non-senior statements, the study finds . |
Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Use (2024.acl-long)
Copied to clipboard
| Challenge: | In this paper, we demonstrate that an inherent waveform pattern in the attention allocation of large language models significantly affects their performance in tasks demanding a high degree of context awareness. |
| Approach: | They propose a method that compensates an attention trough with an attention peak by a process to enhance the model's awareness to various contextual positions. |
| Outcome: | The proposed method improves the performance of a 7B model on the largest tool-use benchmark, comparable to that of GPT-4. |
Extracting Biomedical Entities from Noisy Audio Transcripts (2024.lrec-main)
Copied to clipboard
Nima Ebadi, Kellen Morgan, Adrian Tan, Billy Linares, Sheri Osborn, Emma Majors, Jeremy Davis, Anthony Rios
| Challenge: | Named Entity Recognition (NER) is particularly affected by noise, often termed the ASR-NLP gap. |
| Approach: | They propose a dataset to bridge the ASR-NLP gap in the biomedical domain by extracting adverse drug reactions and mentions of entities from the Brief Test of Adult Cognition by Telephone (BTACT) exam. |
| Outcome: | The proposed method can clean 2,000 clean and noisy recordings and eliminate errors using zero-shot and few-shot methods. |
Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generation (2025.findings-acl)
Copied to clipboard
Hanyin Wang, Chufan Gao, Bolun Liu, Qiping Xu, Guleid Hussein, Mohamad El Labban, Kingsley Iheasirim, Hariprasad Reddy Korsapati, Chuck Outcalt, Jimeng Sun
| Challenge: | Proprietary Large Language Models (LLMs) have demonstrated promising capabilities in clinical text summarization tasks. |
| Approach: | They propose a domain- and task-specific adaptation process for an open-source LLaMA-2 model . LLama-2 can generate high-quality clinical notes from outpatient patient-doctor dialogues . |
| Outcome: | The proposed model can generate clinical notes comparable to those authored by physicians. |
Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Abstention Ability (AA) is a critical aspect of Large Language Model (LLM) reliability. |
| Approach: | They propose a black-box evaluation approach and a new dataset, Abstain-QA, to rigorously assess AA across varied question types, domains, and task types. |
| Outcome: | The proposed evaluation process and new dataset, Abstain-QA, are crafted to rigorously assess AA across varied question types, domains, and task types. |
CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation (2024.acl-long)
Copied to clipboard
| Challenge: | CharacterEval is a benchmark for comprehensive RPCA assessment in Chinese . authors show that Chinese LLMs exhibit more promising capabilities than GPT-4 in role-playing conversation. |
| Approach: | They propose a Chinese benchmark for comprehensive RPCA assessment . they use a dataset of Chinese role-playing dialogues and character profiles . |
| Outcome: | The proposed benchmark demonstrates that Chinese LLMs exhibit more promising capabilities than GPT-4 in Chinese role-playing conversation. |
Finding Spoken Identifications: Using GPT-4 Annotation for an Efficient and Fast Dataset Creation Pipeline (2024.lrec-main)
Copied to clipboard
Maliha Jahan, Helin Wang, Thomas Thebaud, Yinglun Sun, Giang Ha Le, Zsuzsanna Fagyal, Odette Scharenborg, Mark Hasegawa-Johnson, Laureano Moro Velazquez, Najim Dehak
| Challenge: | Existing datasets that are limited to a few dialects, ethnicities, and age groups are not annotated considering these factors. |
| Approach: | They propose a semi-automated dataset creation pipeline that leverages large language models to perform two complex annotation tasks using human annotations as ground truths. |
| Outcome: | The proposed pipeline reduces time required for the filtering and tagging tasks while losing no important information. |
Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning (2024.acl-long)
Copied to clipboard
| Challenge: | Recent studies focus on enhancing large-scale language models' reasoning abilities, but the research question of how to GSM8K Performance vs. computational cost remains. |
| Approach: | They propose to train small-scale language models with their own outputs to avoid relying on large models' outputs. |
| Outcome: | The proposed approach outperforms baseline models with comparable sizes while minimizing the required compute. |
Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive capabilities when leveraging in-context learning. |
| Approach: | They propose a method that discretizes uninformative tokens using a self-supervised pre-training technique. |
| Outcome: | The proposed method achieves state-of-the-art performance across classification tasks while requiring only 0.8% decrease in performance. |
AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understanding (2024.findings-emnlp)
Copied to clipboard
Alessandro Suglia, Claudio Greco, Katie Baker, Jose Part, Ioannis Papaioannou, Arash Eshghi, Ioannis Konstas, Oliver Lemon
| Challenge: | Current Vision-Language Models (VLMs) focus on third-person view videos, neglecting the richness of egocentric perceptual experience. |
| Approach: | They propose to use the Egocentric Video Understanding Dataset (EVUD) to train VLMs on video captioning and question answering tasks specific to egocentric videos. |
| Outcome: | The proposed model outperforms open-source models including strong Socratic models using GPT-4 as a planner by 3.6% and outperformed Claude 3 and Gemini Pro Vision 1.0. |
Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for data-to-text generation are saturated, and there is no way to test them. |
| Approach: | They propose a tool for collecting structured data from public APIs to analyze the behavior of open large language models on the task of data-to-text generation. |
| Outcome: | The proposed model can generate fluent and coherent texts in zero-shot settings from data in common formats collected with Quintd. |
NEWTON: Are Large Language Models Capable of Physical Reasoning? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models have been shown to encapsulate syntactic, semantic, word sense, and common-sense knowledge, but limited exploration of their physical reasoning abilities has been conducted. |
| Approach: | They propose a repository and benchmark to evaluate LLMs' physical reasoning skills . they use a pipeline to generate a variant of the benchmark customized to the objects and attributes relevant for their application. |
| Outcome: | The proposed benchmark examines the reasoning capabilities of language models across reasoning tasks. |
Leveraging Web-Crawled Data for High-Quality Fine-Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, large language models are fine-tuned using expensive human-annotated data or GPT-4 generated data. |
| Approach: | They propose to use web-crawled data to train a language model on a smaller set of data . their results show that the model can convert web data with irregular formats into high-quality ones . |
| Outcome: | The proposed model outperforms open-source models larger than 32B and outperformed open-sourced models such as GPT-3.5. |
Using LLMs to simulate students’ responses to exam questions (2024.findings-emnlp)
Copied to clipboard
Luca Benedetto, Giovanni Aradelli, Antonia Donvito, Alberto Lucchetti, Andrea Cappelli, Paula Buttery
| Challenge: | Existing studies have used Large Language Models to simulate students answering exam questions . a proposed prompt for GPT-3.5 is not suitable for all LLMs, and there is no correlation between the quality of the rationales obtained with the model and the accuracy of the student simulation task. |
| Approach: | They propose a large language model prompt engineered for GPT-3.5 that can be used to answer exam questions simulating students of different skill levels. |
| Outcome: | The proposed prompt is robust to different educational domains and generalise to data unseen during prompt engineering phase. |
Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trends (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have improved summarization, but they still face a challenge of hallucination. |
| Approach: | They propose a taxonomy of errors to address the problem of hallucination in LLMs . they propose two prompt-based approaches for fine-grained error detection . |
| Outcome: | The proposed model outperforms existing metrics in identifying the novel "Contextual Inference" error type. |
CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models exhibit remarkable generative capabilities but can be misused for harmful purposes. |
| Approach: | They propose a framework that transforms natural language inputs into code inputs. |
| Outcome: | The proposed framework bypasses the safety guardrails of all models more than 80% of the time. |
AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator (2025.coling-main)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have demonstrated superior performance in static medical question answering benchmarks, rivaling even human experts. |
| Approach: | They propose a multi-agent framework emulating dynamic medical interactions between Doctor as player and NPCs including Patient and Examiner to assess the performance of LLM-driven Doctor agents in simulated clinical scenarios. |
| Outcome: | The proposed framework emulates dynamic medical interactions between Doctor as player and NPCs including Patient and Examiner. |
CSLM: A Framework for Question Answering Dataset Generation through Collaborative Small Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Collecting high-quality question-answer (QA) pairs is vital for training large language models, but computational demands and associated costs often render such approaches prohibitive for the average researcher. |
| Approach: | They propose a small-scaled, open-source solution that generates QA pairs from documents or raw corpora using large-scale models like Llama-70B. |
| Outcome: | Experiments on domain-specific datasets show that the proposed model can generate high-quality QA pairs, making it accessible to a broader range of researchers. |
Large Language Models Can Not Perform Well in Understanding and Manipulating Natural Language at Both Character and Word Levels? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) still exhibit significant deficiencies in basic language understanding and manipulation. |
| Approach: | They propose a bilingual benchmark to assess the performance of Large language models . they use a set of 15 simple text editing tasks to examine their capabilities . |
| Outcome: | The proposed benchmark aims to assess the performance of Large language models in basic language tasks. |
Axiomatic Preference Modeling for Longform Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have helped bridge the "alignment gap" between the responses of raw pretrained language models and responses that resonate more closely with human preferences. |
| Approach: | They propose to use a axiomatic framework to generate a rich variety of preference signals to uphold these signals. |
| Outcome: | The proposed model outperforms GPT-4 and ChatGPT in preference scoring. |
TRIGO: Benchmarking Formal Mathematical Proof Reduction for Generative Language Models (2023.emnlp-main)
Copied to clipboard
Jing Xiong, Jianhao Shen, Ye Yuan, Haiming Wang, Yichun Yin, Zhengying Liu, Lin Li, Zhijiang Guo, Qingxing Cao, Yinya Huang, Chuanyang Zheng, Xiaodan Liang, Ming Zhang, Qun Liu
| Challenge: | Automated theorem proving (ATP) benchmarks focus on symbolic inference but rarely involve understanding complex number combination reasoning. |
| Approach: | They propose a benchmark that requires a model to reduce a trigonometric expression with step-by-step proof and evaluates a generative LM’s reasoning ability on formulas and ability to manipulate, group, and factor number terms. |
| Outcome: | The proposed benchmark evaluates a generative LM’s reasoning ability on formulas and ability to manipulate, group, and factor number terms. |
MarathiEmoExplain: A Dataset for Sentiment, Emotion, and Explanation in Low-Resource Marathi (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Marathi is the third most widely spoken language in India with over 83 million native speakers . available Marath datasets are limited to coarse sentiment labels and lack fine-grained emotional categorization or interpretability through explanations. |
| Approach: | They propose to annotate Marathi sentences labeled with sentiment, emotion and a corresponding natural language justification. |
| Outcome: | The proposed dataset provides a benchmark for future research in multilingual and explainable NLP. |
TuringQ: Benchmarking AI Comprehension in Theory of Computation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | TuringQ is the first benchmark designed to evaluate the reasoning capabilities of large language models (LLMs) in the theory of computation. |
| Approach: | They propose a benchmark to evaluate the reasoning capabilities of large language models in the theory of computation. |
| Outcome: | The proposed system shows competitive accuracy when compared to human evaluation. |
Story Morals: Surfacing value-driven narrative schemas using large language models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using large language models, we extract and validate story morals across a diverse set of narrative genres. |
| Approach: | They propose a task of narrative schema labelling based on the concept of "story morals" they use large language models to extract and validate story morals across a diverse set of genres . |
| Outcome: | The proposed method extracts and validates story morals across folktales, novels, movies and TV, personal stories from social media and the news using automated metrics and human assessments. |
How Susceptible Are LLMs to Logical Fallacies? (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent studies have focused on LLMs' reasoning abilities, but their rational thinking capacity is not as robust as that of other NLP downstream tasks. |
| Approach: | They propose a diagnostic benchmark to assess the robustness of Large Language Models against logical fallacies by comparing their performance against a scenario where the persuader employs logical fallsacie. |
| Outcome: | The proposed benchmark compares the performance of LLMs in debates on controversial topics against logical fallacies. |
LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models (2024.acl-long)
Copied to clipboard
Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral
| Challenge: | Existing work investigating the logical reasoning ability of large language models has focused only on a couple of inference rules of propositional and first-order logics. |
| Approach: | They propose to use a natural language question-answering dataset to evaluate the logical reasoning ability of large language models. |
| Outcome: | The proposed model performs poorly on a range of natural language questions using chain-of-thought prompting. |
Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination (2024.emnlp-main)
Copied to clipboard
| Challenge: | a large-scale study of linguistic bias exhibited by ChatGPT covers 10 dialects of English . standard varieties of English, especially SAE, dominate available training data . |
| Approach: | They use ChatGPT to generate models that default to "standard" varieties of English . they also use a feature annotation and native speaker evaluation to analyze the responses . |
| Outcome: | The proposed models default to "standard" varieties of English, but non-"standard" ones exhibit stereotyping, demeaning content, lack of comprehension, condescending responses. |
Learning Together to Perform Better: Teaching Small-Scale LLMs to Collaborate via Preferential Rationale Tuning (2025.acl-long)
Copied to clipboard
| Challenge: | Prior studies have demonstrated that LLMs generate step-by-step rationales, but limited data is available to improve their performance in commercial settings due to copyright and legal issues. |
| Approach: | They propose a trainable framework that tunes a (small) LLM to generate outputs from a pool of diverse rationales that selectively improves the downstream task. |
| Outcome: | The proposed framework outperforms several trainable and prompting baselines on maths problem solving, natural language inference, and commonsense reasoning. |
What is the Best Way for ChatGPT to Translate Poetry? (2024.acl-long)
Copied to clipboard
| Challenge: | Despite promising results, our analysis reveals persistent issues in the translations generated by ChatGPT that warrant attention. |
| Approach: | They propose an Explanation-Assisted Poetry Machine Translation method which leverages monolingual poetry explanation as a guiding information for the translation process. |
| Outcome: | The proposed method outperforms traditional translation methods of ChatGPT and the existing online systems in English-Chinese poetry translation. |
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (2024.findings-acl)
Copied to clipboard
| Challenge: | OpenCodeInterpreter-33B provides a high level of performance for code generation, executing, and iterative refinement. |
| Approach: | They propose a family of open-source code systems for generating, executing, and iteratively refining code. |
| Outcome: | The OpenCodeInterpreter-33B performs well on humanEval, MBPP, and EvalPlus benchmarks. |
CodeDPO: Aligning Code Models with Self Generated and Verified Source Code (2025.acl-long)
Copied to clipboard
| Challenge: | Existing training methods for code generation do not improve code correctness and efficiency. |
| Approach: | They propose a framework that integrates preference learning into code generation to improve code correctness and efficiency. |
| Outcome: | The proposed framework improves code correctness and efficiency by integrating preference learning into code generation. |
InFoBench: Evaluating Instruction Following Ability in Large Language Models (2024.findings-acl)
Copied to clipboard
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, Dong Yu
| Challenge: | Existing methods for evaluating Large Language Models (LLMs) ability to follow instructions have not been able to provide a detailed analysis of their compliance with instructions. |
| Approach: | They propose a new metric for evaluating Large Language Models' ability to follow instructions and a benchmark for DRFR. |
| Outcome: | The proposed metric and benchmark compared with traditional scoring methods and explores annotation sources including human experts, crowd-sourced workers, and GPT-4. |
The Language of Trauma: Modeling Traumatic Event Descriptions Across Domains with Explainable AI (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Psychological trauma can manifest following various distressing events, but studies focus on a single aspect of trauma, often neglecting the transferability of findings across different scenarios. |
| Approach: | They propose a language model that fine-tunes a single aspect of trauma to better predict traumatic events across domains. |
| Outcome: | The proposed model outperforms large language models on trauma-related datasets . it also outperformed models on court data, counseling conversations, and forum posts . |
TinyStyler: Efficient Few-Shot Text Style Transfer with Authorship Embeddings (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for text style transfer rely on few-shot capabilities of large language models or complex controllable text generation approaches that are inefficient and underperform on fluency metrics. |
| Approach: | They propose a lightweight but effective approach which leverages a small language model and pre-trained authorship embeddings to perform efficient, few-shot text style transfer. |
| Outcome: | The proposed method outperforms strong approaches such as GPT-4 and performs form attribute style transfer with automatic and human evaluations. |
Can LLMs Understand the Implication of Emphasized Sentences in Dialogue? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Emphasis is a crucial component in human communication, which indicates speaker’s intention and implication beyond pure text in dialogue. |
| Approach: | They propose a benchmark dataset with annotated dialogue samples capturing the implications of emphasis. |
| Outcome: | The proposed evaluation pipeline achieves high correlation with human scoring and commercial LLMs perform better than open-source LLM. |
InstructEval: Instruction-Tuned Text Evaluator from Human Preference (2024.findings-acl)
Copied to clipboard
| Challenge: | InstructEval is a general text evaluator based on open-source Large Language Models (LLMs). |
| Approach: | They propose to build a general multi-aspect text evaluator based on open-source Large Language Models (LLMs) they use extensive open Human Preference Modeling datasets and a small set of multi-spect annotated data to overcome the shortage of annotation resources for multi-task evaluations. |
| Outcome: | The proposed model performs comparable or superior to commercial LLMs like ChatGPT or GPT-4 in terms of both overall and multi-aspect evaluation tasks. |
InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignment (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) can solve graph reasoning and generation tasks with parameter updates without sacrificing performance. |
| Approach: | They propose a structured format verbalizer to unify all graph data into a universal code-like format, which can simply represent the graph without any external graph-specific encoders. |
| Outcome: | The proposed framework outperforms GPT-4 and LLaMA2 in graph reasoning and generation tasks by more than 13% and 38%, respectively. |
Large Language Models for Automated Open-domain Scientific Hypotheses Discovery (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing research on hypothetical induction is limited by the observation annotations in the dataset and the ground truth hypotheses are mostly commonsense knowledge. |
| Approach: | They propose a first dataset for social science academic hypotheses discovery using raw web corpus as observations and propose valid, useful scientific hypothese . they propose 'a multi-module framework' that includes feedback mechanisms to boost performance. |
| Outcome: | The proposed dataset generates valid, novel, and helpful scientific hypotheses, even new to humanity, using open-domain data and a web corpus as observations. |
Leveraging GPT-4 for Automatic Translation Post-Editing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural Machine Translation models still require translation post-editing to rectify errors and enhance quality under critical settings. |
| Approach: | They use GPT-4 to automatically post-edit NMT outputs across several language pairs . they show that GPT4 is adept at translation post- editing, producing meaningful edits . |
| Outcome: | The proposed translation post-editor improves on state-of-the-art language models on English-Chinese, English-German, Chinese-English and German-English language pairs. |
Small Agent Can Also Rock! Empowering Small Language Models as Hallucination Detector (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on hallucination detection rely heavily on closed-source LLMs such as GPT-4. |
| Approach: | They propose an LLM-based agent framework called HaluAgent that integrates LLMs, multi-functional toolbox and a memory mechanism for hallucination detection. |
| Outcome: | The proposed framework integrates the LLM, multi-functional toolbox, and can detect hallucinations on Chinese and English datasets. |
Can LLMs Replace Clinical Doctors? Exploring Bias in Disease Diagnosis by Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a new study examines the bias of disease prediction in large language models . the model biases are prevalent across gender, age range and disease judgment behaviors . |
| Approach: | They propose a prompt-based approach to alleviate the bias in disease prediction with LLMs. |
| Outcome: | The proposed model alleviates the observed bias in disease prediction with LLMs. |
SWAG: Storytelling With Action Guidance (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are used for one-shot creation, but they can produce inconsistent but not necessarily engaging content. |
| Approach: | They propose a novel approach to storytelling with large language models that reduces story writing to a search problem through a two-model feedback loop. |
| Outcome: | The proposed approach outperforms existing methods when evaluated by GPT-4 and through human evaluation. |
Optimizing Chinese Lexical Simplification Across Word Types: A Hybrid Approach (2024.emnlp-main)
Copied to clipboard
| Challenge: | Expensive large language models outperform small models in simplifying complex content words and Chinese idioms from the dictionary. |
| Approach: | They propose to use a retrieval-based interpretation augmentation strategy to refine small models to simplify complex content words and Chinese idioms. |
| Outcome: | The proposed framework outperforms large language models in Chinese Lexical Simplification (CLS) and improves on OOD models. |
Large Language Models for Propaganda Span Annotation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Using propagandistic techniques to manipulate online audiences is increasing in recent years. |
| Approach: | They investigate whether Large Language Models (LLMs) such as GPT-4 can extract propagandistic spans and the potential of employing them to collect more cost-effective annotations. |
| Outcome: | The proposed model provides labels that have higher agreement with expert annotators and lead to specialized models that achieve state-of-the-art over an unseen Arabic testing set. |
LoFTI: Localization and Factuality Transfer to Indian Locales (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models encode vast amounts of world knowledge acquired via training on large web-scale datasets crawled from the internet. |
| Approach: | They propose a benchmark that evaluates an LLM's contextual localization and factual text transfer capabilities by comparing it with a mixtral, Llama3.3-70B and GPT-4 model. |
| Outcome: | The proposed benchmark evaluates Mixtral, Llama3.3-70B, GPT-4 and two other Mixtral-based approaches. |
Probing the “Creativity” of Large Language Models: Can models produce divergent semantic association? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models possess remarkable capacity for processing language, but it remains unclear whether they can further generate creative content. |
| Approach: | They utilize the divergent association task (DAT) to examine the creative thinking of large language models through a cognitive perspective. |
| Outcome: | The proposed model outperforms the greedy search strategy while outperforming the average human level. |
Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Level (2024.emnlp-main)
Copied to clipboard
| Challenge: | General-purpose Large Language Models (LLMs) like GPT-4 have exhibited strong translation abilities. |
| Approach: | They propose to use a model-agnostic model to refine the performance of general-purpose large-language models for machine translation (MT) by utilizing Gemma-2B/7B as the backbone. |
| Outcome: | The proposed model-agnostic and cost-effective tool improves the performance of general-purpose large-language models for machine translation (MT) by integrating it with any general-use LLM. |
From Words to Wires: Generating Functioning Electronic Devices from Natural Language Descriptions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work shows that language models have a capacity for electronic circuit design from high-level textual descriptions, akin to code generation. |
| Approach: | They propose two benchmarks to assess a model's ability to design common microcontroller circuits and code in the Arduino ecosystem. |
| Outcome: | The proposed model knowledge of electrical components and the model's ability to generate full circuits and code in the Arduino ecosystem are evaluated. |
RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models (2024.findings-acl)
Copied to clipboard
Noah Wang, Z.y. Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang, Jie Fu, Junran Peng
| Challenge: | Large Language Models (LLMs) have paved the way for complex tasks such as role-playing. |
| Approach: | They propose a framework to benchmark, elicit, and enhance role-playing abilities in Large Language Models. |
| Outcome: | The proposed framework improves role-playing abilities with 168,093 samples. |
Text Fluoroscopy: Detecting LLM-Generated Text through Intrinsic Features (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized the field of natural language processing because of their excellent performance on various tasks. |
| Approach: | They propose a black-box method with better generalizability for detecting LLM-generated text by mining the intrinsic features of the text to be detected. |
| Outcome: | The proposed method achieves 7.36% and 2.84% improvement in detection performance compared to baselines in detecting texts from different domains generated by GPT-4 and Claude3 respectively. |
Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiency (2024.lrec-main)
Copied to clipboard
Toyin D. Aguda, Suchetha Siddagangappa, Elena Kochkina, Simerjot Kaur, Dongsheng Wang, Charese Smiley
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable performance in data annotation tasks on general domain datasets, but their effectiveness on domain specific datasets remains under-explored. |
| Approach: | They compare the annotations produced by three LLMs against expert annotators and crowdworkers. |
| Outcome: | The proposed models outperform expert crowdworkers and crowd-sourced annotators on domain specific datasets. |
Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluation of natural language generation tasks lack reliable data. |
| Approach: | They propose to use annotations from human and GPT-4 to construct a corpus for NLG evaluation. |
| Outcome: | The proposed corpus can perform flexible and interpretable evaluations without references and surpasses existing models. |
Exploring the Boundaries of GPT-4 in Radiology (2023.emnlp-main)
Copied to clipboard
Qianchu Liu, Stephanie Hyland, Shruthi Bannur, Kenza Bouzid, Daniel Castro, Maria Wetscherek, Robert Tinn, Harshita Sharma, Fernando Pérez-García, Anton Schwaighofer, Pranav Rajpurkar, Sameer Khanna, Hoifung Poon, Naoto Usuyama, Anja Thieme, Aditya Nori, Matthew Lungren, Ozan Oktay, Javier Alvarez-Valle
| Challenge: | Recent success of general-domain large language models has changed the natural language processing paradigm towards a unified foundation model across domains and applications. |
| Approach: | They evaluate the performance of GPT-4 on a variety of radiology tasks . they find it outperforms or matches current SOTA radiology models . |
| Outcome: | The proposed model outperforms or matches current SOTA radiology models on a range of tasks. |
RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Models (2025.acl-long)
Copied to clipboard
Hieu Tran, Zonghai Yao, Zhichao Yang, Junda Wang, Yifan Zhang, Shuo Han, null Feiyun Ouyang, Hong Yu
| Challenge: | Existing work aims to improve reasoning accuracy and factual integrity across large language models for knowledge-intensive tasks such as medical and commonsense reasoning. |
| Approach: | They propose a versatile extension to the mutual reasoning framework (rStar) that enhances reasoning accuracy and factual integrity across large language models. |
| Outcome: | The proposed extension to the mutual reasoning framework improves reasoning accuracy and factual integrity across large language models for complex, knowledge-intensive tasks. |
Toward Reliable Ad-hoc Scientific Information Extraction: A Case Study on Two Materials Dataset (2024.findings-acl)
Copied to clipboard
Satanu Ghosh, Neal Brodnik, Carolina Frey, Collin Holgate, Tresa Pollock, Samantha Daly, Samuel Carton
| Challenge: | Existing methods for ad-hoc schema-based information extraction are brittle and non-transferable, limiting their practicality for this type of one-off extraction task. |
| Approach: | They propose to use GPT-4 to perform ad-hoc schema-based information extraction from scientific literature. |
| Outcome: | The proposed model can replicate two existing material science datasets, one pertaining to multi-principal element alloys and one to silicate diffusion, and draw on their insights to suggest future research directions. |
LLM-Guided Semantic-Aware Clustering for Topic Modeling (2025.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that topic modeling is competitive compared to closed-source methods. |
| Approach: | They propose a semi-supervised topic modeling method that combines LLMs with clustering to improve topic generation and distribution. |
| Outcome: | The proposed method outperforms state-of-the-art methods that utilize GPT-4 on topic alignment and exhibits competitive performance compared to Neural Topic Models on topic quality. |
LONGAGENT: Achieving Question Answering for 128k-Token-Long Documents through Multi-Agent Collaboration (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been successful in understanding language and processing text, but their cost prohibits their practical applications. |
| Approach: | They propose a multi-agent collaboration method that breaks down lengthy documents into smaller, more manageable chunks and organizes the member agents to read their assigned chunks. |
| Outcome: | The proposed method achieves 16.42% and 1.63% accuracy gains over existing models on single-hop and multi-hop QA settings. |
LHMKE: A Large-scale Holistic Multi-subject Knowledge Evaluation Benchmark for Chinese Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing benchmarks for comprehensively evaluating Chinese Large Language Models are insufficient. |
| Approach: | They propose a Large-scale, Holistic, and Multi-subject Knowledge Evaluation benchmark to evaluate Chinese Large Language Models. |
| Outcome: | The proposed benchmark measures the knowledge acquisition capabilities of Chinese Large Language Models across 75 subjects from primary school to professional certification exams. |
Linear Cross-document Event Coreference Resolution with X-AMR (2024.lrec-main)
Copied to clipboard
Shafiuddin Rehan Ahmed, George Arthur Baker, Evi Judge, Michael Reagan, Kristin Wright-Bettner, Martha Palmer, James H. Martin
| Challenge: | Event Coreference Resolution (ECR) is expensive both for automated systems and manual annotations. |
| Approach: | They propose a graphical representation of events anchored around individual mentions using a cross-document version of Abstract Meaning Representation. |
| Outcome: | The proposed model is anchored around individual mentions using a cross-document version of Abstract Meaning Representation. |
Hire a Linguist!: Learning Endangered Languages in LLMs with In-Context Linguistic Descriptions (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing LLMs rarely perform well in unseen, endangered languages . Existing models such as Llama and GPT-4 lack a rich corpus of training data . |
| Approach: | They propose a training-free approach to enable an LLM to process unseen languages that hardly occur in its pre-training. |
| Outcome: | The proposed approach elevates translation capability from GPT-4’s 0 to 10.5 BLEU for 10 language directions. |
From Tarzan to Tolkien: Controlling the Language Proficiency Level of LLMs for Content Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) often output text at a native level of speech, making them difficult to use for contexts where end-users are not fully proficient. |
| Approach: | They propose a framework to control the difficulty level of text generated by Large Language Models for contexts where end-users are not fully proficient. |
| Outcome: | The proposed framework surpasses GPT-4 and other models at fraction of the cost. |
SCALE: Synergized Collaboration of Asymmetric Language Translation Engines (2024.findings-acl)
Copied to clipboard
| Challenge: | In this paper, we introduce SCALE, a collaborative framework that connects a compact Specialized Translation Model (STM) and a general-purpose Large Language Model (LLM) as one unified translation engine. |
| Approach: | They propose a collaborative framework that connects a Specialized Translation Model (STM) and a general-purpose Large Language Model (LLM) as one unified translation engine. |
| Outcome: | The proposed framework outperforms both LLMs and supervised models in high-resource or challenging low-resourced settings. |
Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation (2024.emnlp-main)
Copied to clipboard
| Challenge: | evaluating large language models' output is difficult due to the high cost of human evaluation. |
| Approach: | They propose a family of foundational large autorater models that train on over 100 quality assessment tasks. |
| Outcome: | The proposed model outperforms models on 8 of 12 autorater benchmarks on 53 quality assessment tasks. |
A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment (2025.acl-long)
Copied to clipboard
Jean-Philippe Corbeil, Amin Dada, Jean-Michel Attendu, Asma Ben Abacha, Alessandro Sordoni, Lucas Caccia, Francois Beaulieu, Thomas Lin, Jens Kleesiek, Paul Vozila
| Challenge: | Large language models such as GPT-4 have limited their deployment in clinical settings . a novel framework for adapting SLMs into high-performing clinical models is needed . |
| Approach: | They propose a framework for adapting large language models into high-performing clinical models . they pre-instruct experts on relevant medical and clinical corpora and model merging . |
| Outcome: | The proposed framework outperforms the existing model on the CLUE+ benchmark on medical entities and radiology reports. |
GTA: Gated Toxicity Avoidance for LM Performance Preservation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing Controllable Text Generation methods that generate toxic text can negatively impact the performance of the language model. |
| Approach: | They propose a gated Toxicity Avoidance method that can be applied to any CTG method and evaluate its effectiveness. |
| Outcome: | The proposed method reduces toxicity and preserves performance while preserving the language model's generation performance. |
SedarEval: Automated Evaluation using Self-Adaptive Rubrics (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation paradigms rely on generic scoring rubrics that fail to consider the specificities of each question and its problem-solving process. |
| Approach: | They propose a new evaluation paradigm based on self-adaptive rubrics that mimic a human evaluator's analytical process. |
| Outcome: | The proposed evaluation paradigm achieves higher concordance rate with human graders than existing paradigms, including GPT-4. |
Tuna: Instruction Tuning using Feedback from Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | LLms like LLaMA have shown to be cost-effective for generating better responses . however, the instruction-tuned model has only seen one response per instruction . |
| Approach: | They propose to fine tune an instruction-tuned LLM using probabilistic ranking and contextual ranking approaches to increase the likelihood of generating better responses. |
| Outcome: | The proposed model improves on Super Natural Instructions, LMentry and Vicuna QA. |
Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models (2025.findings-acl)
Copied to clipboard
Haoyang Li, Xuejia Chen, Zhanchao Xu, Darian Li, Nicole Hu, Fei Teng, Yiming Li, Luyu Qiu, Chen Jason Zhang, Li Qing, Lei Chen
| Challenge: | Existing benchmarks focus on linguistic competence or structured mathematical problem-solving, neglecting fundamental numerical reasoning required in real-world scenarios. |
| Approach: | They propose a benchmark to evaluate numerical capabilities for large language models . they use a dataset to assess number recognition, arithmetic operations, contextual retrieval, comparison, summary, and multi-step reasoning. |
| Outcome: | The proposed benchmark evaluates six fundamental numerical capabilities: number recognition, arithmetic operations, contextual retrieval, comparison, summary, and multi-step reasoning. |
Evaluating Evaluation Metrics – The Mirage of Hallucination Detection (2025.findings-emnlp)
Copied to clipboard
Atharva Kulkarni, Yuan Zhang, Joel Ruben Antony Moniz, Xiou Ge, Bo-Hsiang Tseng, Dhivya Piraviperumal, Swabha Swayamdipta, Hong Yu
| Challenge: | a large-scale empirical evaluation of hallucination detection metrics is conducted . hallucinosity is a significant obstacle to the reliability and widespread adoption of language models . |
| Approach: | They conduct large-scale empirical evaluation of hallucination detection metrics . they compare hallucinian language models, language models and decoding methods . |
| Outcome: | The results show that the evaluations of hallucination detection metrics fail to align with human judgments, they say . they also show that evaluations with LLM-based evaluation yield the best overall results . |
GRIZAL: Generative Prior-guided Zero-Shot Temporal Action Localization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to temporally localize videos without prior training examples are lacking due to the complexity of annotated videos. |
| Approach: | They propose a model that uses multimodal embeddings and dynamic motion cues to localize actions effectively. |
| Outcome: | GRIZAL outperforms state-of-the-art zero-shot temporal action localization models on ActivityNet, Thumos14 and Charades-STA datasets. |
Reinforced Query Reasoners for Reasoning-intensive Retrieval Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for retrieval of information excel at textual and semantic matching but struggle in reasoning-intensive retrieval tasks. |
| Approach: | They propose a family of small-scale language models for query reasoning and rewriting in reasoning-intensive retrieval. |
| Outcome: | The proposed model outperforms existing models on a BRIGHT benchmark with BM25 retrievers. |
On the Adaptation of Unlimiformer for Decoder-Only Transformers (2024.lrec-main)
Copied to clipboard
| Challenge: | despite efforts in the community, most common models have a context length of 4k or less. |
| Approach: | They propose to adapt a vector-retrieval augmentation method to decoder-only transformers . they also expand the experimental setup on summarization to include a new task and an instruction-tuned model . |
| Outcome: | The proposed model performs on par with a model with 2x the context length. |
Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter? (2024.emnlp-main)
Copied to clipboard
Nemika Tyagi, Mihir Parmar, Mohith Kulkarni, Aswin Rrv, Nisarg Patel, Mutsumi Nakamura, Arindam Mitra, Chitta Baral
| Challenge: | Existing studies evaluate only the final predicted answer of a puzzle, without providing any finer metrics to evaluate them. |
| Approach: | They propose to use a grid-based evaluation dataset to evaluate LLMs' reasoning abilities and a new error taxonomy to evaluate their reasoning chains. |
| Outcome: | The proposed model outperforms existing prompting methods on a wide range of natural language understanding tasks previously thought to be exclusive to humans. |
MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models (2024.emnlp-main)
Copied to clipboard
Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong
| Challenge: | Existing evaluation frameworks focus on single-turn evaluations, overlooking the models’ capabilities in multi-turn interactions. |
| Approach: | They propose a benchmark to evaluate the multi-turn conversational abilities of large language models (LLMs) by analyzing human-LLM conversations and constructing multi-turned queries for each category using GPT-4. |
| Outcome: | The proposed model outperforms open-source models in multi-turn tasks while retaining and recalling historical information. |
Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models (2024.emnlp-main)
Copied to clipboard
Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varshney, Chitta Baral
| Challenge: | Existing logical reasoning evaluation benchmarks focus on simplistic single-step or multi-step reasoning with limited set of inference rules. |
| Approach: | They propose to use a multi-step logical reasoning evaluation dataset to measure their ability for human-like multi- step logical thinking. |
| Outcome: | The proposed dataset covers three logic types including propositional, first-order, and non-monotonic logic with various inference rules and depths. |
Generalizing Clinical De-identification Models by Privacy-safe Data Augmentation using GPT-4 (2024.emnlp-main)
Copied to clipboard
| Challenge: | De-identification (de-ID) is critical for patient confidentiality in clinical data management due to the difficulty of retaining training corpora and labeling standards vary across institutions. |
| Approach: | They propose to exploit GPT-4 for data augmentation through one-shot and zero-shot prompts to exploit the problem of PHI leakage by redacting PHI before processing. |
| Outcome: | The proposed approach significantly improves on three types of F1 scores in cross-dataset testing. |
RankPrompt: Step-by-Step Comparisons Make Language Models Better Reasoners (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing solutions to reasoning tasks require extensive human annotations or fail in scenarios with inconsistent responses. |
| Approach: | They propose a new method that enables LLMs to self-rank their responses without additional resources. |
| Outcome: | The proposed method improves reasoning performance of ChatGPT and GPT-4 with 13% improvement over existing methods. |
Rapidly Developing High-quality Instruction Data and Evaluation Benchmark for Large Language Models with Minimal Human Effort: A Case Study on Japanese (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have aimed to refine their capacity to accurately follow human instructions and navigate intricate scenarios. |
| Approach: | They propose a method that uses a set of instructions to translate English into Japanese and then generates Japanese instruction data using GPT-4. |
| Outcome: | The proposed method outperforms Japanese-Alpaca models in the evaluation benchmarks without human references. |
STaR-SQL: Self-Taught Reasoner for Text-to-SQL (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for generating step-by-step “chain-of-thought” rationales are limited to text-to-SQL. |
| Approach: | They propose a method that prompts SQL query generation to produce reasoning steps for SQL queries and fine-tunes it on rationales that lead to correct outcomes. |
| Outcome: | The proposed method outperforms agent-like prompting methods on the Spider benchmark. |
Reassessing Semantic Knowledge Encoded in Large Language Models through the Word-in-Context Task (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have propelled significant progress, extending their application across various domains including dialogue systems, text generation, translation systems, and beyond. |
| Approach: | They propose to use the Word-in-Context (WiC) task to reassess the semantic knowledge encoded in large language models (LLMs) they prompt LLMs to generate natural language descriptions that contrast the meanings of the target word in two contextual sentences given in the WiC dataset. |
| Outcome: | The proposed model significantly improves the classification accuracy of the two models. |
GPT-4 Jailbreaks Itself with Near-Perfect Success Using Self-Explanation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using large language models to jailbreak is important for testing safety and security issues. |
| Approach: | They propose an approach that leverages the reflective capabilities of large language models for jailbreaking with only black-box access. |
| Outcome: | The proposed method achieves jailbreak success rates of 98% on GPT-4, 92% on GTP-4 Turbo, and 94% on Llama-3.1-70B in under 7 queries. |
Entity Insertion in Multilingual Linked Corpora: The Case of Wikipedia (2024.emnlp-main)
Copied to clipboard
| Challenge: | a new task for entity insertion in information networks is needed to integrate entities into multilingual linked corpora . text spans in the source and target entities are not available to insert a link to the target entity . a benchmark dataset in 105 languages is used to study the problem of entity inserted in information systems . |
| Approach: | They propose a framework for entity insertion that integrates entities into linked corpora . they compile a benchmark dataset in 105 languages and test it in a zero-shot manner . |
| Outcome: | The proposed framework outperforms baseline models on languages not seen during training with minimal performance drop. |
RATIONALYST: Pre-training Process-Supervision for Improving Reasoning (2025.acl-long)
Copied to clipboard
Dongwei Jiang, Guoxuan Wang, Yining Lu, Andrew Wang, Jingyu Zhang, Chuyu Liu, Benjamin Van Durme, Daniel Khashabi
| Challenge: | RATIONALYST is a model for process-supervision of reasoning based on pretraining on rationale annotations extracted from unlabeled data. |
| Approach: | They propose a model for process-supervision of reasoning based on pre-training on rationale annotations extracted from unlabeled data. |
| Outcome: | RATIONALYST improves reasoning accuracy by 3.9% on representative reasoning benchmarks. |
ReSQL: Self-Improving Framework for Reasoning-Aware Text-to-SQL Dataset Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Experimental results show that ReSQL significantly improves execution accuracy and self-correction ability over strong baselines. |
| Approach: | They propose a framework that generates and learns from its own error-reasoning dataset . it allows models to internalize robust error-reference patterns and apply them to unseen queries . |
| Outcome: | The proposed framework improves execution accuracy and self-correction ability over strong baselines. |
Benchmarking LLMs for Translating Classical Chinese Poetry: Evaluating Adequacy, Fluency, and Elegance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language models fall short of translating culturally significant content . existing models fall behind in achieving such translations, authors say . |
| Approach: | They propose a suitable benchmark for translating classical Chinese poetry into English . they propose RAT, a retrieval-augmented machine translation method that enhances the translation process . |
| Outcome: | The proposed method improves translation quality in terms of adequate, fluent, and elegant translations. |
Can Large Language Models Unlock Novel Scientific Research Ideas? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) and ChatGPT have marked a turning point in the integration of Artificial Intelligence (AI) into people’s everyday lives. |
| Approach: | They conduct a human evaluation of the novelty, relevancy, and feasibility of the generated future research ideas. |
| Outcome: | The proposed models generate more diverse ideas than GPT-4, GPT-3.5, and Gemini 1.0. |
Trojsten Benchmark: Evaluating LLM Problem-Solving in Slovak STEM Competition Problems (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have been used for grading open-ended responses and providing feedback beyond traditional methods. |
| Approach: | They propose a Slovak-language dataset and a rubric-based LLM grading framework . they quantify multistep reasoning performance by difficulty and show consistency under difficult items . |
| Outcome: | The proposed model outperforms existing models on Slovak-language competition problems . the model shows consistent underperformance on harder items and language sensitivity . |