Meeseeks: A Feedback-Driven, Iterative Self-Correction Benchmark evaluating LLMs’ Instruction Following Capability

Challenge:	Existing models lack the ability to adhere to instructions, resulting in suboptimal performance.
Approach:	They propose an automated iterative instruction-following benchmark with integrated feedback mechanism.
Outcome:	The proposed benchmark identifies erroneous components in model responses and provides feedback accurately.

Similar Papers

Overview

Challenge

Approach

Outcome

Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs (2025.acl-long)

Copied to clipboard

Zhe Yang, Yichang Zhang, Yudong Wang, Ziyao Xu, Junyang Lin, Zhifang Sui

Challenge:	Existing approaches to improve self-correction performance of Large Language Models are based on intrinsic selfcorrectione, which allows the model to check and revise its selfgenerated answers without external feedback.
Approach:	They propose to decompose the self-correction capability into confidence and critique capabilities and a metric for overall self-corretion capability evaluation.
Outcome:	The proposed method outperforms vanilla SFT and achieves much higher accuracy after self-correction.

Self-Correction Makes LLMs Better Parsers (2025.findings-emnlp)

Copied to clipboard

Ziyan Zhang, Yang Hou, Chen Gong, Zhenghua Li

Syntactic Text Processing

Challenge:	Large language models (LLMs) have achieved remarkable success across various natural language processing tasks, but they still face challenges in performing fundamental NLP tasks, such as syntactic parsing.
Approach:	They propose a method that leverages grammar rules from existing treebanks to guide LLMs in correcting previous errors.
Outcome:	The proposed method significantly improves performance on in-domain and cross-domain datasets.

LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models (2023.emnlp-main)

Copied to clipboard

Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu

Language Models

Challenge:	Large language models (LLMs) are increasingly lengthy and require longer prompts . this paper presents a coarse-to-fine prompt compression method to reduce cost and increase performance.
Approach:	They propose a coarse-to-fine prompt compression method that maintains semantic integrity under high compression ratios and a token-level iterative compression algorithm to better model the interdependence between compressed contents.
Outcome:	The proposed method yields state-of-the-art performance and allows for up to 20x compression with little performance loss over four datasets from different scenarios.

Harnessing Large Language Models as Post-hoc Correctors (2024.findings-acl)

Copied to clipboard

Zhiqiang Zhong, Kuangyu Zhou, Davide Mottin

Language Models

Challenge:	Recent advances in Large Language Models (LLMs) have demonstrated their effectiveness in a wide range of tasks, including machine translation and commonsense reasoning.
Approach:	They propose a training-free framework that can work as a post-hoc corrector to propose corrections for ML models.
Outcome:	The proposed framework improves the performance of a number of models by up to 39% on text analysis and the challenging molecular predictions.

IterAlign: Iterative Constitutional Alignment of Large Language Models (2024.naacl-long)

Copied to clipboard

Xiusi Chen, Hongzhi Wen, Sreyashi Nag, Chen Luo, Qingyu Yin, Ruirui Li, Zheng Li, Wei Wang

Semantic Text Processing

Challenge:	Empirical results show that iterAlign improves truthfulness, helpfulness, harmlessness and honesty, improving the LLM alignment by up to 13.5% in harmlessness.
Approach:	They propose a data-driven constitution discovery and self-alignment framework called IterAlign to overcome these drawbacks by leveraging red teaming to uncover weaknesses of an LLM.
Outcome:	Empirical results show that iterAlign improves truthfulness, helpfulness, harmlessness and honesty by up to 13.5%.

Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge (2025.emnlp-main)

Copied to clipboard

Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason E Weston, Sainbayar Sukhbaatar

Language Models

Challenge:	Existing methods for improving large language models have focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training.
Approach:	They propose an iterative Meta-Rewarding step where the model judges its own judgements and uses that feedback to refine its judgment skills.
Outcome:	The proposed model improves Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2 and 20.6% to 29.1% on Arena-Hard.

Instruct-of-Reflection: Enhancing Large Language Models Iterative Reflection Capabilities via Dynamic-Meta Instruction (2025.naacl-long)

Copied to clipboard

Liping Liu, Chunhong Zhang, Likang Wu, Chuang Zhao, Zheng Hu, Ming He, Jianping Fan

Language Models

Challenge:	Existing approaches involve models iterating and improving their previous responses based on internal reflection ability or external feedback.
Approach:	They propose a reflection framework that leverages meta-thoughts and self-consistency to enhance the iterative reflection capability of Large LanguageModels.
Outcome:	The proposed framework achieves an average improvement of 10.1% over established baselines in mathematical and commonsense reasoning tasks, highlighting its efficacy and applicability.

ReIFE: Re-evaluating Instruction-Following Evaluation (2025.naacl-long)

Copied to clipboard

Yixin Liu, Kejian Shi, Alexander Fabbri, Yilun Zhao, PeiFeng Wang, Chien-Sheng Wu, Shafiq Joty, Arman Cohan

Robustness in NLP

Challenge:	Existing evaluations of large language models (LLMs) for instruction following are incomplete.
Approach:	They propose to use 25 base LLMs and 15 recently proposed evaluation protocols to evaluate instruction following on 4 human-annotated datasets.
Outcome:	The proposed evaluations identify the best-performing base LLMs and evaluation protocols with a high degree of robustness.

Understanding the Dark Side of LLMs’ Intrinsic Self-Correction (2025.acl-long)

Copied to clipboard

Qingjie Zhang, Di Wang, Haoting Qian, Yiming Li, Tianwei Zhang, Minlie Huang, Ke Xu, Hewu Li, Liu Yan, Han Qiu

Challenge:	Recent studies show that LLMs’ intrinsic self-correction fails without oracle labels as feedback.
Approach:	They propose to use one simple task and three complex tasks with state-of-the-art LLMs like ChatGPT, Llama, and DeepSeek to interpret LLM's intrinsic self-correction.
Outcome:	The proposed methods reveal the dark side of LLMs’ intrinsic self-correction for different tasks, especially for those failure cases.

Do LLMs Catch Their Own Mistakes? A Comprehensive Benchmark for Reflective Tool Use LLMs (2026.findings-acl)

Copied to clipboard

Zheyuan Liu, Liqiang Xiao, Yang Li, Hyokun Yun, Lihong Li, Chao Zhang, Meng Jiang

Dialogue Systems & Conversational Agents

Challenge:	Existing benchmarks primarily evaluate planning and execution success, overlooking the self-reflective dimension of tool use.
Approach:	They propose a benchmark to assess LLMs’ self-reflective reasoning in tool-augmented multi-turn dialogues.
Outcome:	The proposed benchmark covers 10 domains with 88 distinct APIs and 968 annotated dialogues, systematically injecting diverse error types arising from both user and assistant behavior.