Teaching Language Models to Self-Improve by Learning from Language Feedback (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) generate content that can be untruthful or harmful. |
| Approach: | They propose a method that leverages model feedback for alignment . they use a base language model to generate initial responses, critiqued and refined . |
| Outcome: | The proposed method outperforms strong baselines across diverse tasks and model sizes. |
Similar Papers
Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge (2025.emnlp-main)
Copied to clipboard
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason E Weston, Sainbayar Sukhbaatar
| Challenge: | Existing methods for improving large language models have focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. |
| Approach: | They propose an iterative Meta-Rewarding step where the model judges its own judgements and uses that feedback to refine its judgment skills. |
| Outcome: | The proposed model improves Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2 and 20.6% to 29.1% on Arena-Hard. |
Self-Refine Instruction-Tuning for Aligning Reasoning in Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to align reasoning abilities between Large Language Models and Smaller Language Model are supervised fine-tuning and preference optimization. |
| Approach: | They propose a method that elicits Smaller Language Models to self-improve their reasoning abilities via preference optimization. |
| Outcome: | The proposed method outperforms Instruction-tuning on commonsense and math reasoning tasks on common and math scenarios. |
Aligning Large Language Models via Fully Self-Synthetic Data (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to reinforcement learning from human feedback (RLHF) require expensive human-annotated datasets and proprietary models like GPT-4 to annotate preference pairs. |
| Approach: | They propose a self-synthetic framework for LLM alignment where all training data, including prompts (i.e., user queries), responses, and preferences, are generated by the model itself. |
| Outcome: | The proposed framework enhances the model’s chat capabilities on standard benchmarks like AlpacaEval 2.0 while maintaining strong performance on downstream objective tasks. |
Aligning Large Language Models through Synthetic Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | Currently, alignment learning requires significant human demonstrations and feedback from proprietary LLMs such as ChatGPT. |
| Approach: | They propose a framework that uses synthetic feedback to align large language models to human values without extensive human annotations and proprietary LLMs. |
| Outcome: | The proposed model outperforms open-source models on human-annotated demonstrations in alignment benchmarks. |
Self-Instruct: Aligning Language Models with Self-Generated Instructions (2023.acl-long)
Copied to clipboard
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi
| Challenge: | Large “instruction-tuned” language models depend heavily on human-written instruction data . this limited quantity, diversity, and creativity hinders the generality of the tuned model . |
| Approach: | They propose a framework for improving instruction-following capabilities of pretrained language models by bootstrapping off their own generations. |
| Outcome: | The proposed framework outperforms existing public instruction datasets by 5% . it generates instructions, input, and output samples, then filters invalid or similar ones . |
Advancing Large Language Model Attribution through Self-Improving (2024.emnlp-main)
Copied to clipboard
Lei Huang, Xiaocheng Feng, Weitao Ma, Liang Zhao, Yuchun Fan, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin
| Challenge: | Teaching large language models to generate text with citations to evidence sources requires high-quality attribution data, which is costly and labor-intensive. |
| Approach: | They propose a framework for iteratively improving the attribution capability of large language models (LLMs) by attributing output to verifiable sources. |
| Outcome: | Experiments on three open-domain question-answering datasets show that START improves in aggregating information across multiple sources. |
Self-Critique and Refinement for Faithful Natural Language Explanations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has demonstrated that Large Language Models (LLMs) can self-critique and refine their initial outputs, but this capability remains unexplored for improving explanation faithfulness. |
| Approach: | They propose a framework that enables models to improve the faithfulness of their own explanations through an iterative critique and refinement process without external supervision. |
| Outcome: | The proposed framework reduces unfaithfulness rates in three datasets and four state-of-the-art LLMs by 36% compared to 54.81% for baseline. |
Unlocking LLMs’ Self-Improvement Capacity with Autonomous Learning for Domain Adaptation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models that use self-supervised and instruction fine-tuning can be trained using unlabeled corpora. |
| Approach: | They propose to use unlabeled target corpora to adapt large language models to new domains . they propose to employ self-supervised pre-training and instruction fine-tuning methods . |
| Outcome: | The proposed model can adapt to new domains using only a large amount of unlabeled target corpora. |
Small Language Models Need Strong Verifiers to Self-Correct Reasoning (2024.findings-acl)
Copied to clipboard
Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, Lu Wang
| Challenge: | Existing studies show that large language models can self-correct their outputs by generating a critique and revising it based on the critique. |
| Approach: | They propose a pipeline that prompts small language models to collect self-correction data that supports the training of self-refinement abilities. |
| Outcome: | The proposed pipeline improves the self-correction abilities of two models on five datasets spanning math and commonsense reasoning. |
Calibrating LLM-Based Evaluator (2024.lrec-main)
Copied to clipboard
Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
| Challenge: | Existing models for large language models lack the ability to calibrate their outputs towards human preference. |
| Approach: | They propose a multi-stage, gradient-free approach to calibrate an LLM-based evaluator toward human preference. |
| Outcome: | The proposed approach improves correlation with expert evaluation on multiple text quality evaluation datasets. |