How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models are not suitable for task-dependent tasks. |
| Approach: | They propose a generalized self-imitation learning framework which aligns large language models with offline demonstration data. |
| Outcome: | The proposed framework outperforms baselines in many challenging benchmarks . it is available on github.com/tengxiao1/GSIL . |
Similar Papers
Inverse Reinforcement Learning Meets Large Language Model Alignment (2025.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial will provide a comprehensive review of recent advances in LLM alignment . it will highlight the necessity of constructing neural reward models from human data . |
| Approach: | This tutorial will provide a comprehensive review of recent advances in LLM alignment through the lens of inverse reinforcement learning. |
| Outcome: | This tutorial will provide a comprehensive review of recent advances in LLM alignment through the lens of inverse reinforcement learning (IRL). |
Self-Specialization: Uncovering Latent Expertise within Large Language Models (2024.findings-acl)
Copied to clipboard
Junmo Kang, Hongyin Luo, Yada Zhu, Jacob Hansen, James Glass, David Cox, Alan Ritter, Rogerio Feris, Leonid Karlinsky
| Challenge: | Recent studies have demonstrated the effectiveness of self-alignment in which a large language model is aligned to follow general instructions using instructional data generated from the model itself. |
| Approach: | They propose to use human-written seeds to align large language models to follow general instructions to achieve cross-task generalization. |
| Outcome: | The proposed model outperforms base models and models that are generally instruction-tuned or have been adapted to the target domain by a large margin. |
GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNets (2024.emnlp-main)
Copied to clipboard
| Challenge: | Reinforcement learning with human feedback (RLHF) and its offline variant Direct Preference Optimization (DPO) are two of the most important methods for language model (LM) alignment. |
| Approach: | They propose to use a diversity-seeking RL algorithm called GFlowNet-DPO in an offline preference alignment setting to optimize a model's behavior. |
| Outcome: | Empirical results show that the proposed algorithm generates far more diverse responses than the baseline methods and is still relatively aligned with human values in dialog generation and summarization tasks. |
Aligning Large Language Models via Fully Self-Synthetic Data (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to reinforcement learning from human feedback (RLHF) require expensive human-annotated datasets and proprietary models like GPT-4 to annotate preference pairs. |
| Approach: | They propose a self-synthetic framework for LLM alignment where all training data, including prompts (i.e., user queries), responses, and preferences, are generated by the model itself. |
| Outcome: | The proposed framework enhances the model’s chat capabilities on standard benchmarks like AlpacaEval 2.0 while maintaining strong performance on downstream objective tasks. |
Towards Autonomous Tool Utilization in Language Models: A Unified, Efficient and Scalable Framework (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in tool learning for large language models have led to a new trend to allow LLMs to leverage external tools. |
| Approach: | They propose a framework for fine-tuning language models that categorizes queries into three different types . they also introduce an "instruct, execute, and reformat" strategy specifically designed for efficient data annotation . |
| Outcome: | The proposed framework surpasses open-source language models and GPT-3.5/4 on multiple evaluation metrics. |
Self-Instruct: Aligning Language Models with Self-Generated Instructions (2023.acl-long)
Copied to clipboard
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi
| Challenge: | Large “instruction-tuned” language models depend heavily on human-written instruction data . this limited quantity, diversity, and creativity hinders the generality of the tuned model . |
| Approach: | They propose a framework for improving instruction-following capabilities of pretrained language models by bootstrapping off their own generations. |
| Outcome: | The proposed framework outperforms existing public instruction datasets by 5% . it generates instructions, input, and output samples, then filters invalid or similar ones . |
CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current studies have focused on fine-tuning, but the use of instruction tuning is not as effective as fine-cuning. |
| Approach: | They propose a commonality-aware instruction tuning strategy to cluster instruction datasets into distinct groups with three proposed metrics Task, Embedding and Length. |
| Outcome: | The proposed strategy boosts an average improvement of 2.1% on the general domain and 5.2% on the special domain. |
Don’t Just Say “I don’t know”! Self-aligning Large Language Models for Responding to Unknown Questions with Explanations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies investigate ways to refuse to answer unknown questions . Large Language Models (LLMs) display a significant level of overconfidence when answering questions that they are aware of. |
| Approach: | They propose a self-alignment method to utilize Large Language Models to enhance its response-ability to unknown questions. |
| Outcome: | The proposed method is superior to baseline methods on four types of unknown questions. |
Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are renowned for their extensive linguistic knowledge and strong generalization capabilities, but their high computational demands make them unsuitable for resource-constrained environments. |
| Approach: | They propose a framework that integrates a single frozen layer from an LLM into a SLM and fine-tunes the combined model for specific tasks. |
| Outcome: | The proposed framework improves performance across a range of natural language processing tasks, including both natural language understanding and generation. |
Cool-Fusion: Fuse Large Language Models without Training (2025.acl-long)
Copied to clipboard
| Challenge: | Cool-Fusion is a simple yet effective approach to combine two or more heterogeneous large language models . |
| Approach: | They propose a method that fuses the knowledge of two or more heterogeneous large language models to leverage complementary strengths. |
| Outcome: | The proposed method increases accuracy from three strong source LLMs on GSM8K by 17.4%. |