AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training (2025.emnlp-main)
Copied to clipboard
| Challenge: | Empirically, AdamS demonstrates strong performance in various tasks . et al., 2023b): AdamS is efficient, efficient, and model-agnostic. |
| Approach: | They propose a model-agnostic alternative to Adam for large language model pretraining and post-training. |
| Outcome: | The proposed method matches memory footprint of SGD with momentum while delivering superior performance. |
Similar Papers
PAC-tuning: Fine-tuning Pre-trained Language Models with PAC-driven Perturbed Gradient Descent (2023.emnlp-main)
Copied to clipboard
| Challenge: | PAC-tuning is a two-stage fine-tune method for pretrained language models . PAC training minimizes the PACBayes generalization bound to learn proper parameter distribution . |
| Approach: | They propose a two-stage fine-tuning method to minimize the PAC-Bayes generalization bound . they use PAC to inject noise with variance learned in the first stage into the model parameters . |
| Outcome: | The proposed method outperforms baseline methods on 5 GLUE benchmark tasks. |
BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models have driven major progress in NLP, but memory and compute requirements hinder practical deployment. |
| Approach: | They propose a framework that preserves high accuracy while achieving 1-bit weight quantization . the orthogonal-kronecker transformation learns an orthogonale mapping via EM minimization - a new approach to quantization is proposed . |
| Outcome: | The proposed framework achieves 1-bit weight quantization with low activations with low-bit activations. |
LaMDAgent: An Autonomous Framework for Post-Training Pipeline Optimization via LLM Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to optimize large language models rely on manual design or focus on optimizing individual components. |
| Approach: | They propose a LaMDAgent framework that constructs and optimizes end-to-end post-training pipelines by exploring various model improving methods, objects, and their applied orderings based on task-based feedback. |
| Outcome: | The proposed framework achieves a 9.0-point gain in tool-use accuracy without degrading instruction-following, and reduces computational costs. |
Global Adaptive Momentum Meets Local Personalized Perturbation: Efficient Federated LLM Fine-Tuning with Zeroth-Order Gradients (2026.acl-long)
Copied to clipboard
| Challenge: | federated fine-tuning of large language models provides privacy-preserving approach to deploying pervasive generative AI services. |
| Approach: | They propose a federated framework for fine-tuning large language models . they propose unified optimization and local personalized perturbation for ZO gradients . |
| Outcome: | The proposed framework outperforms existing methods for integrating ZO gradients in federated learning over diverse heterogeneous data settings. |
MUZO: Leveraging Multiple Queries and Momentum for Zeroth-Order Fine-Tuning of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models incur memory overhead due to the need for activation storage for back-propagation (BP). |
| Approach: | They propose a method that estimates gradients through finite differences without activation storage for back-propagation. |
| Outcome: | The proposed method demonstrates superior performance in fine-tuning various LLMs. |
Improving Cross-Domain Low-Resource Text Generation through LLM Post-Editing: A Programmer-Interpreter Approach (2024.findings-eacl)
Copied to clipboard
| Challenge: | Large pre-trained language models such as GPT-3.5 and GPT-4 have gained significant attention in natural language research due to limited computational resources or inaccessible parameters. |
| Approach: | They propose a neural programmer-interpreter approach that preserves the domain generalization ability of LLMs while editing their output. |
| Outcome: | The proposed framework significantly improves GPT-3.5’s performance in logical form-to-text conversion and low-resource machine translation, surpassing other state-of-the-art (SOTA) LLM post-editing methods in cross-domain settings. |
SAGE: Sign-Adaptive Gradient for Memory-Efficient LLM Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to train LLMs consume memory equivalent to twice the model size, resulting in a hybrid design that reverts to AdamW and negates the memory gains. |
| Approach: | They propose a new, memory-efficient O(d) adaptive scale that replaces AdamW in a hybrid structure that combines a Lion-style update direction with a memory-saving adaptive scale. |
| Outcome: | The proposed model outperforms existing methods on LLMs up to 1.3B parameters while significantly reducing optimizer state memory. |
Cuckoo: An IE Free Rider Hatched by Massive Nutrition in LLM’s Nest (2025.acl-long)
Copied to clipboard
| Challenge: | Massive high-quality data, both pre-training raw texts and post-training annotations, have been carefully prepared to incubate advanced large language models (LLMs). |
| Approach: | They propose to reframe next-token prediction into extraction for tokens already present in the context of LLMs by reframing next-tongue prediction into IE models. |
| Outcome: | The proposed model learns 102.6M extractive data converted from pre-training and post-training data with better performance than existing pre-trained IE models. |
TESS 2: A Large-Scale Generalist Diffusion Language Model (2025.acl-long)
Copied to clipboard
| Challenge: | Existing instruction-following diffusion models are predominantly trained using an autoregressive paradigm. |
| Approach: | They propose a general instruction-following diffusion language model that outperforms contemporary instruction-tuned diffusion models and matches and sometimes exceeds strong autoregressive (AR) models. |
| Outcome: | The proposed model outperforms and sometimes exceeds existing autoregressive (AR) models on a number of tasks. |
Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Quantization is a practical solution for deploying Large Language Models in resource-constrained environments. |
| Approach: | They propose an outlier-safe pre-training approach that prevents outlier formation . they validate a 1.4B-parameter model on 1 trillion tokens with no outliers . |
| Outcome: | The proposed model achieves a 35.7 average score on 1 trillion tokens with 2% training overhead. |