| Challenge: | Existing memory-efficient methods require second-moment estimates of the per-parameter gradients to maintain their performance. |
| Approach: | They propose to use memory-efficient optimizers to reduce memory usage by preserving second-moment estimates of gradients. |
| Outcome: | The proposed method achieves fast convergence and lower memory usage across training tasks. |
Similar Papers
AdaLomo: Low-memory Optimization with Adaptive Learning Rate (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models require substantial memory for training, thereby setting a high hardware threshold. |
| Approach: | They propose a low-memory optimization technique that reduces memory footprint . they propose an adaptive learning rate for each parameter and a grouped update normalization to stabilize convergence . |
| Outcome: | The proposed low-memory optimization performs better than the prevailing algorithm for large language models, AdamW. |
AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training (2025.emnlp-main)
Copied to clipboard
| Challenge: | Empirically, AdamS demonstrates strong performance in various tasks . et al., 2023b): AdamS is efficient, efficient, and model-agnostic. |
| Approach: | They propose a model-agnostic alternative to Adam for large language model pretraining and post-training. |
| Outcome: | The proposed method matches memory footprint of SGD with momentum while delivering superior performance. |
AdapLeR: Speeding up Inference by Adaptive Length Reduction (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models have shown stellar performance in downstream tasks, but their excessive computational costs and high latency hinder their usage in resource-limited settings. |
| Approach: | They propose a method that dynamically eliminates less contributing tokens through layers, resulting in shorter lengths and consequently lower computational cost. |
| Outcome: | The proposed method shows speedups up to 22x during inference time without much sacrifice in performance. |
Flashback: Memory Mechanism for Enhancing Memory Efficiency and Speed in Deep Sequential Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing deep sequential processing models have problems with memory degradation and inaccurate gradient backpropagation. |
| Approach: | They propose a Flashback property that preserves memory as an identity mapping until it is overwritten by a hidden state at a different time step. |
| Outcome: | The proposed model can be implemented in Transformers and Mamba, and it performs well. |
SAGE: Sign-Adaptive Gradient for Memory-Efficient LLM Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to train LLMs consume memory equivalent to twice the model size, resulting in a hybrid design that reverts to AdamW and negates the memory gains. |
| Approach: | They propose a new, memory-efficient O(d) adaptive scale that replaces AdamW in a hybrid structure that combines a Lion-style update direction with a memory-saving adaptive scale. |
| Outcome: | The proposed model outperforms existing methods on LLMs up to 1.3B parameters while significantly reducing optimizer state memory. |
Guided by Gut: Efficient Test-Time Scaling with Reinforced Intrinsic Confidence (2026.acl-long)
Copied to clipboard
| Challenge: | Guided by Gut (GG) is an efficient self-guided TTS framework for Large Language Models (LLMs) that performs step-by-step reasoning at a low cost without any reward models or verifiers. |
| Approach: | They propose a self-guided TTS framework that enables LLMs to perform step-by-step reasoning at a low cost without any reward models or verifiers. |
| Outcome: | Empirical evaluations show that GG performs better than TTS with PRMs while reducing GPU memory usage by up to 10. |
Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Adaptive Retention is a probabilistic, layer-wise token selection mechanism that learns which representations to keep under a strict global budget M. |
| Approach: | They propose a probabilistic token selection mechanism that learns which representations to keep under a strict global budget M. |
| Outcome: | The proposed method reduces memory usage by 35–45% while improving throughput by 1.8. |
STEP: Staged Parameter-Efficient Pre-training for Large Language Models (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing methods for reducing computational costs during pre-training have been studied, but they often degrade performance under fair conditions. |
| Approach: | They propose a method that combines parameter-efficient tuning and staged training to reduce memory requirements while maintaining comparable performance. |
| Outcome: | The proposed method reduces memory requirements by 40.4% while maintaining comparable performance. |
Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing pruning strategies struggle to enhance robustness against adversarial attacks when continually increasing model sparsity and require a retraining process. |
| Approach: | They propose a pruning strategy that replicates embedding space and feature space of dense language models and aims to conserve more pre-trained knowledge during the pruning process. |
| Outcome: | The proposed pruning strategy replicates embedding space and feature space of dense language models, aiming to conserve more pre-trained knowledge during the pruning process. |
Consistent Accelerated Inference via Confident Adaptive Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Amortized or approximate computational methods increase efficiency, but can result in unpredictable performance costs. |
| Approach: | They propose a method that increases computational efficiency while guaranteeing a specifiable degree of consistency with the original model with high confidence. |
| Outcome: | The proposed method improves on four classification and regression tasks and can be used to predict the performance of the proposed model. |