Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Models (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies show that the energy requirements of current NLP models are growing at a rapid, unsustainable pace. |
| Approach: | They investigate ways to measure energy usage and different hardware settings that can be tuned to reduce energy consumption for training and inference for language models. |
| Outcome: | The proposed techniques can reduce energy consumption for training and inference for language models. |
Similar Papers
Energy Considerations of Large Language Model Inference and Efficiency Optimizations (2025.acl-long)
Copied to clipboard
| Challenge: | Prior benchmarking efforts focused on latency reduction in idealized settings, often overlooking real-world inference workloads that shape energy use. |
| Approach: | They propose a modeling approach that approximates real-world LLM workflows . they show that the effectiveness of inference optimizations is sensitive to workload geometry . |
| Outcome: | The proposed approach reduces energy use by 73% from unoptimized baselines. |
Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are recognized for their exceptional generative capabilities and versatility across various tasks. |
| Approach: | They conduct a comprehensive benchmarking of LLM inference energy across a wide range of NLP tasks to determine the impact of different models, tasks, prompts, and system-related factors on inference. |
| Outcome: | The proposed model energy benchmarks show that quantization and optimal batch sizes can significantly reduce energy usage. |
Efficient Methods for Natural Language Processing: A Survey (2023.tacl-1)
Copied to clipboard
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
| Challenge: | Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data, but using only scale to improve performance means resource consumption also grows. |
| Approach: | They propose to use data, time, storage, or energy to improve model performance. |
| Outcome: | The proposed methods and findings provide guidance for conducting NLP under limited resources and point towards promising research directions for developing more efficient methods. |
Energy and Policy Considerations for Deep Learning in NLP (P19-1)
Copied to clipboard
| Challenge: | Recent advances in hardware and methodology for training neural networks have enabled significant accuracy improvements across many NLP tasks. |
| Approach: | They quantify the approximate financial and environmental costs of training neural network models . they propose actionable recommendations to reduce costs and improve equity in NLP research . |
| Outcome: | The proposed recommendations address the cost and environmental costs of training neural networks for NLP. |
Energy and Carbon Considerations of Fine-Tuning BERT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work quantifying energy costs and associated carbon emissions has focused on pretraining and fine-tuning. |
| Approach: | They perform an empirical study to quantify the energy requirements of language model fine-tuning in the context of pretraining and inference. |
| Outcome: | The proposed model fine-tuning energy and carbon footprints are compared with pre-training and inference energy requirements and outline recommendations for NLP researchers and practitioners. |
Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing calibration methods rescale posterior distributions of classifiers after training. |
| Approach: | They propose to use a noise contrastive estimation technique to train an energy-based model during finetuning of pretrained text encoders. |
| Outcome: | The proposed model can reach a better calibration competitive to strong baselines with little or no loss in accuracy. |
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2021.naacl-main)
Copied to clipboard
| Challenge: | Pretraining ever-larger language models on massive corpora requires enormous amounts of compute. |
| Approach: | They propose to convert textual inputs into cloze questions that contain a task description . they also exploit unlabeled data to improve their performance . |
| Outcome: | The proposed model outperforms GPT-3 with PET/iPET with cloze questions and unlabeled data. |
Scaling Laws for BERT in Low-Resource Settings (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models require huge training corpora, which is unobtainable for most NLP practitioners. |
| Approach: | They propose power-law formulas that relate model size, corpora size and computation power to find the optimal settings in advance given a fixed budget. |
| Outcome: | The proposed models perform better on MLM and NLU tasks on four languages of different linguistic characteristics. |
Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems (2025.emnlp-industry)
Copied to clipboard
Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Hejian Sang, Shao Tang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh S. Pillai, Luke Simon, Rahul Mazumder
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications. |
| Approach: | They propose two techniques for training and deploying small language models that deliver high performance for a variety of industry use cases. |
| Outcome: | The proposed techniques retain much of the quality of larger models while reducing training/serving costs and latency. |
Best Practices for Data-Efficient Modeling in NLG:How to Train Production-Ready Neural Models with Less Data (2020.coling-industry)
Copied to clipboard
Ankit Arun, Soumya Batra, Vikas Bhardwaj, Ashwini Challa, Pinar Donmez, Peyman Heidari, Hakan Inan, Shashank Jain, Anuj Kumar, Shawn Mei, Karthik Mohan, Michael White
| Challenge: | Natural language generation (NLG) is a critical component in conversational systems . Traditionally, NLG components have been deployed using template-based solutions . however, deployment of such model-based systems has been challenging due to high latency and data needs. |
| Approach: | They propose a family of techniques to deploy data-efficient neural solutions for NLG in conversational systems to production. |
| Outcome: | The proposed techniques achieve production quality with light-weight neural network models using fraction of the data needed otherwise. |