Papers by Clara Na
Energy and Carbon Considerations of Fine-Tuning BERT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work quantifying energy costs and associated carbon emissions has focused on pretraining and fine-tuning. |
| Approach: | They perform an empirical study to quantify the energy requirements of language model fine-tuning in the context of pretraining and inference. |
| Outcome: | The proposed model fine-tuning energy and carbon footprints are compared with pre-training and inference energy requirements and outline recommendations for NLP researchers and practitioners. |
Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in hardware, modeling, and optimization for deep neural networks have led to improvements in memory and inference efficiency. |
| Approach: | They propose to combine sharpness-aware minimization with various model compression methods to improve model compressibility. |
| Outcome: | Empirically, optimizing for flatter minima leads to greater compressibility of parameters compared to vanilla Adam when fine-tuning BERT models, with little to no loss in accuracy on the GLUE text classification and SQuAD question answering benchmarks. |
To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Natural language processing (NLP) is in a period of disruptive change that is impacting our methodologies, funding sources, and public perception. |
| Approach: | They conduct interviews with 26 NLP researchers of varying seniority, research area, institution, and social identity to identify cyclical patterns in the field and new shifts without historical parallel . they conclude by discussing shared visions, concerns, and hopes for the future of NLP . |
| Outcome: | The authors identify cyclical patterns in the field, as well as new shifts without historical parallel, including changes in benchmark culture and software infrastructure. |
The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment (2023.emnlp-main)
Copied to clipboard
| Challenge: | Inference is estimated to make up 80 to 90% of ML cloud computing demand . |
| Approach: | They propose to identify bottlenecks in deep learning frameworks that are causing the disparity in model latency as hardware speed increases over time. |
| Outcome: | The proposed models show that the framework tax is increasing as the hardware speed increases over time. |
Scalable Data Ablation Approximations for Language Models through Modular Training and Merging (2024.emnlp-main)
Copied to clipboard
Clara Na, Ian Magnusson, Ananya Harsh Jha, Tom Sherborne, Emma Strubell, Jesse Dodge, Pradeep Dasigi
| Challenge: | Training data compositions for Large Language Models (LLMs) can significantly affect their downstream performance. |
| Approach: | They propose a method which trains individual models on subsets of a training corpus and reuses them across evaluations of combinations of subset. |
| Outcome: | The proposed method improves training efficiency by scaling only linearly with respect to new data. |
Energy Considerations of Large Language Model Inference and Efficiency Optimizations (2025.acl-long)
Copied to clipboard
| Challenge: | Prior benchmarking efforts focused on latency reduction in idealized settings, often overlooking real-world inference workloads that shape energy use. |
| Approach: | They propose a modeling approach that approximates real-world LLM workflows . they show that the effectiveness of inference optimizations is sensitive to workload geometry . |
| Outcome: | The proposed approach reduces energy use by 73% from unoptimized baselines. |