SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | SLMs offer computational efficiency and accessibility, yet a systematic evaluation of their performance and environmental impact remains lacking. |
| Approach: | SLM-Bench evaluates 15 SLMs on 9 NLP tasks using 23 datasets . compared accuracy, computational efficiency, and sustainability metrics . |
| Outcome: | SLM-Bench evaluates 15 SLMs on 9 NLP tasks using 23 datasets spanning 14 domains. |
Similar Papers
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models (2025.naacl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo
| Challenge: | a recent study evaluated language models using abstract evaluation criteria that lack the flexibility and granularity of human assessment. |
| Approach: | They propose a benchmark to evaluate nine distinct language models' capabilities . they use instance-specific evaluation criteria to mirror human evaluation . |
| Outcome: | The proposed benchmark evaluates nine distinct capabilities of language models across 77 tasks. |
Demystifying Small Language Models for Edge Deployment (2025.acl-long)
Copied to clipboard
Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Wei Liu, Jian Luan, Xiwen Zhang, Nicholas D. Lane, Mengwei Xu
| Challenge: | Small language models (SLMs) are a promising solution for resource-constrained devices such as smartphones and the Web of Things. |
| Approach: | They propose to use SLMs to build and optimize a set of small language models that are publicly accessible. |
| Outcome: | The proposed models outperform 7B models in general tasks, while their in-context learning capabilities remain limited and their efficiency has significant optimization potential. |
Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are recognized for their exceptional generative capabilities and versatility across various tasks. |
| Approach: | They conduct a comprehensive benchmarking of LLM inference energy across a wide range of NLP tasks to determine the impact of different models, tasks, prompts, and system-related factors on inference. |
| Outcome: | The proposed model energy benchmarks show that quantization and optimal batch sizes can significantly reduce energy usage. |
Counterspeech Generation using Small Language Models (2026.acl-srw)
Copied to clipboard
| Challenge: | Social media use is growing annually with about 68.5% of the global population active on these platforms as of July 2025. |
| Approach: | They evaluate SLMs ranging from 100 million to 3 billion parameters using simple prompting strategies as well as fine-tuning, combining automatic and robust human evaluations. |
| Outcome: | The proposed models generate relevant, coherent, and high-quality counterspeech, suggesting their suitability for efficient and responsible deployments. |
Waste-Bench: A Comprehensive Benchmark for Evaluating VLLMs in Cluttered Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have paved the way for VisionLarge Language Model (VLLM) capabilities have not been thoroughly explored in cluttered datasets where there is complex environment having deformedshaped objects. |
| Approach: | They propose a dataset specifically designed for waste classification in real-world scenarios, characterized by complex environments and deformed shaped objects. |
| Outcome: | The proposed dataset provides valuable insights into the performance of VLLMs under challenging conditions. |
Navigating the Modern Evaluation Landscape: Considerations in Benchmarks and Frameworks for Large Language Models (LLMs) (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | General-purpose Language Models have changed the world of Natural Language Processing, if not the world itself. |
| Approach: | This tutorial will lay the foundations and explain the basics of evaluation and compare traditional methods to newly developed methods. |
| Outcome: | The tutorial assumes little familiarity with metrics, datasets, prompts and benchmarks . it will compare traditional methods to newly developed methods . |
Evaluating Large Language Models with Enterprise Benchmarks (2025.naacl-industry)
Copied to clipboard
Bing Zhang, Mikio Takeuchi, Ryo Kawahara, Shubhi Asthana, Maruf Hossain, Guang-Jie Ren, Kate Soule, Yifan Mai, Yada Zhu
| Challenge: | Existing benchmarks lack domain-specific datasets for evaluating large language models . existing benchmarks often lack domain specific datasets, which can be difficult to convert to standardized metrics or regulatory issues. |
| Approach: | They propose to use 25 publicly available domain-specific English benchmarks from diverse domains . they propose to combine a wide range of natural language processing tasks for holistic evaluation . |
| Outcome: | The proposed framework includes 25 publicly available domain-specific English benchmarks from diverse enterprise domains like financial services, legal, climate, cyber security, and 2 public Japanese finance benchmarks. |
The Sound of Syntax: Finetuning and Comprehensive Evaluation of Language Models for Speech Pathology (2025.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art multimodal language models (MLMs) show promise for supporting SLPs, but their use remains underexplored due to a limited understanding of their performance in high-stakes clinical settings. |
| Approach: | They propose a taxonomy of real-world use cases of multimodal language models in speech-language pathologies to address this gap. |
| Outcome: | The proposed model outperforms 15 state-of-the-art models in speech-language pathologies across five use cases and achieves improvements of over 30% on domain-specific data. |
Unveiling Environmental Impacts of Large Language Model Serving: A Functional Unit View (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies benchmark carbon emissions but lack a standardized basis for comparison across different model configurations. |
| Approach: | They propose a functional unit (FU) as a standardized basis for comparison across different model configurations and introduce FUEL, the first FU-based framework for evaluating LLM serving’s environmental impact. |
| Outcome: | The proposed framework uncovers key insights and trade-offs in reducing carbon emissions by optimizing model size, quantization strategy, and hardware choice, paving the way for more sustainable LLM serving. |
ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Recent work in language modeling has led to effective SLMs with impressive performance levels across various benchmarks. |
| Approach: | They propose a benchmark that introduces process-level evaluation for commonsense reasoning tasks. |
| Outcome: | The proposed benchmarks show that large language models provide correct answers despite flawed reasoning processes in a substantial portion of cases. |