Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been experiencing seismic growth in size and capabilities, radically transforming the field of NLP. |
| Approach: | They propose a generalized variant of iterative self-critique and self-refinement devoid of external influence and a ranking metric to find the optimal model for a given task considering refined performance and cost. |
| Outcome: | The proposed model improves 8.2% from baseline and even with extremely small memory footprints, outperforms ChatGPT post-refinement. |
Similar Papers
Forget What You Know about LLMs Evaluations - LLMs are Like a Chameleon (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) excel on public benchmarks, but high scores may mask overreliance on dataset-specific surface cues rather than true language understanding. |
| Approach: | They propose a meta-evaluation framework that systematically rephrases benchmark inputs to detect overfitting. |
| Outcome: | The proposed framework detects performance degradation indicative of superficial pattern reliance on dataset-specific cues and distortion levels. |
The ART of LLM Refinement: Ask, Refine, and Trust (2024.naacl-long)
Copied to clipboard
Kumar Shridhar, Koustuv Sinha, Andrew Cohen, Tianlu Wang, Ping Yu, Ramakanth Pasunuru, Mrinmaya Sachan, Jason Weston, Asli Celikyilmaz
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable generative abilities, but can they judge the quality of their own generations and self-improve? |
| Approach: | They propose a reasoning with a refinement strategy called *ART: Ask, Refine, and Trust* that asks necessary questions to decide when an LLM should refine its output and uses it to affirm or deny trust. |
| Outcome: | The proposed reasoning with a refinement strategy achieves a performance gain of +5 points over baselines on two multistep reasoning tasks. |
Personalize Your LLM: Fake it then Align it (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing personalization methods require fine-tuning of large language models for each user, rendering them prohibitively expensive for widespread adoption. |
| Approach: | They propose a retrieval-based personalization approach that uses self-generated personal preference data and representation editing to enable quick and cost-effective personalization. |
| Outcome: | The proposed approach outperforms two personalization baselines by 40% on various tasks. |
LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient (2026.acl-long)
Copied to clipboard
Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Using generic and efficient benchmark generators, human annotators are limited by inefficiency . current benchmark generator methods rely on seed signals, leading to long cycles and high costs . |
| Approach: | They propose a framework to evaluate LLMs as generic benchmark generators and integrate them as BenchMaker. |
| Outcome: | The proposed framework achieves comparable performance to human-annotated benchmarks on most metrics. |
LLMs on a Budget? Say HOLA (2025.emnlp-industry)
Copied to clipboard
Zohaib Hasan Siddiqui, Jiechao Gao, Ebad Shabbir, Mohammad Anas Azeez, Rafiq Ali, Gautam Siddharth Kashyap, Usman Naseem
| Challenge: | Current solutions such as quantization, pruning, and Retrieval-Augmented Generation (RAG) offer only partial optimizations and often sacrifice accuracy, speed, or generality. |
| Approach: | They propose an end-to-end optimization framework for efficient LLM deployment . it leverages Hierarchical Speculative Decoding (HSD) for faster inference without quality loss. |
| Outcome: | HOLA delivers +17.6% EMA on GSM8K, +10.5% MCA on ARC, and reduced latency and memory on edge devices like Jetson Nano. |
Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems (2025.emnlp-industry)
Copied to clipboard
Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Hejian Sang, Shao Tang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh S. Pillai, Luke Simon, Rahul Mazumder
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications. |
| Approach: | They propose two techniques for training and deploying small language models that deliver high performance for a variety of industry use cases. |
| Outcome: | The proposed techniques retain much of the quality of larger models while reducing training/serving costs and latency. |
Unlocking LLMs’ Self-Improvement Capacity with Autonomous Learning for Domain Adaptation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models that use self-supervised and instruction fine-tuning can be trained using unlabeled corpora. |
| Approach: | They propose to use unlabeled target corpora to adapt large language models to new domains . they propose to employ self-supervised pre-training and instruction fine-tuning methods . |
| Outcome: | The proposed model can adapt to new domains using only a large amount of unlabeled target corpora. |
LimRank: Less is More for Reasoning-Intensive Information Reranking (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to rerank information require large-scale fine-tuning, which is computationally expensive. |
| Approach: | They propose an open-source pipeline for generating diverse, challenging, and realistic reranking examples. |
| Outcome: | The proposed model performs competitively on two benchmarks, while being trained on less than 5% of the data typically used in prior work. |
SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have transformed machine learning but have raised significant legal concerns due to their potential to produce text that infringes on copyrights. |
| Approach: | They propose a lightweight, real-time defense mechanism to prevent the generation of copyrighted text by evaluating methods and testing attack strategies. |
| Outcome: | The proposed defense significantly reduces the volume of copyrighted text generated by LLMs by effectively refusing malicious requests. |
Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for data-to-text generation are saturated, and there is no way to test them. |
| Approach: | They propose a tool for collecting structured data from public APIs to analyze the behavior of open large language models on the task of data-to-text generation. |
| Outcome: | The proposed model can generate fluent and coherent texts in zero-shot settings from data in common formats collected with Quintd. |