Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models (LLMs) have demonstrated exceptional performance in various natural language processing (NLP) tasks. |
| Approach: | They propose a new compression paradigm that extracts knowledge from pre-trained language models to construct a knowledge store from which the model can leverage it for effective inference. |
| Outcome: | The proposed model extracts knowledge from LLMs to construct a knowledge store, which the model can leverage for effective inference. |
Similar Papers
Self-Knowledge Guided Retrieval Augmentation for Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown superior performance without task-specific fine-tuning due to the computational costs. |
| Approach: | They propose a method which lets LLMs refer to the questions they have previously encountered and adaptively call for external resources when dealing with new questions. |
| Outcome: | The proposed method outperforms chain-of-thought based and fully retrieval-based methods on multiple datasets and outperformed chain- of-though, chatGPT and InstructGPT. |
Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems (2025.emnlp-industry)
Copied to clipboard
Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Hejian Sang, Shao Tang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh S. Pillai, Luke Simon, Rahul Mazumder
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications. |
| Approach: | They propose two techniques for training and deploying small language models that deliver high performance for a variety of industry use cases. |
| Outcome: | The proposed techniques retain much of the quality of larger models while reducing training/serving costs and latency. |
TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | ChatGPT and GPT-4 are commercial large language models (LLMs) however, they may produce vague responses or incorrect answers in certain specialized domains. |
| Approach: | They propose a token compression scheme that uses summarization and semantic compression to reduce the token size of LLMs. |
| Outcome: | The proposed method reduces token size by doing summarization and semantic compression while reducing token size with only 1.6% accuracy drop. |
When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit excellent performance in various tasks, but memory requirements present a challenge when deploying on memory-limited devices. |
| Approach: | They propose a framework to compress LLM after quantization further, achieving about 2.2x compression ratio. |
| Outcome: | The proposed model can achieve 40% reduction in memory size with negligible loss in accuracy and inference speed. |
How to inject knowledge efficiently? Knowledge Infusion Scaling Law for Pre-training Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that strategically infusing domain knowledge during pretraining can substantially improve downstream performance. |
| Approach: | They propose a knowledge infusion scaling law that predicts the optimal amount of domain knowledge to inject into large LLMs by analyzing their smaller counterparts. |
| Outcome: | The proposed model predicts the optimal amount of domain knowledge to inject into large LLMs by analyzing their smaller counterparts. |
Query Rewriting in Retrieval-Augmented Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on adapting either the retriever or the reader, but this approach is more focused on adaptation of the query itself. |
| Approach: | They propose a new framework for retrieval-augmented Large Language Models . they propose rewrite-retrieve-read instead of retrieve-then-read . |
| Outcome: | The proposed framework improves performance on downstream tasks, open-domain QA and multiple-choice QA. |
The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on LLM compression focuses on general metrics like perplexity or downstream task accuracy. |
| Approach: | They propose to quantify the effect of pruning and quantization on model quality . they use the LAMA and LM-Harness benchmarks to quantify compression techniques . |
| Outcome: | The proposed compression techniques provide faster inference, smaller memory footprints, and enables local deployment. |
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly lengthy and require longer prompts . this paper presents a coarse-to-fine prompt compression method to reduce cost and increase performance. |
| Approach: | They propose a coarse-to-fine prompt compression method that maintains semantic integrity under high compression ratios and a token-level iterative compression algorithm to better model the interdependence between compressed contents. |
| Outcome: | The proposed method yields state-of-the-art performance and allows for up to 20x compression with little performance loss over four datasets from different scenarios. |
Fast Vocabulary Transfer for Language Model Compression (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods to reduce model size and size are expensive and inefficient for some applications. |
| Approach: | They propose a method that relies on vocabulary transfer to reduce model size and inference time while compromising on performance. |
| Outcome: | The proposed method reduces model size and inference time while compromising on performance. |
Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study (2023.emnlp-main)
Copied to clipboard
Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, Anima Anandkumar, Bryan Catanzaro
| Challenge: | a recent study shows that retrieval-augmented LMs can improve text generation quality and accuracy. |
| Approach: | They propose a model that reproduces RETRO parameters while retrieving a text corpus . they find RETRO outperforms GPT on text generation with less repetition . |
| Outcome: | The proposed model outperforms standard retrieval-augmented GPT and retrieval augmented GTP on text generation and accuracy tasks. |