Quantile Regression with Large Language Models for Price Prediction (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to structured prediction tasks focus on point estimates and lack systematic comparison across different methods. |
| Approach: | They propose a novel quantile regression approach that enables LLMs to produce full predictive distributions, improving upon traditional point estimates. |
| Outcome: | The proposed model outperforms encoder architectures, embedding-based methods, and few-shot learning methods in prediction accuracy and distributional calibration. |
Similar Papers
Text-to-Distribution Prediction with Quantile Tokens and Neighbor Context (2026.acl-long)
Copied to clipboard
Yilun Zhu, Yuan Zhuang, Nikhita Vedula, Dushyanta Dhyani, Shaoyuan Xu, Mohsen Bayati, Bryan Wang, Shervin Malmasi
| Challenge: | Existing methods for text regression lack local grounding and rely on shared representations. |
| Approach: | They propose a distributional regression model with quantile tokens that insert dedicated quantiles into the input sequence. |
| Outcome: | The proposed method outperforms baseline models on the inside Airbnb and StackSample datasets. |
LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit (2024.emnlp-industry)
Copied to clipboard
Ruihao Gong, Yang Yong, Shiqiao Gu, Yushi Huang, Chengtao Lv, Yunchen Zhang, Dacheng Tao, Xianglong Liu
| Challenge: | Existing quantization techniques have been categorized as 'simple' and 'highly efficient' however, their configurations vary from each other and cannot be fairly compared . |
| Approach: | They propose a plug-and-play compression toolkit to explore the impact of quantization. |
| Outcome: | The proposed toolkit explores the impact of quantization on large language models. |
A Comprehensive Evaluation of Quantization Strategies for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Quantization studies have focused on instruction-tuned LLMs, leaving their performance on other benchmarks unclear. |
| Approach: | They propose a framework to evaluate quantized large language models using four dimensions . they propose to reduce the bits needed for model weights or activations with minimal performance loss . |
| Outcome: | The proposed framework can retain comparable performance to non-quantized LLMs on most benchmarks. |
Order of Magnitude Speedups for LLM Membership Inference (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are complex and require fine-tuning on proprietary datasets to improve performance and relevance. |
| Approach: | They propose a low-cost membership inference attack that leverages an ensemble of small quantile regression models to determine if a document belongs to the model’s training set. |
| Outcome: | The proposed approach achieves comparable or improved accuracy on fine-tuned LLMs of varying families and across multiple datasets. |
Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems (2025.emnlp-industry)
Copied to clipboard
Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Hejian Sang, Shao Tang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh S. Pillai, Luke Simon, Rahul Mazumder
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications. |
| Approach: | They propose two techniques for training and deploying small language models that deliver high performance for a variety of industry use cases. |
| Outcome: | The proposed techniques retain much of the quality of larger models while reducing training/serving costs and latency. |
QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Structured pruning can reduce model size but results in significant accuracy degradation . quantization and pruning increase the difficulty of fine-tuning, requiring a more refined quantization scheme. |
| Approach: | They propose a structured pruning framework followed by a layer-wise mixed-precision quantization scheme to reduce model memory consumption during fine-tuning and inference. |
| Outcome: | Experiments on benchmark datasets show that QPruner outperforms existing methods in memory savings while maintaining or improving model performance. |
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis (2022.findings-emnlp)
Copied to clipboard
Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, Louis-Philippe Morency
| Challenge: | Pre-trained language models (PLMs) have gained increasing popularity due to compelling prediction performance in diverse natural language processing tasks. |
| Approach: | They compare three popular options for encoding and Temp Scaling for PLMs . they recommend using Temp Loss as uncertainty quantifier and Focal Loss for fine-tuning . |
| Outcome: | Using pre-trained language models, we compare three options on NLP classification tasks and domain shift. |
Predicting Language Models’ Success at Zero-Shot Probabilistic Prediction (2025.findings-emnlp)
Copied to clipboard
Kevin Ren, Santiago Cortes-Gomez, Carlos Miguel Patiño, Ananya Joshi, Ruiqi Lyu, Jingjing Tang, Alistair Turcan, Khurram Yamin, Steven Wu, Bryan Wilder
| Challenge: | Recent work has investigated the capabilities of large language models (LLMs) as zero-shot models for generating individual-level characteristics. |
| Approach: | They conduct a large-scale empirical study of large language models’ zero-shot predictive capabilities across a wide range of tabular prediction tasks. |
| Outcome: | The results show that LLMs perform well on the base prediction task, and when they perform well, they are more likely to provide high-quality predictions. |
EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has shown that large language models are superior to conventional methods in various tasks. |
| Approach: | They propose a data-independent quantization algorithm that leaves outliers in the weight and quantization ranges . they find the algorithm runs over 10 times faster than the data-dependent methods . |
| Outcome: | The proposed method runs over 10 times faster than the data-dependent methods. |
Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models (2024.acl-long)
Copied to clipboard
Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, Kaidi Xu
| Challenge: | Large Language Models (LLMs) show promising results in language generation but often “hallucinate”, making their outputs less reliable. |
| Approach: | They propose to shift attention to more relevant components at token- and sentence-levels for better UQ. |
| Outcome: | The proposed approach improves the performance of a range of popular “off-the-shelf” LLMs with model sizes extending up to 33B parameters. |