Text-to-Distribution Prediction with Quantile Tokens and Neighbor Context (2026.acl-long)
Copied to clipboard
Yilun Zhu, Yuan Zhuang, Nikhita Vedula, Dushyanta Dhyani, Shaoyuan Xu, Mohsen Bayati, Bryan Wang, Shervin Malmasi
| Challenge: | Existing methods for text regression lack local grounding and rely on shared representations. |
| Approach: | They propose a distributional regression model with quantile tokens that insert dedicated quantiles into the input sequence. |
| Outcome: | The proposed method outperforms baseline models on the inside Airbnb and StackSample datasets. |
Similar Papers
Quantile Regression with Large Language Models for Price Prediction (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to structured prediction tasks focus on point estimates and lack systematic comparison across different methods. |
| Approach: | They propose a novel quantile regression approach that enables LLMs to produce full predictive distributions, improving upon traditional point estimates. |
| Outcome: | The proposed model outperforms encoder architectures, embedding-based methods, and few-shot learning methods in prediction accuracy and distributional calibration. |
Data-to-text Generation by Splicing Together Nearest Neighbors (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on data-to-text generation relies on retrieved "neighbors" but instead generates text token-by-token, left-to right. |
| Approach: | They propose to splice together retrieved segments of text from "neighbor" source-target pairs to generate text token-by-token, left-to-right. |
| Outcome: | The proposed method performs on par with strong baselines in terms of automatic and human evaluation, but allows for more interpretable and controllable generation. |
Order of Magnitude Speedups for LLM Membership Inference (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are complex and require fine-tuning on proprietary datasets to improve performance and relevance. |
| Approach: | They propose a low-cost membership inference attack that leverages an ensemble of small quantile regression models to determine if a document belongs to the model’s training set. |
| Outcome: | The proposed approach achieves comparable or improved accuracy on fine-tuned LLMs of varying families and across multiple datasets. |
Predicting Through Generation: Why Generation Is Better for Prediction (2025.acl-long)
Copied to clipboard
Md Kowsher, Nusrat Jahan Prottasha, Prakash Bhat, Chun-Nam Yu, Mojtaba Soltanalian, Ivan Garibay, Ozlem Garibay, Chen Chen, Niloofar Yousefi
| Challenge: | Large Language Models (LLMs) are increasingly used for predictive tasks such as classification and regression. |
| Approach: | They propose a framework that generates output tokens from mas-sive text corpora and a task adapter to ensure consistency between token generation and final prediction. |
| Outcome: | The proposed framework outperforms baseline models on classification and regression benchmarks and the proposed framework consistently outperformed standard baseline models. |
Token Prediction as Implicit Classification to Identify LLM-Generated Text (2023.emnlp-main)
Copied to clipboard
| Challenge: | a novel approach for identifying large language models (LLMs) involved in text generation is proposed . instead of adding an additional classification layer, we reframe the classification task as a next-token prediction task . |
| Approach: | They propose a novel approach for identifying large language models involved in text generation . instead of adding an additional classification layer, they reframe the task as a next-token prediction task . |
| Outcome: | The proposed method performs exceptionally well in the text classification task . it can distinguish distinctive writing styles among various LLMs even without an explicit classifier. |
TReX: Tokenizer Regression for Optimal Data Mixture (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to train and inference tokenizers rely on heuristics or large-scale searches to determine optimal data mixtures. |
| Approach: | They propose a regression-based framework that efficiently predicts the optimal data mixture for tokenizer training. |
| Outcome: | The proposed model outperforms mixtures based on LLaMA3 and uniform distributions by up to 12% in both in- and out-of-distribution compression efficiency. |
Understanding the Influence of Synthetic Data for Text Embedders (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in general purpose text embedders have been driven by training on synthetic training data. |
| Approach: | They propose to use GPT-4 to produce high quality synthetic data that expands existing training datasets for embeddings to new tasks. |
| Outcome: | The proposed dataset is high quality and leads to consistent improvements in performance. |
A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression (2025.acl-long)
Copied to clipboard
| Challenge: | gist-based context compression methods can achieve only slight performance loss on tasks like retrieval-augmented generation and long-document QA, but it faces challenges in tasks like synthetic recall. |
| Approach: | They propose two strategies to improve gist-based context compression in large language models. |
| Outcome: | The proposed methods can achieve only slight performance loss on retrieval-augmented generation and long-document QA tasks, but they face challenges in tasks like synthetic recall. |
Building Static Embeddings from Contextual Ones: Is It Useful for Building Distributional Thesauri? (2022.lrec-1)
Copied to clipboard
| Challenge: | contextual language models are dominant in the field of Natural Language Processing, but they are not suitable for all uses. |
| Approach: | They propose a method for building word or type-level embeddings from contextual models . they evaluate a large set of English nouns from the perspective of extracting semantic similarity relations . |
| Outcome: | The proposed method can be used to build word or type embeddings from contextual models . it can be exploited for a wide set of English nouns, showing it can improve distributional thesauri . |
KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Recent work shows that decoder-only LLMs can serve as strong embedding backbones when fine-tuned with contrastive objectives. |
| Approach: | They propose a framework that activates the latent representation power of frozen LLMs by rerouting the final token's KV states as a prepended prefix. |
| Outcome: | The proposed framework outperforms existing training-free baselines by 10% on MTEB and maintains robust performance on sequences up to 4,096 tokens. |