David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMs (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies of diffusion-based language models have been conducted on a smaller scale. |
| Approach: | They propose to scale an autoregressive diffusion model from 0.4B to 13B parameters and propose techniques to improve its training and inference efficiency. |
| Outcome: | The proposed model is able to combine a large general-purpose diffusion model with smaller, but specialized and contextualized diffusion models at inference time. |
Similar Papers
SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control (2023.acl-long)
Copied to clipboard
| Challenge: | Existing diffusion models for continuous-valued domains have not been adopted for text data. |
| Approach: | They propose a diffusion-based language model with two key design choices . semi-autoregressive model generates blocks of text and allows local context updates . they evaluate it on unconstrained text generation benchmarks . |
| Outcome: | The proposed model outperforms autoregressive models on unconstrained text generation benchmarks on uncontrolled text generation. |
TESS 2: A Large-Scale Generalist Diffusion Language Model (2025.acl-long)
Copied to clipboard
| Challenge: | Existing instruction-following diffusion models are predominantly trained using an autoregressive paradigm. |
| Approach: | They propose a general instruction-following diffusion language model that outperforms contemporary instruction-tuned diffusion models and matches and sometimes exceeds strong autoregressive (AR) models. |
| Outcome: | The proposed model outperforms and sometimes exceeds existing autoregressive (AR) models on a number of tasks. |
dLLM: Simple Diffusion Language Modeling (2026.acl-demo)
Copied to clipboard
| Challenge: | diffusion language models (DLMs) are evolving rapidly but many lack transparent implementations or are scattered across codebases. |
| Approach: | They propose an open-source framework that unifies diffusion language modeling components while remaining flexible enough to support new methods and architectures. |
| Outcome: | dLLM unifies the core components of diffusion language modeling and makes them easy to customize for new designs. |
Segment-Level Diffusion: A Framework for Controllable Long-Form Generation with Diffusion Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Diffusion models have shown promise in text generation, but often struggle with generating long, coherent, and contextually accurate text. |
| Approach: | They propose a framework that enhances diffusion-based text generation through text segmentation, robust representation training with adversarial and contrastive learning, and improved latent-space guidance. |
| Outcome: | The proposed framework improves diffusion-based text generation and improves scalability and fluency. |
LLM2: Let Large Language Models Harness System 2 Reasoning (2025.naacl-short)
Copied to clipboard
| Challenge: | Empirical results on mathematical reasoning benchmarks substantiate the efficacy of Large language models (LLMs). |
| Approach: | They propose a framework that combines an LLM with a process-based verifier to generate plausible candidates and provide timely process-driven feedback to distinguish desirable and undesirable outputs. |
| Outcome: | Empirical results show that LLM2 improves accuracy on GSM8K and self-consistency increases major@20 accuracy. |
Reinforcement Learning for Diffusion LLMs via Energy-Based Gibbs Alignment (2026.acl-long)
Copied to clipboard
| Challenge: | Diffusion Large Language Models (dLLMs) offer parallel decoding and bidirectional context modeling . aligning dLLms with reinforcement learning (RL) remains a challenge . |
| Approach: | They propose a variational framework that reformulates RL for dLLMs as a distribution matching problem. |
| Outcome: | The proposed framework reformulates RL for dLLMs as a distribution matching problem. |
InferenceDynamics: Adaptive LLM Routing through Structured Capability and Knowledge Profiling (2026.acl-long)
Copied to clipboard
Haochen Shi, Tianshi Zheng, Weiqi Wang, Baixuan Xu, Chunyang Li, Chunkit Chan, Tao Fan, Yangqiu Song
| Challenge: | Large Language Model (LLM) routing is a pivotal technique for navigating a diverse landscape of LLMs. |
| Approach: | They propose a flexible and scalable multi-dimensional routing framework that models the capability and knowledge of models. |
| Outcome: | The proposed framework can be used to generalize and identify top-performing models for group-level routing using modern benchmarks including MMLU-Pro, GPQA, BigGenBench, and LiveBench. |
Benchmarking Diffusion Models for Machine Translation (2024.eacl-srw)
Copied to clipboard
| Challenge: | Diffusion models have shown great potential on many generative tasks, but their application to natural language processing (NLP) is still a less explored direction. |
| Approach: | They adapt two diffusion-based text generation models, Diffusion-LM and DiffuSeq, to perform machine translation. |
| Outcome: | The proposed models struggle more on long-range dependencies than other models. |
DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | Autoregressive (AR) large audio language models are expensive in data and computation . prior work shows diffusion-based LALMs can improve audio understanding under matched settings . |
| Approach: | They propose a diffusion-based LALM that upgrades the speech encoder and employs dual semantic and acoustic adapters. |
| Outcome: | a new model improves over existing autoregressive large language models and is competitive to strong AR models . the proposed model can make use of limited training data and improve inference efficiency . a recent study shows that diffusion-based models can improve audio understanding . |
Diffuse Thinking: Exploring Diffusion Language Models as Efficient Thought Proposers for Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated strong capabilities in complex reasoning tasks, yet their autoregressive generation paradigm makes it computationally prohibitive to explore diverse reasoning paths. |
| Approach: | They propose a framework that combines diffusion-based generation with autoregressive evaluation to efficiently generate diverse intermediate reasoning thoughts and employ LLMs as evaluators to assess and select candidates based on their plausibility and correctness. |
| Outcome: | The proposed framework improves inference efficiency while maintaining competitive or superior reasoning accuracy. |