Papers with NTP
Beyond Tokens: Concept-Level Training Objectives for LLMs (2026.eacl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) are trained with a surprisingly narrow objective: predicting the next token in a sequence. |
| Approach: | They propose a shift from token-level to concept-level prediction where concepts group multiple surface forms of the same idea. |
| Outcome: | The proposed model improves on human-level models on diverse NLP benchmarks. |
SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe (2026.acl-long)
Copied to clipboard
| Challenge: | Efforts to improve instruction tuning often focus on higher-quality supervised fine-tuning datasets, typically requiring data filtering with proprietary LLMs or human annotation. |
| Approach: | They propose a Mixup-based recipe that elevates LLM instruction tuning without relying on well-curated datasets. |
| Outcome: | The proposed model improves instruction-following and healthcare-specific tasks with consistent improvements across LLM families and SFT datasets. |
Reinforcement Learning on Pre-Training Data (2026.acl-long)
Copied to clipboard
Siheng Li, Kejiao Li, Zenan Xu, Guanhua Huang, Kun Li, Haoyuan Wu, null Wujiajia, Zihao Zheng, Chenchen Zhang, Kun Shi, Xue Gong, Qi Yi, Ruibin Xiong, Tingqiang Xu, Yuhao Jiang, Jianfeng Yan, Yuyuan Zeng, Guanghui Xu, Jinbao Xue, Zhijiang xu, Zheng Fang, Shuai LI, Qibin Liu, Xiaoxue Li, Zhuoyu Li, Yangyu Tao, Fei Gao, Cheng Jiang, Bochao Wang, Kai Liu, Jianchen Zhu, Wai Lam, Bo Zhou, Di Wang
| Challenge: | Recent progress in large language models is driven by scaling of training compute through pre-training with nexttoken prediction (NTP) or post-training (RL) Pre-training using NTP enables models to acquire extensive knowledge and skills from general data, but it suffers from data inefficiency and catastrophic forgetting in continual learning settings. |
| Approach: | They propose to scale training compute through pre-training with next-token prediction (NTP) or post-training by scaling reinforcement learning (RL) to improve learning from general data. |
| Outcome: | Experiments on multiple benchmarks and models show that the proposed approach improves continual pre-training and provides a strong foundation for post-training on Qwen3-8B-Base. |
Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to learn internal world models rely on one-step supervision . however, standard MTP suffers from structural hallucinations . |
| Approach: | They propose a method which anchors predictions to ground-truth hidden state trajectories. |
| Outcome: | The proposed method bridges the gap between discrete tokens and continuous state representations, reducing structural hallucinations, and improving robustness to perturbations. |
Fast Quiet-STaR: Thinking Without Thought Tokens (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models have achieved impressive performance across a range of tasks, but further gains require more than scaling up model sizes or training data. |
| Approach: | They propose a method that gradually reduces the number of thought tokens . this method allows models to internalize more abstract reasoning processes . |
| Outcome: | The proposed framework preserves the benefits of token-level reasoning while reducing computational cost. |
Pre-Training Curriculum for Multi-Token Prediction in Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Multi-token prediction (MTP) is a pre-training objective for language models . prior work has shown that smaller language models struggle with the MTP objective . |
| Approach: | They propose a curriculum learning strategy that uses multiple prediction heads to predict the next tokens at each prediction step. |
| Outcome: | The proposed curriculum improves performance and output quality while retaining the benefits of self-speculative decoding. |
Planning-Aware Code Infilling via Horizon-Length Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to fill-in-the-middle (FIM) often fail to generate content that aligns well with the surrounding context. |
| Approach: | They propose a training objective that teaches models to predict the number of remaining middle tokens at each step. |
| Outcome: | The proposed training objective improves FIM performance by up to 24% on diverse benchmarks across file-level and repository-level. |