Curricular Next Conversation Prediction Pretraining for Transcript Segmentation (2023.findings-eacl)
Copied to clipboard
| Challenge: | Prior research on document segmentation has focused on segmenting documents such as Wikipedia articles. |
| Approach: | They propose to pretrain a model to identify consecutive conversations to address these challenges . they introduce a curriculum to Advanced NCP to make the task more relevant to the downstream task . |
| Outcome: | The proposed model outperforms previous models in speech recognition errors and is robust to speech recognition. |
Similar Papers
Transition-Matrix Regularization for Next Dialogue Act Prediction in Counselling Conversations (2026.findings-acl)
Copied to clipboard
| Challenge: | Evaluated on a 60-class German counselling taxonomy, this improves macro-F1 by 9–42% relative depending on encoder and corpus-derived transition patterns. |
| Approach: | They propose to use a KL regularization term to align next dialogue act distributions with corpus-derived transition patterns to improve macro-F1 by 9–42% relative to encoders. |
| Outcome: | The proposed term improves macro-F1 by 9–42% relative to encoders and significantly improves dialogue-flow alignment. |
Pretraining Methods for Dialog Context Representation Learning (P19-1)
Copied to clipboard
| Challenge: | Existing methods for pretraining dialog context encoders are still in their infancy. |
| Approach: | They propose to use unsupervised pretraining objectives for dialog context representations to fine-tune and evaluate them on a set of downstream dialog tasks. |
| Outcome: | The proposed methods improve performance on a set of dialog tasks and are less data hungry. |
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
Pre-Training Curriculum for Multi-Token Prediction in Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Multi-token prediction (MTP) is a pre-training objective for language models . prior work has shown that smaller language models struggle with the MTP objective . |
| Approach: | They propose a curriculum learning strategy that uses multiple prediction heads to predict the next tokens at each prediction step. |
| Outcome: | The proposed curriculum improves performance and output quality while retaining the benefits of self-speculative decoding. |
Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning (2026.eacl-long)
Copied to clipboard
| Challenge: | Curriculum learning has improved efficiency across machine learning domains, but remains underexplored for language model pretraining. |
| Approach: | They present a systematic investigation of curriculum learning in LLM pretraining . they use vanilla curriculum learning, pacing-based sampling, and interleaved curricula . |
| Outcome: | The proposed framework accelerates convergence in early and mid-training phases, reducing training steps by 18-45% to reach baseline performance. |
FutureTOD: Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue (2023.acl-long)
Copied to clipboard
| Challenge: | Existing pre-trained language models rely on a contrastive framework and are difficult to use in practice. |
| Approach: | They propose a dialogue pre-training model which distills future knowledge to the representation of the previous dialogue context using a self-training framework. |
| Outcome: | The proposed model can be applied to various downstream dialogue tasks. |
Improving Pretraining Techniques for Code-Switched NLP (2023.acl-long)
Copied to clipboard
| Challenge: | Multilingual pretraining models for code-switched inputs are a key component of NLP applications. |
| Approach: | They propose to use masked language modeling techniques to mask code-switched text that are cognizant of language boundaries prior to masking. |
| Outcome: | The proposed techniques improve performance on two downstream tasks, Question Answering (QA) and Sentiment Analysis (SA), compared to standard pretraining techniques. |
One-Dimensional Object Detection for Streaming Text Segmentation of Meeting Dialogue (2025.findings-acl)
Copied to clipboard
Rui He, Zhongqing Wang, Minjie Qiang, Hongling Wang, Yifan.zhang Yifan.zhang, Hua Xu, Shuai Fan, Guodong Zhou
| Challenge: | Current text segmentation models exhibit numerous limitations, such as imbalances in labels that affect the stability of model training and discrepancies between the model’s training tasks (sentence classification) and the actual text segmenting. |
| Approach: | They implement a sliding window-based segmentation method and employ two different levels of sliding window based balanced label strategies to stabilize the training process of the streaming segmentation model. |
| Outcome: | The proposed method is robust, controllable, and achieves state-of-the-art performance. |
Gap-K%: Measuring Top-1 Prediction Gap for Detecting Pretraining Data (2026.acl-long)
Copied to clipboard
| Challenge: | Existing state-of-the-art methods for pretraining data are largely undisclosed, resulting in ethical and copyright concerns. |
| Approach: | They propose a method that leverages the log probability gap between the top-1 predicted token and the target token, incorporating a sliding window strategy to capture local correlations and mitigate token-level fluctuations. |
| Outcome: | The proposed method outperforms baselines on WikiMIA and MIMIR benchmarks and achieves state-of-the-art performance. |
Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies describe how to apply contrastive learning to the language domain but it is difficult to apply data augmentation methods directly to language modeling. |
| Approach: | They propose a memory-efficient continual pretraining method that applies contrastive learning with novel data augmentation and curriculum learning. |
| Outcome: | The proposed method outperforms baseline models on sentence-level tasks with only 70% of memory compared to the baseline model. |