| Challenge: | a new study examines pretrained language models' ability to model discourse and pragmatic phenomena. |
| Approach: | They propose a sentence intrusion detection task using a dictionary dataset . they show that pretrained LMs perform impressively in in-domain evaluation . |
| Outcome: | The proposed dataset shows that pretrained LMs perform impressively in in-domain evaluation, but experience a substantial drop in the cross-domain setting, indicating limited generalization capacity. |
Similar Papers
Discourse Probing of Pretrained Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing work on probing of pretrained language models has focused on sentence-level syntactic tasks. |
| Approach: | They introduce document-level discourse probing to evaluate the ability of pretrained LMs to capture document- level relations. |
| Outcome: | The proposed model performs best in encoder, but only in the encoder layer. |
Probing Language Models for Pre-training Data Detection (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown impressive capabilities, while raising concerns about the data contamination due to privacy issues and leakage of benchmark datasets in the pre-training phase. |
| Approach: | They propose to utilize the probing technique to examine the model’s internal activations to detect pre-training data contamination by examining the model's internal activates. |
| Outcome: | The proposed method outperforms baselines and achieves state-of-the-art performance on both WikiMIA and ArxivMIA, with additional experiments confirming its efficacy. |
Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent models for unsupervised representation learning of text have put little focus on discourse-level representations. |
| Approach: | They propose an inter-sentence objective for pretraining language models that models discourse coherence and the distance between sentences. |
| Outcome: | The proposed model outperforms the BERT-Large model on the discourse representation benchmark DiscoEval and yields gains of 2%-6% absolute even for tasks that do not explicitly evaluate discourse. |
Pretrained Language Models for Sequential Sentence Classification (D19-1)
Copied to clipboard
| Challenge: | Recent successful models for document-level understanding have used hierarchical encoding and CRFs to capture dependencies between subsequent labels. |
| Approach: | They propose a pretrained language model that captures contextual dependencies without hierarchical encoding nor a CRF. |
| Outcome: | The proposed model captures contextual dependencies without hierarchical encoding nor a CRF on four datasets, including a new dataset of structured scientific abstracts. |
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020.acl-main)
Copied to clipboard
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A. Smith
| Challenge: | Language models prerained on text from a wide variety of sources form the foundation of today’s NLP. |
| Approach: | They propose to tailor a pretrained model to the domain of a target task by using domain-adaptive pretraining in-domain. |
| Outcome: | The proposed model can be tailored to the domain of a target task and perform well under both high- and low-resource settings. |
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
Identifying Pre-training Data in LLMs: A Neuron Activation-Based Detection Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for detecting pre-training data in large language models rely on superficial features like prediction confidence and loss, resulting in mediocre performance. |
| Approach: | They propose a new algorithm to analyze neuron activation patterns between training and non-training data in large language models to improve their performance. |
| Outcome: | The proposed algorithm outperforms existing methods across three benchmarks and multiple LLMs. |
Dive into the Chasm: Probing the Gap between In- and Cross-Topic Generalization (2024.findings-eacl)
Copied to clipboard
| Challenge: | Pre-trained language models perform well in In-Topic setups, but face challenges in Cross-Topical setups where testing data is derived from distinct topics. |
| Approach: | They propose a probing-based approach to analyze pre-trained language models in a Cross-Topic setup to better understand the reasons behind generalization gaps. |
| Outcome: | The proposed approach improves on pre-trained language models in In-Topic setups and Cross-Topical scenarios. |
Tag&Tab: Pretraining Data Detection in Large Language Models Using Keyword-Based Membership Inference Attack (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies on detecting pretraining data in large language models have focused on sentence-level membership inference attacks (MIAs) but these methods often exhibit poor accuracy, failing to account for the semantic importance of textual content and word significance. |
| Approach: | They propose a method that leverages established natural language processing techniques to tag keywords in input text and then uses them to obtain probabilities and calculate their average log-likelihood to determine input text membership. |
| Outcome: | The proposed method exploits established natural language processing techniques to tag keywords in input text and calculate their average log-likelihood to determine input text membership. |
What do Large Language Models Learn beyond Language? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretraining on text confers models with useful ‘inductive biases’ for non-linguistic reasoning. |
| Approach: | They investigate whether pre-training on text confers these models with helpful ‘inductive biases’ for non-linguistic reasoning. |
| Outcome: | The proposed models outperform non-pretrained models on 19 non-linguistic tasks and show that they retain inductive biases even when training on multi-lingual text and computer code. |