Discursive Circuits: How Do Language Models Understand Discourse Relations? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Discourse structure is essential for ensuring language models behave safely and ethically. |
| Approach: | They propose a task where a model completes a discourse given a specified relation . they propose CUDR task that enables activation patching to make circuit discovery feasible . |
| Outcome: | The proposed model recovers discourse understanding in the English PDTB-based CuDR task. |
Similar Papers
What Context Features Can Transformer Language Models Use? (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. |
| Approach: | They propose to use lexical and structural information to ablate usable information in transformer language models. |
| Outcome: | The proposed model improves when conditioning on contexts of thousands of previous tokens. |
Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies on reasoning in language models have sparked a debate on whether they can learn systematic inferential principles or merely exploit superficial patterns in the training data. |
| Approach: | They propose a method for circuit discovery aimed at interpreting syllogistic inference . they uncover a circuit involving middle-term suppression that elucidates how LMs transfer information to derive valid conclusions from premises. |
| Outcome: | The proposed method elucidates how LMs transfer information to derive valid conclusions from premises. |
Is Partial Linguistic Information Sufficient for Discourse Connective Disambiguation? A Case Study of Concession (2025.acl-srw)
Copied to clipboard
| Challenge: | Discourse relations are often not linguistically marked, but there are various connectives that explicitly signal discourse relations. |
| Approach: | They analyze linguistic features that play an important role in disambiguation of polysemous connectives in Japanese by performing a neural language model. |
| Outcome: | The proposed model performed well after removal of one of the two arguments that constitute the discourse relation, but significantly degraded disambiguation performance. |
Towards Identifying Alternative-Lexicalization Signals of Discourse Relations (2022.coling-1)
Copied to clipboard
| Challenge: | Existing shallow discourse parsing methods have been limited to identifying relations signaled by a discourse connective and those without a signal. |
| Approach: | They propose to identify relations signalled by a discourse connective and those without . they compare a pattern-based approach and a sequence labeling model . |
| Outcome: | The proposed approach is based on a pattern-based approach and a sequence labeling model. |
Exploring Discourse Structures for Argument Impact Classification (2021.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that discourse structures influence the persuasiveness of arguments. |
| Approach: | They propose to fuse sentence-level structural discourse information with contextualized features derived from large-scale language models to investigate how discourse relations influence argument impact. |
| Outcome: | The proposed model improves its backbone RoBERTa around 1.67%, compared with other models, but side effects are brought by other models. |
Identifying Semantic Induction Heads to Understand In-Context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance, but lack of transparency in their inference logic raises concerns about their trustworthiness. |
| Approach: | They conduct a detailed analysis of the operations of attention heads to understand their in-context learning of LLMs. |
| Outcome: | The proposed analysis of attention heads reveals that they increase the output logits of object tokens and recall objects . the proposed model is a novel approach to understand the in-context learning of large language models. |
Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to pre-training/fine-tuning are focusing on the alignment of pre-trained and fine-tuned PLMs with large-scale discourse structures. |
| Approach: | They propose a novel approach to infer discourse information for arbitrarily long documents using supervised, distantly supervised and simple baselines. |
| Outcome: | The proposed approach shows that the captured discourse information is local and general, even across fine-tuning tasks. |
Towards Interpretable Sequence Continuation: Analyzing Shared Circuits in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent work aims to reverse engineer transformer models into human-readable representations . transformers exhibit strong capabilities on linguistic tasks, but their complex architectures make them difficult to interpret. |
| Approach: | They extend transformer models into human-readable representations that implement algorithmic functions by analyzing sequence continuation tasks. |
| Outcome: | The proposed model can be reverse-engineered into human-readable representations that implement algorithmic functions. |
Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts? (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in augmenting Large Language Models (LLMs) with auxiliary information have significantly revolutionized their efficacy in knowledge-intensive tasks. |
| Approach: | They propose a systematic framework to identify whether LLMs’ responses are attributed to either generated or retrieved contexts. |
| Outcome: | The proposed framework identifies whether LLMs’ responses are attributed to either generated or retrieved contexts. |
A Joint Model for Dropped Pronoun Recovery and Conversational Discourse Parsing in Chinese Conversational Speech (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work regards dropped pronoun recovery and conversational discourse parsing as two separate tasks and tackles them separately. |
| Approach: | They propose a neural model for dropped pronoun recovery and conversational discourse parsing in Chinese conversational speech. |
| Outcome: | The proposed model outperforms the state-of-the-art models on a new dataset . the proposed model is based on linguistic and semantic information from Chinese conversational speech . |