Mining Discourse Markers for Unsupervised Sentence Representation Learning (N19-1)
Copied to clipboard
| Challenge: | Current state of the art systems in NLP heavily rely on manually annotated datasets, which are expensive to obtain and are ineffective to extract. |
| Approach: | They propose to automatically discover sentence pairs with relevant discourse markers and apply it to massive amounts of data. |
| Outcome: | The proposed method can learn transferable sentence embeddings from 174 discourse markers even for rare markers such as “coincidentally” or “amazingly”. |
Similar Papers
DiscSense: Automated Semantic Analysis of Discourse Markers (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing models for predicting discourse markers have been used to study link between markers and semantic relations . |
| Approach: | They use a model trained to predict discourse markers between sentence pairs to predict plausible markers between sentences with a known semantic relation. |
| Outcome: | The proposed method predicts markers between sentence pairs with a known semantic relation . the resulting dataset, named DiscSense, is publicly available . |
Distributed Marker Representation for Ambiguous Discourse Markers and Entangled Relations (2023.acl-long)
Copied to clipboard
| Challenge: | Discourse markers are natural representations of discourse in our daily language. |
| Approach: | They propose to use unlimited discourse marker data to learn a Distributed Marker Representation by bridging markers with sentence pairs. |
| Outcome: | The proposed model outperforms existing models on the implicit discourse relation recognition task and provides strong interpretability. |
DisSent: Learning Sentence Representations from Explicit Discourse Relations (P19-1)
Copied to clipboard
| Challenge: | Existing models train on vast amounts of text or require costly, manually curated datasets. |
| Approach: | They propose to leverage the discourse relations between sentences to curate a high quality sentence relation task by leveraging explicit discourse relations. |
| Outcome: | The proposed model can be used to learn the meaning of two sentences in a bidirectional LSTM sentence encoder. |
Discourse Representation Parsing for Sentences and Documents (P19-1)
Copied to clipboard
| Challenge: | Experimental results show that our model outperforms competitive baselines by a wide margin. |
| Approach: | They propose a neural model which parses discourse structures of arbitrary length and granularity. |
| Outcome: | The proposed model outperforms baseline models on sentence- and document-level benchmarks. |
Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for creating metaembeddings from static word embeddings have been proposed, but they are not tied to a particular downstream task. |
| Approach: | They propose a sentence-level meta-embedding learning method that takes contextualised word embedding models and learns a phrase embeddable that preserves complementary strengths of the input source NLMs. |
| Outcome: | The proposed method outperforms existing methods on semantic textual similarity benchmarks on a supervised baseline and on token-level embeddings. |
Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference (P18-1)
Copied to clipboard
| Challenge: | Existing approaches to natural language inference focus on interaction architectures of sentences . but, we propose to transfer knowledge from discourse markers to augment the model . |
| Approach: | They propose to transfer knowledge from discourse markers to augment the quality of the NLI model. |
| Outcome: | The proposed method achieves state-of-the-art performance on large-scale datasets. |
SLM: Learning a Discourse Language Representation with Sentence Unshuffling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent models for learning discourse language representations focus on bottom or top-level representations, but they do not capture intermediate-size structures in natural languages such as sentences and the relationships among them. |
| Approach: | They propose a new objective for learning a discourse language representation in a self-supervised manner by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering. |
| Outcome: | The proposed model improves the original BERT model on downstream tasks by large margins. |
A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the CHATGPT Era and Beyond (2024.eacl-long)
Copied to clipboard
Abhinav Ramesh Kashyap, Thanh-Tung Nguyen, Viktor Schlegel, Stefan Winkler, See-Kiong Ng, Soujanya Poria
| Challenge: | Sentence representations are a critical component in NLP applications such as retrieval, question answering, and text classification. |
| Approach: | They present a systematic review of the literature on sentence representations focusing mostly on deep learning models. |
| Outcome: | The proposed methods highlight the key contributions and challenges in this area and suggest potential avenues for improving the quality and efficiency of sentence representations. |
Bipartite Graph Pre-training for Unsupervised Extractive Summarization with Graph Convolutional Auto-Encoders (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to rank sentences using pre-trained embeddings create a gap due to different optimization objectives. |
| Approach: | They propose a pre-trained embedding process that optimizes informative sentences . they use sentence-word bipartite graphs to model intra-sentential distinctive features . |
| Outcome: | The proposed model outperforms heavy BERT- or RoBERTa-based sentence ranking methods by providing summary-worthy representations. |
Contextualized and Generalized Sentence Representations by Contrastive Self-Supervised Learning: A Case Study on Discourse Relation Analysis (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to learn contextualized and generalized sentence representations are limited by the size of manually annotated data. |
| Approach: | They propose a method to learn contextualized and generalized sentence representations using contrastive self-supervised learning. |
| Outcome: | The proposed method outperforms baseline methods based on BERT, XLNet, and RoBERTa in English and Japanese and outperformed strong baseline methods. |