Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations (D19-1)
Copied to clipboard
| Challenge: | Prior work on pretrained sentence embeddings and benchmarks focused on the capabilities of stand-alone sentences. |
| Approach: | They propose a test suite of tasks to evaluate whether sentence representations include broader context information. |
| Outcome: | The proposed training objectives help to encode different aspects of information in document structures. |
Similar Papers
SentEval: An Evaluation Toolkit for Universal Sentence Representations (L18-1)
Copied to clipboard
| Challenge: | a toolkit for evaluating the quality of universal sentence representations is available for download and preprocessing . word embeddings are not trained to perform well on one specific task, but their value lies in their transferability . evaluation of general-purpose word and sentence embeddables has been problematic . |
| Approach: | They propose a toolkit to evaluate the quality of universal sentence representations. |
| Outcome: | The proposed toolkit includes scripts to download and preprocess datasets and an easy interface to evaluate sentence encoders. |
Learning Visually Grounded Sentence Representations (N18-1)
Copied to clipboard
| Challenge: | Unsupervised sentence representation models suffer from the grounding problem because of lack of association between symbols and external information. |
| Approach: | They train a sentence encoder to predict image features of a caption and use them as sentence representations. |
| Outcome: | The proposed model improves on word embeddings and word representations on standard benchmarks. |
Bipartite Graph Pre-training for Unsupervised Extractive Summarization with Graph Convolutional Auto-Encoders (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to rank sentences using pre-trained embeddings create a gap due to different optimization objectives. |
| Approach: | They propose a pre-trained embedding process that optimizes informative sentences . they use sentence-word bipartite graphs to model intra-sentential distinctive features . |
| Outcome: | The proposed model outperforms heavy BERT- or RoBERTa-based sentence ranking methods by providing summary-worthy representations. |
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Sentence embeddings are useful for language processing tasks, but it is unclear how to produce them from encoder-decoder models. |
| Approach: | They investigate the effects of scaling up sentence encoders to 11B parameters on sentence embeddings from text-to-text transformers (T5) . |
| Outcome: | The proposed models outperform the previous best models on both SentEval and SentGLUE transfer tasks. |
Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent models for unsupervised representation learning of text have put little focus on discourse-level representations. |
| Approach: | They propose an inter-sentence objective for pretraining language models that models discourse coherence and the distance between sentences. |
| Outcome: | The proposed model outperforms the BERT-Large model on the discourse representation benchmark DiscoEval and yields gains of 2%-6% absolute even for tasks that do not explicitly evaluate discourse. |
SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features (2022.aacl-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (S3BERT) embeddings are composed of explainable sub-embeddings that emphasize various sentence meaning features. |
| Approach: | They propose to induce Semantically Structured Sentence BERT embeddings (S3BERT) that emphasize various sentence meaning features. |
| Outcome: | The proposed model shows high correlation to human similarity ratings, but lacks interpretability. |
AMR Beyond the Sentence: the Multi-sentence AMR corpus (C18-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is limited to capturing the semantics of individual sentences. |
| Approach: | They propose a corpus that annotates coreference and similar phenomena on top of existing AMRs. |
| Outcome: | The proposed corpus is compared with existing corpora on sentence-level semantics . it shows that it can be used for information extraction and question answering . |
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (P18-1)
Copied to clipboard
| Challenge: | a lack of understanding of the properties of sentence embeddings is limiting the use of the techniques. |
| Approach: | They propose 10 probing tasks designed to capture simple linguistic features of sentences . they use three different encoders to train embeddings in eight different ways . |
| Outcome: | The proposed tasks capture key linguistic features of sentences, but they are difficult to infer from them. |
DisSent: Learning Sentence Representations from Explicit Discourse Relations (P19-1)
Copied to clipboard
| Challenge: | Existing models train on vast amounts of text or require costly, manually curated datasets. |
| Approach: | They propose to leverage the discourse relations between sentences to curate a high quality sentence relation task by leveraging explicit discourse relations. |
| Outcome: | The proposed model can be used to learn the meaning of two sentences in a bidirectional LSTM sentence encoder. |