Less Mature is More Adaptable for Sentence-level Language Modeling (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies fine-tune encoders or contrastive learning approaches to learn sentence representations. |
| Approach: | They propose to use sentence-level models to study how sentence representations influence downstream task performance. |
| Outcome: | The proposed models outperform token-level models in terms of time and data efficiency. |
Similar Papers
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
What is the best recipe for character-level encoder-only modelling? (2023.acl-long)
Copied to clipboard
| Challenge: | aims to benchmark recent progress in language understanding models that output contextualised representations at the character level. |
| Approach: | They aim to find the best way to build and train character-level BERT-like models by comparing architectural innovations with pretraining objectives. |
| Outcome: | The proposed model outperforms a token-based model on a set of evaluation tasks with a fixed training procedure. |
SLM: Learning a Discourse Language Representation with Sentence Unshuffling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent models for learning discourse language representations focus on bottom or top-level representations, but they do not capture intermediate-size structures in natural languages such as sentences and the relationships among them. |
| Approach: | They propose a new objective for learning a discourse language representation in a self-supervised manner by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering. |
| Outcome: | The proposed model improves the original BERT model on downstream tasks by large margins. |
Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have limited generalization capabilities and performance challenges. |
| Approach: | They evaluate 15 different backbone LLMs and non-LLMs to evaluate their performance . larger models and extensive pre-training consistently enhance in-domain accuracy and data efficiency . |
| Outcome: | The results show that larger models and extensive pre-training enhance in-domain accuracy and data efficiency. |
Character-level Representations Improve DRS-based Semantic Parsing Even in the Age of BERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | a new method of analysis based on semantic tags demonstrates that character-level representations improve performance across a subset of selected semantic phenomena. |
| Approach: | They combine character-level and contextual language model representations to improve performance on Discourse Representation Structure parsing. |
| Outcome: | The proposed model improves performance on a subset of selected semantic phenomena. |
On the Sentence Embeddings from Pre-trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained contextual representations like BERT have been widely used for NLP tasks. |
| Approach: | They propose to transform anisotropic sentence embedding distribution to smooth and isotropic Gaussian distribution by normalizing flows that are learned with an unsupervised objective. |
| Outcome: | The proposed method achieves significant performance gains over state-of-the-art embeddings on a variety of semantic textual similarity tasks. |
Decoupling Generalization and Adaptation in Meta-Learning for Large Language Models (2026.acl-short)
Copied to clipboard
| Challenge: | Adapting large language models to specific downstream tasks requires multi-step fine-tuning with substantial training data, incurring significant computational overhead. |
| Approach: | They propose a framework that separates learning generalizable initializations and adaptation through dedicated parameter spaces. |
| Outcome: | The proposed framework outperforms existing meta-learning and standard multi-task baselines on common-sense reasoning, mathematics, logic, medical and coding benchmarks. |
Pretrained Language Models for Sequential Sentence Classification (D19-1)
Copied to clipboard
| Challenge: | Recent successful models for document-level understanding have used hierarchical encoding and CRFs to capture dependencies between subsequent labels. |
| Approach: | They propose a pretrained language model that captures contextual dependencies without hierarchical encoding nor a CRF. |
| Outcome: | The proposed model captures contextual dependencies without hierarchical encoding nor a CRF on four datasets, including a new dataset of structured scientific abstracts. |
BERT Has More to Offer: BERT Layers Combination Yields Better Sentence Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Obtaining sentence representations from BERT-based models is valuable as it takes less time to pre-compute a one-time representation of the data and then use it for the downstream tasks. |
| Approach: | They propose to combine certain layers of a BERT-based model rested on the data set and model to achieve substantially better results. |
| Outcome: | The proposed method outperforms baseline models on seven semantic textual similarity datasets and on eight transfer data sets. |
Augmenting BERT-style Models with Predictive Coding to Improve Discourse-level Representations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models do not produce suitable representations at the discourse level. |
| Approach: | They propose to augment BERT-style language models with a mechanism that allows them to learn suitable discourse-level representations by incorporating top-down connections that operate at the intermediate layers of the network. |
| Outcome: | The proposed approach improves in 6 out of 11 tasks by detecting discourse relationship detection. |