Is Language Modeling Enough? Evaluating Effective Embedding Combinations (2020.lrec-1)
Copied to clipboard
Rudolf Schneider, Tom Oberhauser, Paul Grundmann, Felix Alexander Gers, Alexander Loeser, Steffen Staab
| Challenge: | specialized embeddings are not available for tasks like entity linking or paragraph classification. |
| Approach: | They evaluate whether universal embeddings can be complemented by specialized embeddables. |
| Outcome: | The proposed embeddings outperform state-of-the-art embeddables without any fine-tuning. |
Similar Papers
Embedding Strategies for Specialized Domains: Application to Clinical Entity Recognition (P19-2)
Copied to clipboard
| Challenge: | Off-the-shelf word embeddings tend to perform poorly on texts from specialized domains such as clinical reports. |
| Approach: | They combine off-the-shelf contextual embeddings with static word2vec embedders trained on a small in-domain corpus built from task data to reach and sometimes outperform representations learned from a large corpus in the medical domain. |
| Outcome: | The proposed embedding strategies outperform representations learned from a large corpus in the medical domain. |
Embeddings in Natural Language Processing (2020.coling-tutorials)
Copied to clipboard
| Challenge: | Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts . |
| Approach: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors . |
| Outcome: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations . |
Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? (2023.findings-eacl)
Copied to clipboard
| Challenge: | obtaining document embeddings at document level is challenging due to computational requirements and lack of appropriate data. |
| Approach: | They compare methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models. |
| Outcome: | The proposed methods produce document-level representations from sentences in 8 languages . the results show that a clever combination of sentence embeddings is usually better than encoding the full document as a single unit. |
A Deeper Look into Dependency-Based Word Embeddings (N18-4)
Copied to clipboard
| Challenge: | Word embeddings trained with dependency contexts excel at different tasks, and enhanced dependencies often improve performance. |
| Approach: | They propose to use dependency-based word embeddings to capture semantic similarity rather than relatedness. |
| Outcome: | The results show that word embeddings trained with Universal and Stanford dependencies excel at different tasks and that enhanced dependencies often improve performance. |
Leveraging Meta-Embeddings for Bilingual Lexicon Extraction from Specialized Comparable Corpora (C18-1)
Copied to clipboard
| Challenge: | Recent studies on bilingual lexicon extraction from specialized comparable corpora show differences in performance . lack of large specialized corporan to build efficient representations can be partially explained . |
| Approach: | They propose to use character-based embedding models to combine different embeddable models . they emphasize how character-driven embeddance models outperform other models on quality . |
| Outcome: | The proposed model outperforms other models on quality of extracted bilingual lexicons . comparable corpora are an interesting and practical alternative to parallel corporation . |
More Embeddings, Better Sequence Labelers? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work suggests contextual embeddings improve sequence labeling accuracy . but, there is no definite conclusion on whether concatenating different kinds of embeddables is effective . |
| Approach: | They propose a family of contextual embeddings that improves sequence labeling accuracy . they conduct extensive experiments on 3 tasks over 18 datasets and 8 languages . |
| Outcome: | The proposed family of contextual embeddings improves the accuracy of sequence labelers over non-contextual embedders. |
BioReddit: Word Embeddings for User-Generated Biomedical NLP (D19-62)
Copied to clipboard
| Challenge: | a corpus of medical-themed posts was scrapped from Reddit to train word embeddings on downstream tasks. |
| Approach: | They propose to train word embeddings from a corpus of medical forums from reddit scrapping posts from medical-themed subreddits. |
| Outcome: | The proposed system outperforms embeddings trained on general purpose data or on scientific papers when applied on user-generated content. |
Are Word Embeddings Really a Bad Fit for the Estimation of Thematic Fit? (2020.lrec-1)
Copied to clipboard
| Challenge: | In recent years, vectors derived from neural network training have replaced count-based distributional semantic models as a de facto standard for word representation in NLP. |
| Approach: | They propose to evaluate count models and word embeddings on thematic fit estimation by taking into account a larger number of parameters and verb roles and introducing dependency-based embedders in the comparison. |
| Outcome: | The proposed model outperforms count models and word embeddings in thematic fit estimation tasks while introducing dependency-based embedders. |
Frustratingly Easy Performance Improvements for Low-resource Setups: A Tale on BERT and Segment Embeddings (2022.lrec-1)
Copied to clipboard
| Challenge: | Understanding why contextualized embeddings work is still an active area of research. |
| Approach: | They propose to use a BERT architecture to encode a sub-word, position and a segment embedding as input representations for each sub- word. |
| Outcome: | The proposed model performs well on single-sentence prediction tasks while swapping segment IDs in paired-sentent tasks. |
Towards Unified Task Embeddings Across Multiple Models: Bridging the Gap for Prompt-Based Large Language Models and Beyond (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing task embedding methods rely on fine-tuned, task-specific language models, which hinders their adaptability to prompt-guided Large Language Models (LLMs). |
| Approach: | They propose a framework for unified task embedding that harmonizes task embeds from various models within a single vector space. |
| Outcome: | The proposed framework harmonizes task embeddings from various models within a single vector space. |