Give your Text Representation Models some Love: the Case for Basque (2020.lrec-1)
Copied to clipboard
Rodrigo Agerri, Iñaki San Vicente, Jon Ander Campos, Ander Barrena, Xabier Saralegi, Aitor Soroa, Eneko Agirre
| Challenge: | Word embeddings and pre-trained language models are expensive to train and are often used by small companies and research groups to build their own. |
| Approach: | They propose to use word embeddings and pre-trained language models to build rich representations of text and improve NLP tasks. |
| Outcome: | The proposed models perform better than publicly available versions in downstream NLP tasks for Basque. |
Similar Papers
Go Simple and Pre-Train on Domain-Specific Corpora: On the Role of Training Data for Text Classification (2020.coling-main)
Copied to clipboard
| Challenge: | Pre-trained language models provide the foundations for state-of-the-art performance across a wide range of natural language processing tasks, including text classification. |
| Approach: | They compare the performance of a linear classifier based on word embeddings with a pre-trained language model, i.e., BERT, across a wide range of datasets and classification tasks. |
| Outcome: | The proposed method outperforms baselines in standard datasets with large training sets, but in settings with small training datasets it performs better. |
Advances in Pre-Training Distributed Word Representations (L18-1)
Copied to clipboard
| Challenge: | Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications. |
| Approach: | They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations. |
| Outcome: | The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data. |
Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? (2023.findings-eacl)
Copied to clipboard
| Challenge: | obtaining document embeddings at document level is challenging due to computational requirements and lack of appropriate data. |
| Approach: | They compare methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models. |
| Outcome: | The proposed methods produce document-level representations from sentences in 8 languages . the results show that a clever combination of sentence embeddings is usually better than encoding the full document as a single unit. |
Generating Datasets with Pretrained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to obtain high-quality sentence embeddings from pretrained language models require labeled data or finetuned on large set of labeles. |
| Approach: | They propose to use generative abilities of large and high-performing PLMs to generate entire datasets of labeled text pairs from scratch and fine tune much smaller and more efficient models. |
| Outcome: | The proposed approach outperforms baselines on several semantic textual similarity datasets. |
Fusing Label Embedding into BERT: An Efficient Improvement for Text Classification (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to improve text classification performance of pre-trained models have been used to improve their performance. |
| Approach: | They propose a method for improving BERT's performance by using a label embedding technique while keeping almost the same computational cost. |
| Outcome: | The proposed method improves BERT's performance on six text classification benchmark datasets while keeping almost the same computational cost. |
BERTrade: Using Contextual Embeddings to Parse Old French (2022.lrec-1)
Copied to clipboard
| Challenge: | a growing interest in digital humanities for automatic processing and annotation of historical texts is generating new models for historical languages. |
| Approach: | They use POS-tagging and dependency parsing to evaluate contextual word embedding models . Old French is one of the historical languages for which they have the largest amount of syntactically annotated data . |
| Outcome: | The proposed model can be used to improve performance in Old French, the authors show . they use POS-tagging and dependency parsing to evaluate the model's quality . |
Improving Text Embeddings with Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for obtaining text embeddings require complex training pipelines . authors leverage proprietary LLMs to generate diverse synthetic data for text embeds based on 93 languages . |
| Approach: | They propose a method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. |
| Outcome: | The proposed method achieves strong performance on competitive text embedding benchmarks without using any labeled data. |
Learning Word Vectors for 157 Languages (L18-1)
Copied to clipboard
| Challenge: | Distributed word representations, or word vectors, have been used in natural language processing for many tasks. |
| Approach: | They propose to use the encyclopedia Wikipedia and the common crawl corpus to train distributed word representations on large corpora and use them in downstream tasks. |
| Outcome: | The proposed model performs very well on 10 languages for which evaluation dataset exists. |
Leveraging Meta-Embeddings for Bilingual Lexicon Extraction from Specialized Comparable Corpora (C18-1)
Copied to clipboard
| Challenge: | Recent studies on bilingual lexicon extraction from specialized comparable corpora show differences in performance . lack of large specialized corporan to build efficient representations can be partially explained . |
| Approach: | They propose to use character-based embedding models to combine different embeddable models . they emphasize how character-driven embeddance models outperform other models on quality . |
| Outcome: | The proposed model outperforms other models on quality of extracted bilingual lexicons . comparable corpora are an interesting and practical alternative to parallel corporation . |
Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020.aacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have shown impressive results for cross-lingual transfer, but due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. |
| Approach: | They propose to transfer the knowledge from monolingual pretrained models to multilingual ones to improve zero-shot cross-lingual classification by using machine translation systems. |
| Outcome: | The proposed methods outperform vanilla multilingual fine-tuning on two cross-lingual classification benchmarks. |