Analogy-Guided Evolutionary Pretraining of Binary Word Embeddings (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing binary word embeddings are derived from pretrained real-valued embeddables through different simple transformations, which often break the semantic consistency and the “arithmetic” properties of the embedded words. |
| Approach: | They propose a genetic algorithm to learn binary word embeddings from scratch by preserving the semantic relationships between words and the arithmetic properties of the embeddables themselves. |
| Outcome: | Evaluating 16, 32, and 64-bit word embeddings on Mikolov’s word analogy task shows that 95% of the time, the best fit is ranked in the top 5 most similar words in terms of cosine similarity. |
Similar Papers
Distilling Relation Embeddings from Pretrained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models capture a surprisingly rich amount of lexical knowledge, but it is unclear to what extent relation embeddings can be used to encode relational knowledge. |
| Approach: | They found that word vector differences capture lexical relations . relationship embeddings can be used to encode relational knowledge . |
| Outcome: | The results are highly competitive on analogy (unsupervised) and relation classification (supervised) benchmarks, even without any task-specific fine-tuning. |
Multilingual Culture-Independent Word Analogy Datasets (2020.lrec-1)
Copied to clipboard
| Challenge: | In text processing, deep neural networks use word embeddings as an input. |
| Approach: | They propose to use benchmark datasets to compare the quality of word embeddings in text processing . they use a word analogy task in Croatian, English, Estonian, Finnish, Latvian, Lithuanian, Russian, Slovenian, and Swedish . |
| Outcome: | The proposed datasets are culturally independent and cross-lingual for the languages used. |
MGAD: Multilingual Generation of Analogy Datasets (L18-1)
Copied to clipboard
| Challenge: | Existing methods for word embedding evaluation are computationally expensive and task-specific. |
| Approach: | They propose a minimally supervised method for generating word embedding evaluation datasets for a large number of languages using existing dependency treebanks and parsers. |
| Outcome: | The proposed method evaluates three popular word embedding algorithms against these datasets and shows that their performance varies between syntactic categories. |
Tiny Word Embeddings Using Globally Informed Reconstruction (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for word embedding reconstruction use only local information of subwords and pre-trained word embeds. |
| Approach: | They propose a global loss function that uses words other than the target word to improve word embedding reconstruction by a factor of 200. |
| Outcome: | The proposed method reduces the model size of pre-trained word embeddings by a factor of 200 while preserving its quality. |
Comparing Pretrained Multilingual Word Embeddings on an Ontology Alignment Task (L18-1)
Copied to clipboard
| Challenge: | Existing word embeddings capture a string's semantics and can be trained for multiple languages. |
| Approach: | They propose to compare three different multilingual pretrained word embedding repositories with a string-matching baseline and use it to compute semantic similarities of strings in different languages. |
| Outcome: | The proposed method produces correct alignments on a non-standard dataset on all four languages. |
Obtaining Better Static Word Embeddings Using Contextual Embedding Models (2021.acl-long)
Copied to clipboard
| Challenge: | Recent contextual word embeddings have prohibitively high computational cost in many use-cases and are hard to interpret. |
| Approach: | They propose a distillation method which is an extension of CBOW-based training and improves computational efficiency of NLP applications. |
| Outcome: | The proposed method outperforms existing models and existing models in terms of quality and performance. |
Quantifying Context Overlap for Training Word Embeddings (D18-1)
Copied to clipboard
| Challenge: | Experimental results show that word embeddings can be improved using word embeds . word embedings are a popular form of natural language processing . |
| Approach: | They propose to estimate second order co-occurrence relations based on context overlap . they use the augmented data to enhance word embeddings learning . |
| Outcome: | The proposed model improves word vectors for word similarity and downstream NLP tasks. |
A Retrofitting Model for Incorporating Semantic Relations into Word Embeddings (2020.coling-main)
Copied to clipboard
| Challenge: | Existing word embedding models mix semantic similarity with other types of relatedness. |
| Approach: | They propose a model that leverages relational knowledge available in a knowledge resource to improve word embeddings. |
| Outcome: | The proposed model improves word embeddings on synonymy, antonymy and hypernymy relations in WordNet and significantly improves lexical entailment detection task. |
Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation (P19-1)
Copied to clipboard
| Challenge: | Contextual embeddings address the problem of meaning conflation hampering word embeddables. |
| Approach: | They propose a method that creates sense-level embeddings with full-coverage of WordNet without recourse to explicit sense distributions or task-specific modelling. |
| Outcome: | The proposed method surpasses previous systems using powerful models and is robust when ignoring part-of-speech and lemma features. |
Methods for Numeracy-Preserving Word Embeddings (2020.emnlp-main)
Copied to clipboard
Dhanasekar Sundararaman, Shijing Si, Vivek Subramanian, Guoyin Wang, Devamanyu Hazarika, Lawrence Carin
| Challenge: | Word embedding models capture semantic relationships between words but fail to capture numerical properties associated with numbers. |
| Approach: | They propose a method to assign and learn embeddings for numbers using word embedders. |
| Outcome: | The proposed model outperforms pre-trained word embedding models across multiple examples of two tasks. |