Papers with word2vec
Robust to Noise Models in Natural Language Processing Tasks (P19-2)
Copied to clipboard
| Challenge: | Existing spelling correction systems are far from perfect for noise-sensitive texts . a new way to handle noise is to make models robust to noise. |
| Approach: | They propose a robust to noise word embeddings model which outperforms existing models in different tasks. |
| Outcome: | The proposed model outperforms existing models in three downstream tasks and shows improvements in noise robustness over existing models. |
Generating Text through Adversarial Training Using Skip-Thought Vectors (N19-3)
Copied to clipboard
| Challenge: | Existing approaches to use word embeddings for text generation have been limited. |
| Approach: | They propose to use GANs with word embeddings to reproduce writing style in text . they use a sentence embeddable vector to model people's way of expression . |
| Outcome: | The proposed model outperforms baseline text generation networks across several metrics including BLEU-n, METEOR and ROUGE. |
Magnitude: A Fast, Efficient Universal Vector Embedding Utility Package (D18-2)
Copied to clipboard
| Challenge: | Magnitude is an open source Python package that performs common operations up to 6,000 times faster than Gensim. |
| Approach: | They present a Python tool for utilizing vector embeddings that performs common operations up to 6,000 times faster than Gensim. |
| Outcome: | The Magnitude package performs common operations up to 6,000 times faster than Gensim and introduces several novel features for improved robustness like out-of-vocabulary lookups. |
attr2vec: Jointly Learning Word and Contextual Attribute Embeddings with Factorization Machines (N18-1)
Copied to clipboard
| Challenge: | popular word embeddings are used to learn vector representations from the context of words. |
| Approach: | They propose a framework for jointly learning embeddings for words and contextual attributes based on factorization machines. |
| Outcome: | The proposed framework improves on a text classification task compared to learning embeddings independently. |
Embedding Syntax and Semantics of Prepositions via Tensor Decomposition (N18-1)
Copied to clipboard
| Challenge: | Existing methods on preposition representation treat prepositions no different from content words (e.g., word2vec and GloVe). |
| Approach: | They propose to use word-triple counts to capture a preposition’s interaction with its attachment and complement and derive preposition embeddings via tensor decomposition on a large unlabeled corpus. |
| Outcome: | The proposed model is comparable to or better than the state-of-the-art on multiple standardized datasets. |
GNEG: Graph-Based Negative Sampling for word2vec (P18-2)
Copied to clipboard
| Challenge: | Generally speaking, negative sampling is the best choice for distributed word representation learning. |
| Approach: | They hypothesize that taking into account global, corpus-level information and generating a different noise distribution for each target word better satisfies the requirements of negative examples for each training word. |
| Outcome: | The proposed approach boosts the word analogy task by about 5% and improves the performance on word similarity tasks by about 11% compared to the baseline. |
Word-Node2Vec: Improving Word Embedding with Document-Level Non-Local Word Co-occurrences (N19-1)
Copied to clipboard
| Challenge: | Existing word embedding algorithms make a strong assumption that words are semantically related only if they co-occur locally within a window of fixed size. |
| Approach: | They propose a graph-based word embedding method that relies on locality to capture the semantic association between words that co-occur frequently but non-locally within documents. |
| Outcome: | The proposed method outperforms word2vec and glove on a range of different tasks, such as predicting word-pair similarity, word analogy and concept categorization. |
comp-syn: Perceptually Grounded Word Embeddings with Color (2020.coling-main)
Copied to clipboard
Bhargav Srinivasa Desikan, Tasker Hull, Ethan Nadler, Douglas Guilbeault, Aabir Abubakar Kar, Mark Chu, Donald Ruggiero Lo Sardo
| Challenge: | Existing approaches to natural language processing ignore embodied sensory aspects of language. |
| Approach: | They propose a Python package that provides word embeddings based on Google Image search results. |
| Outcome: | The proposed package provides word embeddings based on the color distributions of Google Image search results. |
Paraphrases do not explain word analogies (2021.eacl-main)
Copied to clipboard
| Challenge: | Several attempts have been made to explain distributional word embeddings as linguistic regularities as directions. |
| Approach: | They propose to use an analogy to explain why linguistic regularities should hold in distributional word embeddings. |
| Outcome: | The proposed explanation does not hold empirically. |
An Evaluation Dataset for Identifying Communicative Functions of Sentences in English Scholarly Papers (2020.lrec-1)
Copied to clipboard
| Challenge: | Formulaic expressions are used by authors of scientific papers because they convey specific communicative functions in the rhetorical structure of papers. |
| Approach: | They created a manually annotated dataset to detect formulaic expressions in sentences using a seed list of labelled formulaic words. |
| Outcome: | The proposed dataset can detect communicative functions in sentences using a seed list of labelled expressions from scholarly papers in the ACL Anthology. |
Semi-Supervised Sequence Modeling with Cross-View Training (D18-1)
Copied to clipboard
| Challenge: | Unsupervised representation learning algorithms such as word2vec and ELMo only learn from task-specific labeled data during the main training phase. |
| Approach: | They propose a semi-supervised learning algorithm that improves the representations of a Bi-LSTM sentence encoder using a mix of labeled and unlabeled data. |
| Outcome: | The proposed algorithm improves the representations of a Bi-LSTM sentence encoder using a mix of labeled and unlabeled data. |
ANALOGICAL - A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models (2023.findings-acl)
Copied to clipboard
Thilini Wijesiriwardene, Ruwan Wickramarachchi, Bimal Gajera, Shreeyash Gowaikar, Chandan Gupta, Aman Chadha, Aishwarya Naresh Reganti, Amit Sheth, Amitava Das
| Challenge: | Modern large language models are evaluated on extrinsic measures based on benchmarks such as GLUE and SuperGLUE. |
| Approach: | They propose a benchmark to intrinsically evaluate large language models across a taxonomy of analogies of long text with six levels of complexity. |
| Outcome: | The proposed benchmark evaluates LLMs across a taxonomy of analogies of long text with six levels of complexity. |
Enhancing Out-of-Vocabulary Estimation with Subword Attention (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to learn OOV word representations use advanced architectures like attention on the context of the word, but they tend to use simple structures like ngram addition or character based convolutional neural networks (CNN) |
| Approach: | They propose a transformer-based OOV estimation model that uses attention mechanisms on both the context and the subwords to learn OOV representations. |
| Outcome: | The proposed model outperforms current state-of-the-art models on OOV representations based on attention mechanisms on the context and subwords . |
Subword-level Word Vector Representations for Korean (P18-1)
Copied to clipboard
| Challenge: | Existing research on word vectors for English focuses on decomposing words into subword units and using subwords to improve performance. |
| Approach: | They propose to decompose Korean words into the jamo-level, beyond the character-level . they develop Korean test sets for word similarity and analogy and make them publicly available . |
| Outcome: | The proposed method outperforms word2vec and character-level skip-grams on similarity and analogy tasks and contributes positively toward downstream NLP tasks such as sentiment analysis. |
Class-based LSTM Russian Language Model with Linguistic Information (2020.lrec-1)
Copied to clipboard
| Challenge: | LSTM models can be used in speech recognition systems at N-best or lattice rescoring stage. |
| Approach: | They propose to use word frequency and linguistic information to generate class-based LSTM Russian language models with various numbers of classes. |
| Outcome: | The proposed models outperform word-based models and word2vec models in terms of perplexity, training time, and word error rate. |
Specializing Word Vectors by Spectral Decomposition on Heterogeneously Twisted Graphs (2020.coling-main)
Copied to clipboard
| Challenge: | Word vectors have a tendency to conflate semantic similarity with semantic relatedness . a new method is proposed to retrofit word vectors with lexical constraints . |
| Approach: | They propose a method that heterogeneously retrofits a similarity matrix with lexical constraints. |
| Outcome: | The proposed method has a competitive performance compared with the state-of-the-art methods. |
Exploring a Choctaw Language Corpus with Word Vectors and Minimum Distance Length (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing tools to explore low resource languages that require no expert knowledge or substantial labor are limited. |
| Approach: | They introduce additions to the Choctaw corpus by using off-the-shelf tools word2vec and Linguistica to create new computational resources for the American indigenous language. |
| Outcome: | The proposed tools can be implemented with minimal labor in the American indigenous language Choctaw. |
Language in a (Search) Box: Grounding Language Learning in Real-World Human-Machine Interaction (2021.naacl-main)
Copied to clipboard
| Challenge: | Scholarly work in this area uses toy worlds and synthetic linguistic data, but grounded language learning offers several practical and scientific advantages. |
| Approach: | They propose to model teacher-learner dynamics through natural interactions occurring between users and search engines. |
| Outcome: | The proposed model is better than non-grounded models on compositionality and zero-shot inference tasks. |
Building a List of Synonymous Words and Phrases of Japanese Compound Verbs (L18-1)
Copied to clipboard
| Challenge: | Japanese is rich in compound verbs consisting of two verbs joined together. |
| Approach: | They built a database of Japanese "Verb + Verb" compounds semi-automatically . they extracted Japanese compound verbs from corpus and found suitable clusters . |
| Outcome: | The proposed database extracts synonymous expressions of Japanese compound verbs from corpus . it then links the results to the "Compound Verb Lexicon" |
Profiling of Intertextuality in Latin Literature Using Word Embeddings (2021.naacl-main)
Copied to clipboard
| Challenge: | a new study examines the intertextual relationships between authors in classical Latin literature . a large corpus of lemmatized Latin is used to train word embeddings . |
| Approach: | They propose to train an optimized word2vec model on a large corpus of Latin . they then replicate a previous study of the Roman historian Livy using hand-crafted stylometric features. |
| Outcome: | The proposed model outperforms a widely used lexical search method on Latin epic poetry . it advances the development of core computational resources for a major premodern language . |
Embeddings models for Buddhist Sanskrit (2022.lrec-1)
Copied to clipboard
| Challenge: | Despite extensive scholarly endeavors, much uncertainty still surrounds this body of literature, especially regarding matters of chronology, authorship, compositional history. |
| Approach: | They propose a corpus of Buddhist texts, a general corpus and word similarity and word analogy datasets for embeddings models. |
| Outcome: | The proposed models perform better on semantic similarity and word analogy tasks than on contextual models. |
Facilitating Corpus Usage: Making Icelandic Corpora More Accessible for Researchers and Language Users (2020.lrec-1)
Copied to clipboard
| Challenge: | Gigaword corpus is a large text corpus used in natural language processing . large corpora are needed to achieve better performance in the field of NLP . |
| Approach: | They propose a set of tools to facilitate the use of the Icelandic Gigaword Corpus . they provide n-grams based on the corpus, and a variety of pre-trained word embeddings models . |
| Outcome: | The proposed tools facilitate the use of the Icelandic Gigaword corpus in the field of Natural Language Processing and other fields. |
Dirichlet-Smoothed Word Embeddings for Low-Resource Settings (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing count-based word embeddings are superseded by machine-learning methods like word2vec and GloVe, but in many settings there is not much text data available. |
| Approach: | They propose to use positive pointwise mutual information (PPMI) weighted co-occurrence matrices to compute word embeddings from a corpus using large amounts of text data. |
| Outcome: | The proposed method outperforms word2vec and the state-of-the-art for low-resource settings and obtains competitive results for Maltese and Luxembourgish. |
Do NLP Models Know Numbers? Probing Numeracy in Embeddings (D19-1)
Copied to clipboard
| Challenge: | Existing models cannot capture numeracy, but they can be useful for complex reasoning tasks. |
| Approach: | They investigate numerical reasoning capabilities of a question-answering model . they probe token embedding methods on synthetic list maximum, number decoding, and addition tasks. |
| Outcome: | The proposed model excels on questions that require numerical reasoning, i.e., it already captures numeracy. |
Variants of Vector Space Reductions for Predicting the Compositionality of English Noun Compounds (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing approaches to predict the degree of compositionality of noun compounds are based on comparing compounds and their constituents within a vector space and using distributional similarity as a proxy to predict their degree of semantic relatedness. |
| Approach: | They propose to use distributional similarity as a proxy to predict the semantic relatedness between the compounds and their constituents as the compound’s degree of compositionality. |
| Outcome: | The proposed methods are most successful and stable in terms of dimensionality and part-of-speech reductions. |
Embedding Words in Non-Vector Space with Unsupervised Graph Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | GraphGlove is an unsupervised graph word representations that are learned end-to-end. |
| Approach: | They propose a method to learn weighted graph word representations end-to-end using a weighteable weighte . they adopt a hierarchical graph representation method and modify the GloVe training algorithm to learn graph representations. |
| Outcome: | The proposed method outperforms vector-based methods on word similarity and analogy tasks. |
Attention Word Embedding (2020.coling-main)
Copied to clipboard
| Challenge: | Word embedding models learn semantically rich vector representations of words . popular word embedders include word2vec, GloVe, and fastText . |
| Approach: | They propose an AWE-S model which integrates the attention mechanism into the CBOW model and incorporates subword information. |
| Outcome: | The proposed model outperforms the state-of-the-art model on word similarity datasets and when used for initialization of NLP models. |
Afaan Oromo Hate Speech Detection and Classification on Social Media (2022.lrec-1)
Copied to clipboard
| Challenge: | Hate and offensive speech on social media is a global problem that suffers the community especially, for an under-resourced language like Afaan Oromo. |
| Approach: | They develop a model to detect and classify Afaan Oromo hate speech on social media using different machine learning algorithms. |
| Outcome: | The proposed model outperforms existing models in gender, religion, race, and offensive speech on social media. |