| Challenge: | Generally speaking, negative sampling is the best choice for distributed word representation learning. |
| Approach: | They hypothesize that taking into account global, corpus-level information and generating a different noise distribution for each target word better satisfies the requirements of negative examples for each training word. |
| Outcome: | The proposed approach boosts the word analogy task by about 5% and improves the performance on word similarity tasks by about 11% compared to the baseline. |
Similar Papers
Batch IS NOT Heavy: Learning Word Representations From All Samples (P18-1)
Copied to clipboard
| Challenge: | Stochastic Gradient Descent with negative sampling is the most prevalent approach to learn word representations. |
| Approach: | They propose a method that uses batch gradient learning to generate word representations from all training samples. |
| Outcome: | The proposed method outperforms sampling-based methods on several benchmark tasks. |
Word-Node2Vec: Improving Word Embedding with Document-Level Non-Local Word Co-occurrences (N19-1)
Copied to clipboard
| Challenge: | Existing word embedding algorithms make a strong assumption that words are semantically related only if they co-occur locally within a window of fixed size. |
| Approach: | They propose a graph-based word embedding method that relies on locality to capture the semantic association between words that co-occur frequently but non-locally within documents. |
| Outcome: | The proposed method outperforms word2vec and glove on a range of different tasks, such as predicting word-pair similarity, word analogy and concept categorization. |
Rethinking Negative Sampling for Handling Missing Entity Annotations (2022.acl-long)
Copied to clipboard
| Challenge: | Empirical studies show low missampling rate and high uncertainty are both essential for achieving promising performances with negative sampling. |
| Approach: | They propose an adaptive and weighted sampling distribution that further improves negative sampling by introducing missampling and uncertainty concepts. |
| Outcome: | The proposed approach improves on synthetic and well-annotated datasets in terms of F1 score and loss convergence. |
Negative Sampling Techniques in Dense Retrieval: A Survey (2026.findings-eacl)
Copied to clipboard
| Challenge: | Information Retrieval (IR) is fundamental to many modern NLP applications. |
| Approach: | They propose a taxonomy that categorizes negative sampling techniques in dense IR . they analyze them with respect to trade-offs between effectiveness, computational cost, implementation difficulty . |
| Outcome: | The proposed taxonomy categorizes techniques using random, static/dynamically mined, and synthetic datasets. |
Structure Aware Negative Sampling in Knowledge Graphs (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for learning low-dimensional representations of entities and relations in knowledge graphs employing corruption distributions that generate hard negative samples. |
| Approach: | They propose a structure-aware negative sampling strategy that utilizes the rich graph structure by selecting negative samples from a node’s k-hop neighborhood. |
| Outcome: | The proposed method finds semantically meaningful negatives and is competitive with SOTA approaches while requires no additional parameters nor difficult adversarial optimization. |
Sampling Matters! An Empirical Study of Negative Sampling Strategies for Learning of Matching Models in Retrieval-based Dialogue Systems (D19-1)
Copied to clipboard
| Challenge: | Existing studies focus on constructing a matching model with sophisticated neural architectures, but do little to how to effectively learn such architectures from data. |
| Approach: | They propose to sample negative examples to automatically construct a training set for effective model learning in retrieval-based dialogue systems by using four sampling strategies. |
| Outcome: | The proposed learning method improves the performance of matching models on two benchmarks with three matching models. |
Clustering-Aware Negative Sampling for Unsupervised Sentence Representation (2023.findings-acl)
Copied to clipboard
| Challenge: | Using clustering-aware learning, in-batch negatives are often ignored in sentence representation learning. |
| Approach: | They propose a method that integrates cluster information into contrastive learning for unsupervised sentence representation learning. |
| Outcome: | The proposed method compares favorably with baselines on semantic textual similarity tasks. |
Don’t Mess with Mister-in-Between: Improved Negative Search for Knowledge Graph Completion (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for knowledge graph completion use a dual-encoding framework with a bottleneck that allows for fast approximate search over a vast collection of candidates. |
| Approach: | They propose to use a dual-encoder framework to find more informative negatives by searching for candidates with high lexical overlaps. |
| Outcome: | The proposed methods improve on the large-scale Wikidata5M dataset and combine different kinds of strategies to achieve state-of-the-art performance. |
Pneg: Prompt-based Negative Response Generation for Dialogue Response Selection Task (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for synthesizing adversarial negative responses are limited by their scalability and cost. |
| Approach: | They propose a method for generating adversarial negative responses using a large-scale language model. |
| Outcome: | The proposed method outperforms other methods on dialogue selection tasks. |
NegatER: Unsupervised Discovery of Negatives in Commonsense Knowledge Bases (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in knowledge base construction techniques focus on the acquisition of positive (true) KB statements, but negative (false) statements are important for discriminative reasoning. |
| Approach: | They propose a framework that ranks potential negatives in commonsense KBs using a contextual language model. |
| Outcome: | The proposed framework ranks negatives in commonsense KBs using a language model . it yields positives that are more grammatical, coherent, and informative . |