Card-660: Cambridge Rare Word Dataset - a Reliable Benchmark for Infrequent Word Representation Models (D18-1)
Copied to clipboard
| Challenge: | Existing benchmarks for rare word representation are lacking for evaluation and comparison . a task-based evaluation does not provide a solid basis for comparing different models . |
| Approach: | They propose to use an expert-annotated word similarity dataset to evaluate rare word representation techniques. |
| Outcome: | The proposed dataset provides a reliable benchmark for rare word representation techniques. |
Similar Papers
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations (N19-1)
Copied to clipboard
| Challenge: | Existing word embeddings cannot model the dynamic nature of words’ semantics, i.e., the property of words to correspond to potentially different meanings. |
| Approach: | They propose a large-scale Word in Context dataset, called WiC, which is curated by experts and can be used to evaluate context-sensitive representations. |
| Outcome: | The proposed models outperform the standard evaluation dataset for the purpose and highlight their shortcomings. |
Unsupervised Cross-Lingual Representation Learning (P19-4)
Copied to clipboard
| Challenge: | a comprehensive survey of cutting-edge weakly-supervised and unsupervised cross-lingual word representations is presented . |
| Approach: | This tutorial provides a comprehensive survey of recent work on weakly-supervised and unsupervised cross-lingual word representations. |
| Outcome: | This tutorial provides a comprehensive survey of cutting-edge weakly-supervised and unsupervised word representations. |
Dirichlet-Smoothed Word Embeddings for Low-Resource Settings (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing count-based word embeddings are superseded by machine-learning methods like word2vec and GloVe, but in many settings there is not much text data available. |
| Approach: | They propose to use positive pointwise mutual information (PPMI) weighted co-occurrence matrices to compute word embeddings from a corpus using large amounts of text data. |
| Outcome: | The proposed method outperforms word2vec and the state-of-the-art for low-resource settings and obtains competitive results for Maltese and Luxembourgish. |
Advances in Pre-Training Distributed Word Representations (L18-1)
Copied to clipboard
| Challenge: | Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications. |
| Approach: | They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations. |
| Outcome: | The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data. |
Probabilistic FastText for Multi-Sense Word Embeddings (P18-1)
Copied to clipboard
| Challenge: | Probabilistic FastText model for word embeddings captures word senses, sub-word structure, and uncertainty information. |
| Approach: | They propose a model for word embeddings that captures multiple word senses . they represent each word with a Gaussian mixture density, with each vector representing an n-gram . |
| Outcome: | The proposed model outperforms dictionary-level probabilistic embeddings on word-similarity benchmarks. |
A Short Survey on Sense-Annotated Corpora (2020.lrec-1)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a key task in Natural Language Understanding. |
| Approach: | They propose to use sense-annotated corpora for supervised Word Sense Disambiguation. |
| Outcome: | The proposed methods have been compared with knowledge-based approaches and have shown to be more efficient when they are available. |
Lacking the Embedding of a Word? Look it up into a Traditional Dictionary (2022.findings-acl)
Copied to clipboard
Elena Sofia Ruzzetti, Leonardo Ranaldi, Michele Mastromattei, Francesca Fallucchi, Noemi Scarpato, Fabio Massimo Zanzotto
| Challenge: | Word embeddings are powerful dictionaries, but they fail to give sense to rare words . a large body of research is devoted to devising ways to capture word meaning . |
| Approach: | They propose to use definitions retrieved from traditional dictionaries to build word embeddings for rare words. |
| Outcome: | The proposed methods outperform state-of-the-art methods for embeddings of unknown words . the proposed methods significantly outperformed the BERT method for OOV words compared to the proposed method . |
CoNLL#: Fine-grained Error Analysis and a Corrected Test Set for CoNLL-03 English (2024.lrec-main)
Copied to clipboard
| Challenge: | a glass ceiling for named entity recognition systems has been suggested for 2021 . however, the performance of the most popular NER benchmarks has plateaued since then . we investigate what NER models are still struggling with . |
| Approach: | They perform a fine-grained evaluation of the model outputs by adding document annotations to the CoNLL-03 English dataset to identify lingering errors. |
| Outcome: | The proposed model is able to correct errors and guide future work. |
Analyzing the Surprising Variability in Word Embedding Stability Across Languages (2021.emnlp-main)
Copied to clipboard
| Challenge: | Word embeddings are powerful representations that form the foundation of many natural language processing architectures. |
| Approach: | They explore word embedding stability in a wide range of languages to gain insight into their stability. |
| Outcome: | The proposed results provide insights into word embedding stability in English and other languages. |
AttributionBench: How Hard is Automatic Attribution Evaluation? (2024.findings-acl)
Copied to clipboard
| Challenge: | generative search engines enhance the reliability of large language model responses by providing cited evidence. |
| Approach: | They propose to use a benchmark to evaluate whether a large language model supports the generated responses or not . |
| Outcome: | The proposed benchmark shows that even a fine-tuned GPT-3.5 only achieves around 80% macro-F1 under a binary classification formulation. |