| Challenge: | Word composition models do not take ambiguity of words and context into consideration for learning representations, and thus suffer from the inaccurate representation of semantics. |
| Approach: | They propose a model-free context-aware word composition model which takes the latent semantic information as global context for learning representations. |
| Outcome: | The proposed model improves on existing composition models at different granularities and shows that it can be used to learn semantics. |
Similar Papers
Segmentation-free compositional n-gram embedding (N19-1)
Copied to clipboard
| Challenge: | Existing word embedding models depend on word segmentation, but this method is difficult when corpora written in noisy or unsegmented languages. |
| Approach: | They propose a new method that models words, phrases and sentences seamlessly without word segmentation. |
| Outcome: | The proposed method is very effective for noisy corpora written in unsegmented languages such as Chinese and Japanese. |
Grounded Compositional Outputs for Adaptive Language Modeling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Language models are a key component of natural language processing, but their size is a problem because they are typically trained with a closed output vocabulary derived from the training data. |
| Approach: | They propose a fully compositional output embedding layer for language models that is grounded in semantically related words and free-text definitions. |
| Outcome: | The proposed model outperforms state-of-the-art methods and adaptation approaches on cross-domain modeling and cross-learning tasks. |
Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations (D19-1)
Copied to clipboard
| Challenge: | Contextualized word representations are effective in downstream tasks such as question answering, named entity recognition, and sentiment analysis. |
| Approach: | They propose to integrate pre-trained contextualized word representations into a neural network that captures the whole sentence and the word representation in the sentence. |
| Outcome: | The proposed approach outperforms the state-of-the-art approach that makes use of non-contextualized word embeddings on multiple benchmark WSD datasets. |
A Systematic Study of Leveraging Subword Information for Learning Word Representations (N19-1)
Copied to clipboard
| Challenge: | Existing word representation models for morphologically rich languages use subword-level information, but their systematic comparative analysis across typologically diverse languages and tasks is still missing. |
| Approach: | They propose a framework for learning subword-informed word representations that allows for easy experimentation with different segmentation and composition components. |
| Outcome: | The proposed framework allows for easy experimentation with different segmentation and composition components, as well as advanced techniques based on position embeddings and self-attention. |
Are representations built from the ground up? An empirical examination of local composition in language models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Compositionality is a hallmark of human language, but many phrases are non-compositional . a study by a team of researchers shows that LMs may not be able to distinguish between compositional and non-composable phrases. |
| Approach: | They propose to predict LM-internal representations of longer phrases given their constituents . they find that the representation of a parent phrase can be predicted with some accuracy . |
| Outcome: | The proposed model can predict a parent phrase with some accuracy given its children's transformations, but this is not the case. |
An Unsupervised Method for Learning Representations of Multi-word Expressions for Semantic Classification (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for learning multi-word expressions have language sparsity and are not supervised. |
| Approach: | They propose an unsupervised approach to learning a compositional representation function for multi-word expressions . they use a Tratz dataset to train the composition function on the word-semantic relation . |
| Outcome: | The proposed method outperforms the previous state-of-the-art method on the Tratz dataset with an F1 score of 50.4%. |
Do Neural Language Models Inferentially Compose Concepts the Way Humans Can? (2024.lrec-main)
Copied to clipboard
| Challenge: | a new study shows that language models and humans may rely on different approaches to represent and compose lexical items across sentence structure. |
| Approach: | They propose to use a dataset to test the performance of neural language models and humans on inferentially driven conceptual compositions. |
| Outcome: | The proposed model elicits probability estimates for a noun in a minimally composed phrase . RoBERTa, BERT-large, and GPT-2 exhibited the closest resemblance to human responses . |
Enhancing the Context Representation in Similarity-based Word Sense Disambiguation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing similarity-based systems focus on learning sense embeddings using only the sentence where the word appears, neglecting its global context. |
| Approach: | They propose a contextoriented embedding technique that takes better advantage of both word-level and sense-level global context of an ambiguous word for disambiguation. |
| Outcome: | The proposed method improves on all-words WSD benchmarks in knowledge-based category by large margins. |
Sparsity Makes Sense: Word Sense Disambiguation Using Sparse Contextualized Word Representations (2020.emnlp-main)
Copied to clipboard
| Challenge: | Using sparse word embeddings is highly applicable for word sense disambiguation (WSD) . |
| Approach: | They propose an overcomplete set of semantic basis vectors that allows for sparse word representations. |
| Outcome: | The proposed framework achieves an aggregated F score of 78.8 over five standard word sense disambiguating benchmark datasets. |
Putting Words in Context: LSTM Language Models and Lexical Ambiguity (P19-1)
Copied to clipboard
| Challenge: | In language, a word can contribute a very different meaning depending on the context . lexical ambiguity involves both morphosyntactic and semantic aspects . |
| Approach: | They propose a method to probe hidden representations for lexical and contextual information about words. |
| Outcome: | The proposed method shows that both types of information are represented to a large extent, but there is room for improvement for contextual information. |