| Challenge: | Existing word-based approaches to learning word representations are blind to subword information in words. |
| Approach: | They propose a character-based word representation approach to learn word representations from characters. |
| Outcome: | The proposed model outperforms baseline models that regard words as atomic units . the proposed model achieves 18.5% improvement on average in perplexity for morphologically rich languages . |
Similar Papers
Representation Learning for Unseen Words by Bridging Subwords to Semantic Networks (2020.lrec-1)
Copied to clipboard
| Challenge: | Pre-trained word embeddings only include words that appeared in corpora where pre-tried embedds are learned. |
| Approach: | They propose a method to represent out-of-vocabulary words using subword information and knowledge. |
| Outcome: | The proposed method improves performance over baselines that only use subwords or knowledge to represent OOV words. |
A Systematic Study of Leveraging Subword Information for Learning Word Representations (N19-1)
Copied to clipboard
| Challenge: | Existing word representation models for morphologically rich languages use subword-level information, but their systematic comparative analysis across typologically diverse languages and tasks is still missing. |
| Approach: | They propose a framework for learning subword-informed word representations that allows for easy experimentation with different segmentation and composition components. |
| Outcome: | The proposed framework allows for easy experimentation with different segmentation and composition components, as well as advanced techniques based on position embeddings and self-attention. |
Few-Shot Representation Learning for Out-Of-Vocabulary Words (P19-1)
Copied to clipboard
| Challenge: | Existing methods for learning word embedding assume there are enough occurrences for each word in the corpus to accurately estimate the representation of words. |
| Approach: | They propose to fit a representation function to predict an oracle embedding vector based on limited contexts. |
| Outcome: | The proposed model outperforms existing methods in constructing an accurate embedding for OOV words and improves downstream tasks when the embeddable is utilized. |
Enhancing Out-of-Vocabulary Estimation with Subword Attention (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to learn OOV word representations use advanced architectures like attention on the context of the word, but they tend to use simple structures like ngram addition or character based convolutional neural networks (CNN) |
| Approach: | They propose a transformer-based OOV estimation model that uses attention mechanisms on both the context and the subwords to learn OOV representations. |
| Outcome: | The proposed model outperforms current state-of-the-art models on OOV representations based on attention mechanisms on the context and subwords . |
Learning Better Internal Structure of Words for Sequence Labeling (D18-1)
Copied to clipboard
| Challenge: | a gap exists between methods for learning representations of sentences and words . authors propose a convolutional neural architecture with no down-sampling for learning words based on character embeddings . |
| Approach: | They propose a funnel-shaped wide convolutional neural architecture with no down-sampling for learning words' internal structure. |
| Outcome: | The proposed model outperforms other character embedding models on six sequence labeling datasets. |
Evaluating Sub-word Embeddings in Cross-lingual Models (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing approaches to learning sub-word embeddings for out-of-vocabulary words have not considered sub- word embedds in cross-lingual models. |
| Approach: | They propose to use sub-word embeddings to form cross-lingual embeddables for out-of-vocabulary (OOV) words for which no embeddibles are available. |
| Outcome: | The proposed bilingual lexicon induction task shows that sub-word embeddings can be leveraged to form cross-lingual embeddables for OOV words. |
Robust Backed-off Estimation of Out-of-Vocabulary Embeddings (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to solving out-of-vocabulary (OOV) words use subwords to represent oov words with a bag of subword. |
| Approach: | They propose a method to estimate oov word embeddings by referring to pre-trained word embeds for known words with similar surfaces to target ov words. |
| Outcome: | The proposed method improves word similarity tasks and biomedical tasks even with weak baselines. |
Subword-based Compact Reconstruction of Word Embeddings (N19-1)
Copied to clipboard
| Challenge: | Existing word-based word embeddings are based on subword information and memory-shared embeddables. |
| Approach: | They propose a method for reconstructing pre-trained word embeddings using subword information using memory-shared embedds and a variant of the key-value-query self-attention mechanism. |
| Outcome: | The proposed method can imitate well-trained word embeddings in a small fixed space while preventing quality degradation across several linguistic benchmark datasets. |
Improving Word Embeddings through Iterative Refinement of Word- and Character-level Models (2020.coling-main)
Copied to clipboard
| Challenge: | Embedding of rare and out-of-vocabulary words is an important open NLP problem . standard embedding models are not useful for recommending jobs to users with rare or unseen words . |
| Approach: | They propose to train a character-level neural network to reproduce word embeddings . they then use the model to assign vectors to any input string, including rare words . |
| Outcome: | The proposed method outperforms existing methods on word similarity data sets and can be applied to job title normalization in the e-recruitment domain. |
From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | Current models for natural language understanding require a preprocessing step to convert raw text into discrete tokens. |
| Approach: | They propose a hierarchical open-vocabulary language model that adopts a shallow Transformer architecture to learn word representations from their characters and a deep inter-word Transformer module that contextualizes each word representation by attending to the entire word sequence. |
| Outcome: | The proposed model outperforms baselines on various downstream tasks and is robust to textual corruption and domain shift. |