| Challenge: | a gap exists between methods for learning representations of sentences and words . authors propose a convolutional neural architecture with no down-sampling for learning words based on character embeddings . |
| Approach: | They propose a funnel-shaped wide convolutional neural architecture with no down-sampling for learning words' internal structure. |
| Outcome: | The proposed model outperforms other character embedding models on six sequence labeling datasets. |
Similar Papers
Contextual String Embeddings for Sequence Labeling (C18-1)
Copied to clipboard
| Challenge: | Recent advances in language modeling have made it viable to model language as distributions over characters. |
| Approach: | They propose to leverage internal states of a trained character language model to produce a new type of word embeddings. |
| Outcome: | The proposed embeddings outperform the state-of-the-art on four classic sequence labeling tasks. |
Dissecting Contextual Word Embeddings: Architecture and Representation (D18-1)
Copied to clipboard
| Challenge: | Existing work on learning contextual representations has used LSTM-based biLMs, but there is no reason to believe this is effective. |
| Approach: | They propose to use pre-trained bidirectional language models to learn contextual word embeddings for four NLP tasks and to use them to study the effects of architecture on endtask accuracy. |
| Outcome: | The proposed models outperform word embeddings for four NLP tasks and all learn representations that vary with network depth. |
Bringing Emerging Architectures to Sequence Labeling in NLP (2026.eacl-long)
Copied to clipboard
| Challenge: | Pretrained Transformer encoders are the dominant approach to sequence labeling . however, few have been applied to sequence labels on flat or simplified tasks . |
| Approach: | They propose to use pretrained Transformer encoders to model relations across words . they find that the architectures adapt well across tagging tasks that vary in complexity . |
| Outcome: | The proposed architectures perform well across tagging tasks across languages and datasets. |
Pre-trained language model representations for language generation (N19-1)
Copied to clipboard
| Challenge: | Pre-trained language model representations have been successful in a wide range of language understanding tasks. |
| Approach: | They propose to use pre-trained language model representations to integrate them into sequence to sequence models and apply it to machine translation and abstractive summarization. |
| Outcome: | The proposed model is able to perform 5.3 BLEU in machine translation and 5.3 on the full text version of CNN/DailyMail. |
From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | Current models for natural language understanding require a preprocessing step to convert raw text into discrete tokens. |
| Approach: | They propose a hierarchical open-vocabulary language model that adopts a shallow Transformer architecture to learn word representations from their characters and a deep inter-word Transformer module that contextualizes each word representation by attending to the entire word sequence. |
| Outcome: | The proposed model outperforms baselines on various downstream tasks and is robust to textual corruption and domain shift. |
Learning to Generate Word Representations using Subword Information (C18-1)
Copied to clipboard
| Challenge: | Existing word-based approaches to learning word representations are blind to subword information in words. |
| Approach: | They propose a character-based word representation approach to learn word representations from characters. |
| Outcome: | The proposed model outperforms baseline models that regard words as atomic units . the proposed model achieves 18.5% improvement on average in perplexity for morphologically rich languages . |
Deep Contextualized Word Representations (N18-1)
Copied to clipboard
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer
| Challenge: | a new type of deep contextualized word representation is proposed for language understanding problems . word vectors are learned functions of the internal states of a deep bidirectional language model . |
| Approach: | They propose a new type of deep contextualized word representation that models complex features of word use and how they vary across linguistic contexts. |
| Outcome: | The proposed representations improve the state of the art across six challenging NLP problems. |
What is the best recipe for character-level encoder-only modelling? (2023.acl-long)
Copied to clipboard
| Challenge: | aims to benchmark recent progress in language understanding models that output contextualised representations at the character level. |
| Approach: | They aim to find the best way to build and train character-level BERT-like models by comparing architectural innovations with pretraining objectives. |
| Outcome: | The proposed model outperforms a token-based model on a set of evaluation tasks with a fixed training procedure. |
VCWE: Visual Character-Enhanced Word Embeddings (N19-1)
Copied to clipboard
| Challenge: | Currently, word embeddings are playing a pivotal role in many natural language processing tasks. |
| Approach: | They propose a model to learn Chinese word embeddings via three-level composition . they use convolutional neural network to extract intra-character compositionality from character shape . |
| Outcome: | The proposed model performs better on word similarity, sentiment analysis, named entity recognition and part-of-speech tagging tasks. |
Advances in Pre-Training Distributed Word Representations (L18-1)
Copied to clipboard
| Challenge: | Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications. |
| Approach: | They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations. |
| Outcome: | The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data. |