Explaining Character-Aware Neural Networks for Word-Level Prediction: Do They Discover Linguistic Rules? (D18-1)
Copied to clipboard
| Challenge: | Character-level features are used in many natural language processing algorithms but little is known about the character-level patterns they learn. |
| Approach: | They extend contextual decomposition technique to convolutional neural networks and bidirectional long-term memory networks to evaluate and compare these models for morphological tagging on three morphology-dependent languages. |
| Outcome: | The proposed models implicitly discover understandable linguistic rules for morphological tagging on three morphology-dependent languages. |
Similar Papers
What do character-level models learn about morphology? The case of dependency parsing (D18-1)
Copied to clipboard
| Challenge: | morphologically rich languages require character-level input models to learn morphology, but some models are poor at disambiguating some words . authors of this study show that character- level models learn a lot from input input . explicit modeling of morphologies is expensive and expensive, authors say . |
| Approach: | They compare character-level models to an oracle with explicit morphological analysis . they show that explicitly modeling morphology improves their best model . |
| Outcome: | The results show that character-level models learn morphology better than word models . the authors compare character-based models to oracles on 12 languages with morphological typologies . |
How Do Language Models Acquire Character-Level Information? (2026.eacl-long)
Copied to clipboard
| Challenge: | Language models (LMs) implicitly encode character-level information, despite not being explicitly provided during training. |
| Approach: | They analyze how language models acquire character-level knowledge by comparing them to standard settings. |
| Outcome: | The results show that LMs do not treat words as opaque tokens, but instead treat them as tokens. |
Learning to Discover, Ground and Use Words with Segmental Neural Language Models (P19-1)
Copied to clipboard
| Challenge: | Existing models of word learning do not account for the long-range dependencies manifest in language and that are easily captured by recurrent neural networks. |
| Approach: | They propose a segmental neural language model that unifies word discovery, learning how words fit together to form sentences, and by conditioning the model on visual context, how words’ meanings ground in representations of nonlinguistic modalities. |
| Outcome: | The proposed model learns predictive distributions better than character LSTM models, discovers words competitively with nonparametric Bayesian word segmentation models, and improves on both. |
What is the best recipe for character-level encoder-only modelling? (2023.acl-long)
Copied to clipboard
| Challenge: | aims to benchmark recent progress in language understanding models that output contextualised representations at the character level. |
| Approach: | They aim to find the best way to build and train character-level BERT-like models by comparing architectural innovations with pretraining objectives. |
| Outcome: | The proposed model outperforms a token-based model on a set of evaluation tasks with a fixed training procedure. |
Interpretability for Morphological Inflection: from Character-level Predictions to Subword-level Rules (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural models for morphological inflection have recently attained very high results, but their interpretation remains challenging. |
| Approach: | They propose a linguistically-motivated variant to the encoder-decoder model with attention that incorporates a character-level cross-attention mechanism and a self-attention module over substrings of the input. |
| Outcome: | The proposed model performs well on three typologically-different languages and is highly interpretable. |
What do tokens know about their characters and how do they know it? (2022.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models that use subword tokenization schemes can succeed at a variety of language tasks that require character-level information. |
| Approach: | They propose to use word tokenization schemes to probe what word pieces encode . they show that larger models can encode character-level information . |
| Outcome: | The proposed models can encode character-level information and perform better on non-Latin alphabets. |
Character-Level Translation with Self-attention (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for character-level neural machine translation operate on word-level, which makes them memory inefficient because of large vocabulary sizes. |
| Approach: | They propose a transformer-based model and a novel variant that uses convolutions to combine information from nearby characters to facilitate character interactions. |
| Outcome: | The proposed model outperforms the standard transformer model and learns more robust character alignments on bilingual and multilingual translation datasets. |
Character-Level Feature Extraction with Densely Connected Networks (C18-1)
Copied to clipboard
| Challenge: | Existing methods to generate character-level features with neural architectures such as CNN or Recurrent Neural Network (RNN) are slow and generate position-independent features. |
| Approach: | They propose a method that uses a densely connected network to extract character-level features from words using CNN and RNN. |
| Outcome: | The proposed method shows robustness and effectiveness while being faster than CNN- or RNN-based methods. |
Character-Level Models versus Morphology in Semantic Role Labeling (P18-1)
Copied to clipboard
| Challenge: | Character-level models are used for high-level semantic analysis tasks such as semantic role labeling. |
| Approach: | They train character-level models that use word, character and morphology level information . they analyze how performance of characters compare to words and a variety of morphological typologies . |
| Outcome: | The results shed light on important characteristics of character-level models and their semantic capability. |
Neural Factor Graph Models for Cross-lingual Morphological Tagging (P18-1)
Copied to clipboard
| Challenge: | Existing approaches to morphological tagging are limited by the assumption that tag sets overlap . a limited amount of data is available for most languages to learn these morphology taggers. |
| Approach: | They propose a method for cross-lingual morphological tagging that relaxes this assumption . they use factorial conditional random fields with neural network potentials to smooth over superficial differences in the surface forms . |
| Outcome: | The proposed model can smooth over superficial differences in the surface forms and generate unseen or rare tag sets. |