| Challenge: | Neural sequence-to-sequence models with soft attention outperform monotonic models . current dominant method is the neural sequenceto-Sequency model with soft focus . |
| Approach: | They develop a hard attention sequence-to-sequence model that enforces strict monotonicity and learns alignment jointly. |
| Outcome: | The proposed model achieves state-of-the-art on grapheme-to-phoneme conversion and morphological inflection generation. |
Similar Papers
Hard Non-Monotonic Attention for Character-Level Transduction (D18-1)
Copied to clipboard
| Challenge: | Character-level string-to-string transductions are an important component of NLP tasks . hard non-monotonic attention models have been used for sequence modeling tasks involving characters . |
| Approach: | They propose an exact algorithm for marginalizing over the exponential number of non-monotonic alignments between two strings. |
| Outcome: | The proposed algorithm outperforms soft attention and improves performance over stochastic approximation. |
On Biasing Transformer Attention Towards Monotonicity (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing work has focused on learning monotonic attention behavior via specialized attention functions or pretraining. |
| Approach: | They introduce a monotonicity loss function compatible with standard attention mechanisms and test it on sequence-to-sequence tasks. |
| Outcome: | The proposed monotonicity loss function can achieve largely monotonic behavior on grapheme-to-phoneme conversion, morphological inflection, transliteration, and dialect normalization tasks. |
Imitation Learning for Neural Morphological String Transduction (D18-1)
Copied to clipboard
| Challenge: | Recent studies have shown that neural transition-based models can be used for morphological tasks such as inflection generation and lemmatization without a character aligner or warm start. |
| Approach: | They propose to use imitation learning to train a neural transition-based string transducer for morphological tasks such as inflection generation and lemmatization. |
| Outcome: | The proposed model eliminates the need for a character aligner or warm start and achieves state-of-the-art performance on several datasets. |
Neural Transition-based String Transduction for Limited-Resource Setting in Morphology (C18-1)
Copied to clipboard
| Challenge: | Morphological string transduction involves mapping one word form into another, possibly given a feature specification for the mapping. |
| Approach: | They propose a neural transition-based model that uses a simple set of edit actions for morphological transduction tasks such as reinflection and reinflation. |
| Outcome: | The proposed model outperforms state-of-the-art systems on low and medium training-set sizes and is competitive in the high-resource setting. |
Applying the Transformer to Character-level Transduction (2021.eacl-main)
Copied to clipboard
| Challenge: | morphological inflection generation and historical text normalization tasks are character-level tasks that outperform recurrent models. |
| Approach: | They propose a technique to handle feature-guided character-level transduction that further improves performance. |
| Outcome: | The transformer outperforms recurrent models on morphological inflection and historical text normalization tasks. |
Pushing the Limits of Low-Resource Morphological Inflection (D19-1)
Copied to clipboard
| Challenge: | Recent advances in morphological inflection generation have limited resources . antonisa and colleagues present a battery of improvements to improve performance under low-resource conditions . |
| Approach: | They propose a two-step attention architecture for the inflection decoder that uses two-segments attention and a multi-single-syllabic attention architecture. |
| Outcome: | The proposed model outperforms the state-of-the-art in low-resource languages by 15 percentage points . the proposed model also shows that it can be used to model monolingual data hallucinations . |
Local String Transduction as Sequence Labeling (C18-1)
Copied to clipboard
| Challenge: | String transduction and sequence labeling are often treated as separate entities and often give treatment to different problems in NLP. |
| Approach: | They propose to reduce string transduction to sequence labeling by using a finite-state technique that uses string transducing and sequence labelling. |
| Outcome: | The proposed method performs better than seq2seq models and yields state-of-the-art results in several cases. |
An Empirical Study of Building a Strong Baseline for Constituency Parsing (P18-2)
Copied to clipboard
| Challenge: | Sequence-to-sequence models have been used for natural language generation tasks such as machine translation and summarization. |
| Approach: | They propose to build a strong baseline based on general purpose sequence-to-sequence models for constituency parsing. |
| Outcome: | The proposed model outperforms existing models in natural language generation tasks without any explicit task-specific knowledge or architecture of constituent parsing. |
Simulating Hard Attention Using Soft Attention (2026.tacl-1)
Copied to clipboard
| Challenge: | a central element of hard attention is attention, which computes a weighted average of values from all unmasked positions. |
| Approach: | They propose transformers that can simulate hard attention by using temperature scaling and positional embeddings. |
| Outcome: | The proposed transformers can effectively focus all attention on a subset of positions. |
Surprisingly Easy Hard-Attention for Sequence to Sequence Learning (D18-1)
Copied to clipboard
| Challenge: | Existing attention mechanisms are hard and hard, but they are more accurate when trained. |
| Approach: | They propose to use a beam approximation of the joint distribution between attention and output to train sequence to sequence learning. |
| Outcome: | The proposed method is compared to existing attention mechanisms on five translation tasks and shows consistent gains on the same tasks. |