Making a Point: Pointer-Generator Transformers for Disjoint Vocabularies (2020.aacl-srw)
Copied to clipboard
| Challenge: | Existing neural models rely on an overlap between source and target vocabularies to perform sequence-to-sequence tasks. |
| Approach: | They propose a pointer-generator transformer model for disjoint vocabularies that does not rely on an overlap between source and target vocs. |
| Outcome: | The proposed model outperforms a standard pointer-generator transformer by an average of 5.1 WER over 15 languages. |
Similar Papers
Optimizing Transformer for Low-Resource Neural Machine Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Language pairs with limited amounts of parallel data remain a challenge for neural machine translation. |
| Approach: | They propose to optimize a Transformer model for low-resource conditions to improve translation quality by 7.3 BLEU points compared to the default settings. |
| Outcome: | The proposed model improves translation quality up to 7.3 BLEU points compared to the default settings on the IWSLT14 training data compared with the Transformer model. |
EFTNAS: Searching for Efficient Language Models in First-Order Weight-Reordered Super-Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Depending on the size of transformer-based models, they can be restricted from deployment in resource-constrained environments. |
| Approach: | They propose to combine neural architecture search and network pruning techniques to generate and train weight-sharing super-networks that contain efficient transformer-based models. |
| Outcome: | The proposed model achieves high-performing, high-performance subnetworks on the general language understanding evaluation and the Stanford Question Answering Dataset. |
Applying the Transformer to Character-level Transduction (2021.eacl-main)
Copied to clipboard
| Challenge: | morphological inflection generation and historical text normalization tasks are character-level tasks that outperform recurrent models. |
| Approach: | They propose a technique to handle feature-guided character-level transduction that further improves performance. |
| Outcome: | The transformer outperforms recurrent models on morphological inflection and historical text normalization tasks. |
Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2020.tacl-1)
Copied to clipboard
| Challenge: | Unsupervised pre-training of large neural models has revolutionized Natural Language Processing. |
| Approach: | They propose to use pre-trained checkpoints for Sequence Generation to initialize a Transformer-based sequence-to-sequence model that is compatible with these checkpoint. |
| Outcome: | The proposed model is compatible with pre-trained BERT, GPT-2, and RoBERTa checkpoints and achieves state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentance Fusion. |
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2021.tacl-1)
Copied to clipboard
Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, Marianne Winslett
| Challenge: | Popular pre-trained Transformers have improved performance for various NLP tasks by sizable margins, but are too resource-hungry and computation-intensive to suit low-capacity devices or applications with strict latency requirements. |
| Approach: | They present a literature review of the compression of Transformers, focusing on the popular BERT model, which has attracted considerable research attention. |
| Outcome: | The proposed models improve Sentiment analysis, paraphrase detection, machine reading comprehension, question answering, text summarization, and other tasks by sizable margins. |
Rethinking Document-level Neural Machine Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Neural machine translation models are weak enough for document-level translation . current models only translate sentences individually, resulting in poor document coherence . |
| Approach: | They propose to use the original Transformer model to test document-level neural machine translation . they find that the original transformer models can achieve strong results for document translation if trained properly . |
| Outcome: | The proposed model outperforms sentence-level models on nine datasets and two sentence- level datasets across six languages. |
Low-resource Neural Machine Translation: Benchmarking State-of-the-art Transformer for Wolof<->French (2022.lrec-1)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) systems can translate between French (FR) 1 and Wolof (WO, ISO 639-3), a lowresource Niger-Congo language mainly spoken in Senegal (Gamble, 1950). |
| Approach: | They propose two neural machine translation systems based on sequence-to-sequence with attention and Transformer architectures to translate between French (FR) 1 and Wolof (WO, ISO 639-3). |
| Outcome: | The proposed models outperform the classic sequence-to-sequence model in all settings and are less sensitive to noise. |
Deep Copycat Networks for Text-to-Text Generation (D19-1)
Copied to clipboard
| Challenge: | Text-to-text generation tasks require copying words from the input to the output. |
| Approach: | They propose a transformer-based pointer network for text-to-text generation which generates more abstractive summaries and a further extension of this architecture for automatic post-editing. |
| Outcome: | The proposed model outperforms existing models in text-to-text generation tasks and improves translation accuracy. |
A Primer in BERTology: What We Know About How BERT Works (2020.tacl-1)
Copied to clipboard
| Challenge: | a new study examines the current state of knowledge about the BERT model . the model is a stack of transformer encoder layers that are based on multiple self-attention ''heads'' |
| Approach: | They present a survey of over 150 studies of the popular Transformer-based model BERT . they discuss the current state of knowledge about how BERT works and how it is represented . |
| Outcome: | The proposed model is based on the Transformer-based model with state-of-the-art results . the proposed model has little cognitive motivation and is too small to perform ablation studies . |
Choose Your Transformer: Improved Transferability Estimation of Transformer Models on Classification Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for NLP tasks require fine-tuning, but it is computationally infeasible. |
| Approach: | They propose an approach that inexpensively estimates a ranking of the expected performance of a given set of transformer language models for a specific task. |
| Outcome: | The proposed model improves the Pearson correlation coefficient between the true model ranks and the estimate. |