| Challenge: | Recent advances in sequence modeling have highlighted the strengths of the transformer architecture. |
| Approach: | They propose a general lattice transformer for speech translation where the input is the output of the automatic speech recognition (ASR) they propose 'controllable' lattica attention mechanism to consume latent representations. |
| Outcome: | The proposed model outperforms baseline and lattice LSTM on the Chinese-English translation task. |
Similar Papers
Lattice-Based Transformer Encoder for Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) takes deterministic sequences for source representations. However, word-level or subword-level segmentation has multiple choices to split a source sequence with different word segmentors or different subword vocabulary sizes. |
| Approach: | They propose lattice-based encoders to explore effective word or subword representations in an automatic way during training. |
| Outcome: | The proposed encoders can explore effective word or subword representation in an automatic way during training. |
Neural Speech Translation using Lattice Transformations and Graph Networks (D19-53)
Copied to clipboard
| Challenge: | Existing work on end-to-end systems bypass the need for intermediate representations, but this approach is limited in practical applications. |
| Approach: | They propose a lattice-tosequence model which uses lattics as encoders and graph networks to address two problems by applying latticae transformations and a neural model. |
| Outcome: | The proposed model beats pipeline approaches while being orders of magnitude faster than previous work. |
Speechformer: Reducing Information Loss in Direct Speech Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to speech-to-text translation (ST) use a pipeline of two sub-components - an automatic speech recognition (ASR) and a machine translation (MT) model. |
| Approach: | They propose an architecture that avoids initial lossy compression and aggregates information only at a higher level according to more informed linguistic criteria. |
| Outcome: | The proposed architecture achieves gains of up to 0.8 BLEU on the standard MuST-C corpus and up to 4.0 BLUE in a low resource scenario. |
Multiformer: A Head-Configurable Transformer-Based Model for Direct Speech Translation (2022.naacl-srw)
Copied to clipboard
| Challenge: | Existing approaches to address speech tasks with a self-attention mechanism are expensive and lead to information loss. |
| Approach: | They propose a Transformer-based model which uses different attention mechanisms on each head to bias the self-attention towards the extraction of more diverse token interactions. |
| Outcome: | The proposed model outperforms baseline models by 0.7 BLEU in the speech task. |
Character-Level Translation with Self-attention (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for character-level neural machine translation operate on word-level, which makes them memory inefficient because of large vocabulary sizes. |
| Approach: | They propose a transformer-based model and a novel variant that uses convolutions to combine information from nearby characters to facilitate character interactions. |
| Outcome: | The proposed model outperforms the standard transformer model and learns more robust character alignments on bilingual and multilingual translation datasets. |
Self-Attentional Models for Lattice Inputs (P19-1)
Copied to clipboard
| Challenge: | Existing work has extended recurrent neural networks to model lattice inputs but these models suffer from slow computation speeds. |
| Approach: | They propose to extend the paradigm of self-attention to handle lattice inputs by adding probabilistic reachability masks that incorporate latticae structure into the model and support lattics if available. |
| Outcome: | The proposed model outperforms baseline models while being much faster to compute than previous models. |
Chinese NER Using Lattice LSTM (P18-1)
Copied to clipboard
| Challenge: | Chinese named entity recognition (NER) is a fundamental task in information extraction. |
| Approach: | They propose a lattice-structured LSTM model for Chinese named entity recognition (NER) model leverages word and word sequence information to encode a sequence of input characters and all potential words that match a lexicon. |
| Outcome: | The proposed model outperforms word-based and character-based models on Chinese NER datasets. |
Improving the Transformer Translation Model with Document-Level Context (D18-1)
Copied to clipboard
| Challenge: | Existing models for document-level context translation ignore documentlevel context. |
| Approach: | They propose a document-level context encoder to represent document- level context and integrate it into the Transformer model. |
| Outcome: | Experiments on NIST Chinese-English and IWSLT French-English datasets show that the proposed translation model outperforms the Transformer model significantly. |
ETC: Encoding Long and Structured Inputs in Transformers (2020.emnlp-main)
Copied to clipboard
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, Li Yang
| Challenge: | Existing models for natural language processing (NLP) have been challenging to scale attention to longer inputs. |
| Approach: | They propose an extended Transformer construction architecture that scales attention to longer inputs by combining global-local attention with relative position encodings and a "Contrastive Predictive Coding" objective. |
| Outcome: | The proposed architecture scales attention to longer inputs and encodes structured inputs. |
Learning Source Phrase Representations for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to machine translation have been shown to be effective for long sentences . however, the attentional network can't capture long-distance dependencies . |
| Approach: | They propose a multi-head attention mechanism which generates phrase representations from token representations and incorporates them into the Transformer translation model to enhance its ability to capture long-distance relationships. |
| Outcome: | The proposed model can be computed in parallel and improves on the WMT 14 tasks. |