| Challenge: | Autoregressive (AR) encoder-decoder neural networks are slow in sequential prediction of natural language to machine-readable parse trees. |
| Approach: | They propose a technique that tokenizes a parse tree into subtrees and generates one subtrea per decoding step. |
| Outcome: | The proposed approach shows 4.6 times faster decoding speed and comparable speed but significantly higher accuracy compared to non-autoregressive (NAR) models. |
Similar Papers
Tree-Structured Non-Autoregressive Decoding for Sequence-to-Sequence Text Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Autoregressive Transformers suffer from high inference latency due to sequential token generation. |
| Approach: | They propose a tree-structured non-autoregressive decoding paradigm that bridges autoregressive and non-automatic decoding. |
| Outcome: | The proposed paradigm outperforms autoregressive and non-autoregressive decoding in machine translation and paraphrase generation. |
Insertion-based Tree Decoding (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing non-autoregressive decoders that are sub-linear can speed up inference for longer sequences. |
| Approach: | They propose a sub-linear nonautoregressive tree decoder that uses tree-based insertion operations to generate trees in sub-lines . they evaluate their approach on semantic parsing and compare it against strong baselines . |
| Outcome: | The proposed approach achieves competitive accuracies while reducing the number of decoding steps. |
Fast WordPiece Tokenization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for tokenization of text are not efficient, but they are based on Aho-Corasick's algorithm. |
| Approach: | They propose an efficient algorithm for WordPiece tokenization using a longest-match-first strategy . they propose an algorithm whose tokenization complexity is strictly O(n) |
| Outcome: | The proposed method is 8.2x faster than HuggingFace Tokenizers and 5.1x faster on average for general text tokenization. |
Non-Autoregressive Semantic Parsing for Compositional Task-Oriented Dialog (2021.naacl-main)
Copied to clipboard
| Challenge: | Semantic parsing using sequence-to-sequence models is stymied by higher compute requirements and higher latency. |
| Approach: | They propose a non-autoregressive approach to predict semantic parse trees with an efficient seq2seq model architecture. |
| Outcome: | The proposed architecture achieves an 81% reduction in latency on TOP dataset and retains competitive performance over non-pretrained models on three different semantic parsing datasets. |
Tetra-Tagging: Word-Synchronous Parsing with Linear-Time Inference (2020.acl-main)
Copied to clipboard
| Challenge: | Using custom architectures, constituency parsers are limited and require specialized hardware. |
| Approach: | They propose an algorithm that assigns labels to each word in a sentence in parallel and then performs a reconciliation phase to extract a tree in (empirically) linear time. |
| Outcome: | The proposed model achieves 95.4 F1 on the WSJ test set while also achieving substantial speedups compared to current state-of-the-art parsers with comparable accuracies. |
SmBoP: Semi-autoregressive Bottom-up Semantic Parsing (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing semantic parsers decode syntax using a top-down depth-first traversal. |
| Approach: | They propose a semi-autoregressive bottom-up parser that constructs at decoding step t the top-K sub-trees of height t. |
| Outcome: | The proposed method achieves 2.2x speed-up in decoding time and 5x speed up in training time on a zero-shot semantic parsing benchmark. |
Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsing (2021.findings-emnlp)
Copied to clipboard
Akshat Shrivastava, Pierce Chuang, Arun Babu, Shrey Desai, Abhinav Arora, Alexander Zotov, Ahmed Aly
| Challenge: | a novel approach to map utterances to semantic frames is based on non-autoregressive parsers that shift the decoding task from text generation to span prediction. |
| Approach: | They propose a non-autoregressive, task-oriented parser which shifts the decoding task from text generation to span prediction and produces endpoints as opposed to text. |
| Outcome: | The proposed model bridges the quality gap between non-autoregressive and autoregressive parsers, achieving 87 EM on TOPv2 and shows a 70% reduction in latency and 83% reduction in memory at beam size 5 compared to prior non-regressives. |
Enriched In-Order Linearization for Faster Sequence-to-Sequence Constituent Parsing (2020.acl-main)
Copied to clipboard
| Challenge: | Sequence-to-sequence constituent parsing requires a linearization to represent trees as sequences. Top-down tree linearizations have achieved the best accuracy to date. |
| Approach: | They propose to use an in-order shift-reduce linearization instead of a top-down tree linearization to represent trees as sequences. |
| Outcome: | The proposed approach achieves the best accuracy to date on the English PTB dataset among fully-supervised single-model sequence-to-sequence constituent parsers. |
Syntactically Supervised Transformers for Faster Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Standard decoders for neural machine translation generate a single token per timestep, which slows inference . a series of controlled experiments demonstrates that SynST decodes sentences 5x faster than the baseline autoregressive Transformer. |
| Approach: | They propose a syntactically supervised Transformer that generates all target tokens in one shot . synST is a variant of the Transformer architecture that autoregressively predicts a chunked parse tree . |
| Outcome: | The proposed method decodes sentences 5x faster than the baseline method on En-De and En-Fr datasets while achieving higher BLEU scores. |
SUBS: Subtree Substitution for Compositional Semantic Parsing (2022.naacl-main)
Copied to clipboard
| Challenge: | Semantic parsing models fail at compositional generalization due to lack of reasoning ability. |
| Approach: | They propose to use subtree substitution for compositional data augmentation to increase the number of subtreas with similar semantic functions as exchangeable. |
| Outcome: | The proposed method improves performance on Scan and GeoQuery, and new SOTA on compositional split of GeoQuery. |