Papers by Akshat Shrivastava
Conversational Semantic Parsing (2020.emnlp-main)
Copied to clipboard
Armen Aghajanyan, Jean Maillard, Akshat Shrivastava, Keith Diedrick, Michael Haeger, Haoran Li, Yashar Mehdad, Veselin Stoyanov, Anuj Kumar, Mike Lewis, Sonal Gupta
| Challenge: | Structured representations for task-oriented assistant systems are limited due to the limitations of the representation. |
| Approach: | They propose a semantic representation for task-oriented conversational systems that can represent co-reference and context carryover. |
| Outcome: | The proposed model improves the best results on ATIS, SNIPS, TOP and DSTC2 by up to 5 points for slot-carryover. |
Non-Autoregressive Semantic Parsing for Compositional Task-Oriented Dialog (2021.naacl-main)
Copied to clipboard
| Challenge: | Semantic parsing using sequence-to-sequence models is stymied by higher compute requirements and higher latency. |
| Approach: | They propose a non-autoregressive approach to predict semantic parse trees with an efficient seq2seq model architecture. |
| Outcome: | The proposed architecture achieves an 81% reduction in latency on TOP dataset and retains competitive performance over non-pretrained models on three different semantic parsing datasets. |
Retrieve-and-Fill for Scenario-based Task-Oriented Semantic Parsing (2023.eacl-main)
Copied to clipboard
Akshat Shrivastava, Shrey Desai, Anchit Gupta, Ali Elkahky, Aleksandr Livshits, Alexander Zotov, Ahmed Aly
| Challenge: | Task-oriented semantic parsing models have achieved strong results in recent years, but they often face obstacles adapting to novel settings with distinct semantics and scarce data. |
| Approach: | They propose a scenario-based semantic parsing model which isolates coarse-grained and fine-grounded aspects of the task and solves them with off-the-shelf neural modules. |
| Outcome: | The proposed model outperforms previous approaches in high-resource, low-resourced, and multilingual settings, and is modular, differentiable, interpretable, and allows extra supervision from scenarios. |
Muppet: Massive Multi-task Representations with Pre-Finetuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows gains from pre-training and fine-tuning that are multi-task . but it can be difficult to know which intermediate tasks will best transfer . |
| Approach: | They propose a large-scale learning stage for pre-finetuning between pre-training and fine-tun. |
| Outcome: | The proposed model improves performance on pretrained discriminators and generation models on a wide range of tasks while improving sample efficiency during fine-tuning. |
Small But Funny: A Feedback-Driven Approach to Humor Distillation (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been used to transfer knowledge from LLMs to smaller, smaller language models (SLMs). |
| Approach: | They propose to assign a dual role to the LLM as a “teacher” generating data, as well as evaluating the student’s performance. |
| Outcome: | The proposed approach narrows the performance gap between LLMs and larger models by incorporating feedback into the data. |
Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsing (2021.findings-emnlp)
Copied to clipboard
Akshat Shrivastava, Pierce Chuang, Arun Babu, Shrey Desai, Abhinav Arora, Alexander Zotov, Ahmed Aly
| Challenge: | a novel approach to map utterances to semantic frames is based on non-autoregressive parsers that shift the decoding task from text generation to span prediction. |
| Approach: | They propose a non-autoregressive, task-oriented parser which shifts the decoding task from text generation to span prediction and produces endpoints as opposed to text. |
| Outcome: | The proposed model bridges the quality gap between non-autoregressive and autoregressive parsers, achieving 87 EM on TOPv2 and shows a 70% reduction in latency and 83% reduction in memory at beam size 5 compared to prior non-regressives. |
Introducing Semantics into Speech Encoders (2023.acl-long)
Copied to clipboard
Derek Xu, Shuyan Dong, Changhan Wang, Suyoun Kim, Zhaojiang Lin, Bing Liu, Akshat Shrivastava, Shang-Wen Li, Liang-Hsuan Tseng, Guan-Ting Lin, Alexei Baevski, Hung-yi Lee, Yizhou Sun, Wei Wang
| Challenge: | Existing self-supervised speech encoders contain primarily acoustic rather than semantic information. |
| Approach: | They propose a task-agnostic unsupervised way to incorporate semantic information from large language model (LLM) systems into self-supervised speech encoders without labeled audio transcriptions. |
| Outcome: | The proposed approach improves spoken language understanding (SLU) performance by over 5% on intent classification (IC), with modest gains in named entity resolution (NER) and slot filling (SF), and spoken question answering (SQA) score by over 22%. |
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (2024.acl-long)
Copied to clipboard
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, Carole-Jean Wu
| Challenge: | Large Language Models (LLMs) have been deployed to many applications, yet their high compute and memory requirements lead to high financial and energy costs when deployed to GPU servers. |
| Approach: | They propose an end-to-end solution to speed-up inference of large language models . they apply layer dropout, and show that it increases the accuracy of early exit at earlier layers without adding any auxiliary layers or modules to the model. |
| Outcome: | The proposed method shows speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing task. |
PRoDeliberation: Parallel Robust Deliberation for End-to-End Spoken Language Understanding (2024.findings-emnlp)
Copied to clipboard
Trang Le, Daniel Lazar, Suyoun Kim, Shan Jiang, Duc Le, Adithya Sagar, Aleksandr Livshits, Ahmed Aly, Akshat Shrivastava
| Challenge: | End-to-end models for Spoken Language Understanding have been autoregressive, resulting in higher latencies. |
| Approach: | They propose a method that uses Connectionist Temporal Classification to train robust non-autoregressive deliberation models. |
| Outcome: | The proposed method achieves 10x latency reduction over autoregressive models while preserving ability to correct ASR mistranscriptions. |
Treepiece: Faster Semantic Parsing via Tree Tokenization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Autoregressive (AR) encoder-decoder neural networks are slow in sequential prediction of natural language to machine-readable parse trees. |
| Approach: | They propose a technique that tokenizes a parse tree into subtrees and generates one subtrea per decoding step. |
| Outcome: | The proposed approach shows 4.6 times faster decoding speed and comparable speed but significantly higher accuracy compared to non-autoregressive (NAR) models. |