Papers by Pradeep Natarajan
Error Detection in Large-Scale Natural Language Understanding Systems Using Transformer Models (2021.findings-acl)
Copied to clipboard
| Challenge: | Large-scale conversational assistants process every utterance using multiple models for domain, intent and named entity recognition. |
| Approach: | They combine utterance encodings from a RoBERTa model with the Nbest hypothesis produced by the production system to detect domain classification errors. |
| Outcome: | The proposed approach outperforms bi-LSTM models and a standalone model by 2.2% to 32.2% by ensembling multiple models. |
FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained models need fine-tuning on tens of thousands of examples to achieve good results. |
| Approach: | They propose a framework that leverages pre-trained text-to-text models and aligns them with their pre-training framework. |
| Outcome: | The proposed framework outperforms the XLM-Roberta-large on multiple QA benchmarks and is applicable to multilingual situations. |
FPI: Failure Point Isolation in Large-scale Conversational Assistants (2022.naacl-industry)
Copied to clipboard
| Challenge: | Large-scale conversational assistants can cause errors in their modules . a machine learning system can analyze large volumes of data and isolate the source of error . |
| Approach: | They propose a machine learning system that embeds incoming request and context using pre-trained transformer models and encodes additional metadata features to output failure point predictions. |
| Outcome: | The proposed system obtains 92.2% of human performance while scaling to analyze the entire traffic in 8 different languages of a large-scale conversational assistant. |
CGF: Constrained Generation Framework for Query Rewriting in Conversational AI (2022.emnlp-industry)
Copied to clipboard
Jie Hao, Yang Liu, Xing Fan, Saurabh Gupta, Saleh Soltan, Rakesh Chada, Pradeep Natarajan, Chenlei Guo, Gokhan Tur
| Challenge: | Large-scale conversational AI agents such as Alexa, Siri and Google Assistant help millions of users to perform a lot of tasks. |
| Approach: | They propose a Constrained Generation Framework for query rewriting at global and personalized levels. |
| Outcome: | The proposed framework significantly boosts the query rewriting performance. |
Adversarial Robustness for Large Language NER models using Disentanglement and Word Attributions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) tasks are becoming more challenging due to the introduction of complex tagsets, which often leads to the failure of existing NER systems in accurately recognizing these entities. |
| Approach: | They propose a novel attack which relies on disentanglement and word attribution techniques to learn an embedding and identifying important words across both components. |
| Outcome: | The proposed approach improves the F1 score over the original LLM model by 8% and 18% on CoNLL-2003 and Ontonotes 5.0 datasets respectively. |
Improving Large-Scale Conversational Assistants using Model Interpretation based Training Sample Selection (2022.emnlp-industry)
Copied to clipboard
Stefan Schroedl, Manoj Kumar, Kiana Hajebi, Morteza Ziyadi, Sriram Venkatapathy, Anil Ramakrishna, Rahul Gupta, Pradeep Natarajan
| Challenge: | Large-scale, voice-based conversational assistants process each utterance through a multi-stage pipeline that includes wakeword detection, automatic speech recognition (ASR), natural language understanding (NLU), entity resolution, and textto-speech. |
| Approach: | They propose a method to identify customer implicitly satisfied with Alexa's responses by leveraging interpretations of model behavior. |
| Outcome: | The proposed approach produces statistically significant improvements in both offline and online tests. |