Papers by Tanay Dixit
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on counterfactual data augmentation only considered restricted classes of perturbations, limiting their effectiveness. |
| Approach: | They propose a retrieval-augmented framework for creating diverse counterfactual perturbations for CDA. |
| Outcome: | Experiments on natural language inference and sentiment analysis show that the proposed framework can be used to encourage diversity in manually authored perturbations. |
RETAIN: Interactive Tool for Regression Testing Guided LLM Migration (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly integrated into diverse applications. |
| Approach: | They propose a tool specifically designed for regression testing during LLM migrations. |
| Outcome: | RETAIN (REgression Testing guided LLM migrAtIoN) provides a tool specifically designed for regression testing during LLM migrations. |
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022.emnlp-main)
Copied to clipboard
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, Xudong Shen
| Challenge: | a benchmark of 1,616 diverse NLP tasks and their expert-written instructions is used to test generalization of models to unseen tasks . a recent study shows that instruction-following models outperform instruction-based models by over 9% . |
| Approach: | They build a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. |
| Outcome: | The proposed model outperforms existing instruction-following models by over 9% on the benchmark despite being smaller. |
Improving Factuality of Abstractive Summarization without Sacrificing Summary Quality (2023.acl-short)
Copied to clipboard
| Challenge: | Recent studies have shown that most abstractive summarization models are unfaithful and suffer from a wide range of hallucination. |
| Approach: | They propose a candidate summary generation and ranking technique to improve summary factuality without sacrificing quality. |
| Outcome: | The proposed method shows that the model trained using the proposed method improves on factuality and similarity-based metrics without conflicting with the model. |
Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for natural language generation are inadequate . existing metrics are not robust against simple perturbations and disagree with scores assigned by humans to perturbed output. |
| Approach: | They propose to propose checks which perturb the output and target a specific criteria and then use them to refine their evaluation. |
| Outcome: | The proposed templates show that existing evaluation metrics are not robust against simple perturbations and disagree with human scores on the perturbed output. |
IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation Metrics for Indian Languages (2023.acl-long)
Copied to clipboard
Ananya Sai B, Tanay Dixit, Vignesh Nagarajan, Anoop Kunchukuttan, Pratyush Kumar, Mitesh M. Khapra, Raj Dabre
| Challenge: | Recent studies on machine translation systems focus on high-resource languages, but focus has shifted to low-resourced languages. |
| Approach: | They evaluate 16 metrics from a multidimensional quality metric dataset . they show pre-trained metrics have higher correlations with annotator scores . |
| Outcome: | The proposed evaluations show that pre-trained metrics outperform COMET on Indian languages. |