Papers by Ayush Kumar
Spot the BlindSpots: Systematic Identification and Quantification of Fine-Grained LLM Biases in Contact Center Call Summarization (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Abstractive summarization is a core application in contact centers, where Large Language Models generate millions of summaries of call transcripts daily. |
| Approach: | They propose a framework that uses an LLM as a zero-shot classifier to derive categorical distributions for each bias dimension in a pair of transcripts and its summary. |
| Outcome: | The proposed framework identifies and quantifies 15 operational bias dimensions and measures them using two metrics: Fidelity Gap and Coverage. |
The Bull and the Bear: Summarizing Stock Market Discussions (2022.lrec-1)
Copied to clipboard
| Challenge: | a dataset of 7888 reddit posts and 400 posts is used to summarize stock market topics. |
| Approach: | They curate discussions on social media platforms and construct an abstractive summarization dataset. |
| Outcome: | The proposed dataset consists of 7888 Reddit posts and summaries for 400 posts . it is robustly evaluated and will be made publicly available . |
Investigating the Role and Impact of Disfluency on Summarization (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Existing studies have focused on disfluency detection and removal, with limited studies into its impact on downstream tasks. |
| Approach: | They propose to incorporate disfluency in summarization models to reduce the impact of replacement disfluencies on natural language processing tasks. |
| Outcome: | The proposed model improves on both public and real-life datasets and shows that it can handle disfluent data with up to 6.99-point degradation in Rouge-L score and replacement disfluencies have the highest negative impact. |
Mirage: A Diagnostic Framework for Evaluating the Realism of Synthetic Contact Center Dialogue Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | capturing the full realism of real agent–customer interactions remains a challenge . privacy constraints and data scarcity limit the availability of real conversations . |
| Approach: | They propose a diagnostic evaluation framework for synthetic dialogue generation . they benchmarked strategies guided by structured supervision on call attributes . |
| Outcome: | The proposed framework outperforms synthetic transcripts on quality assurance tasks . it shows that current generation strategies lack sentiment fidelity, disfluency modeling, behavioral variation and conversational realism . |
Entity Resolution and Location Disambiguation in the Ancient Hindu Temples Domain using Web Data (N18-5)
Copied to clipboard
| Challenge: | Existing systems for resolving entities and disambiguating locations based on publicly available web data are challenging because of the limited information available on the Web. |
| Approach: | They propose a system for resolving entities and disambiguating locations based on publicly available web data in the domain of ancient Hindu Temples. |
| Outcome: | The proposed system resolves entities and disambiguates locations with high confidence using grammar rules and clustering algorithms. |
Substance over Style: Evaluating Proactive Conversational Coaching Agents (2025.acl-long)
Copied to clipboard
Vidya Srinivas, Xuhai Xu, Xin Liu, Kumar Ayush, Isaac Galatzer-Levy, Shwetak Patel, Daniel McDuff, Tim Althoff
| Challenge: | Recent NLP research has focused on single-turn tasks with well-defined objectives or evaluation criteria. |
| Approach: | They describe five multi-turn coaching agents that exhibit distinct conversational styles and evaluate them through a user study. |
| Outcome: | The authors compare user feedback with third-person evaluations from health experts and an LM to find that stylistic components in absence of core functionality are viewed negatively. |
Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly deployed in contact-center Quality Assurance (QA) relying on web-scale training data raises concerns regarding demographic and behavioral biases that may distort workforce assessment. |
| Approach: | They propose to evaluate the fairness of large language models on 3,000 real-world transcripts . they find systematic disparities in the CFR and MASD across different dimensions . |
| Outcome: | Evaluating 18 LLMs on 3,000 real-world contact center transcripts, they find systematic disparities . larger, more strongly aligned models show lower unfairness, though fairness does not track accuracy. |
Entity Exchange in the Wild: A Diagnostic Study of LLM Based Real-World Conversational Entity Extraction (2026.acl-industry)
Copied to clipboard
| Challenge: | Prior work has examined the impact of transcription noise and cross-turn reasoning, but it has not systematically analyzed how entity-exchange phenomena themselves shape extraction performance. |
| Approach: | They evaluate 16 large language models on 6,387 real-world customer–agent conversations spanning 12 entity types across numeric, alphanumeric, temporal, and free-text categories. |
| Outcome: | The proposed model improves on the extracted entities across all three axes yielding average gains of up to 6.4% across models. |