Papers by Kevin Leach
One Agent To Rule Them All: Towards Multi-agent Conversational AI (2022.findings-acl)
Copied to clipboard
Christopher Clarke, Joseph Peper, Karthik Krishnamurthy, Walter Talamonti, Kevin Leach, Walter Lasecki, Yiping Kang, Lingjia Tang, Jason Mars
| Challenge: | Increasing volume of conversational agents (CAs) on the market has resulted in users being burdened with learning and adopting multiple agents to accomplish their tasks. |
| Approach: | They propose a task BBAI: Black-Box Agent Integration that integrates multiple black-box CAs at scale. |
| Outcome: | The proposed system outperforms existing benchmarks in the BBAI: Black-Box Agent Integration task. |
Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have not studied the performance of intent classifiers against hard-negative out-of-scope utterances. |
| Approach: | They propose to generate hard-negative OOS data using ChatGPT and evaluate them against three benchmark intent classifiers. |
| Outcome: | The proposed method improves classifiers' robustness against hard-negative out-of-scope utterances and general OOS data. |
An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (D19-1)
Copied to clipboard
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, Jason Mars
| Challenge: | Task-oriented dialog systems need to know when a query falls outside their range of supported intents. |
| Approach: | They propose a dataset that includes queries that are out-of-scope and 150 intent classes over 10 domains. |
| Outcome: | The proposed dataset includes queries that are out-of-scope, i.e., queries that do not fall into any of the system’s supported intents. |
Inconsistencies in Crowdsourced Slot-Filling Annotations: A Typology and Identification Methods (2020.coling-main)
Copied to clipboard
| Challenge: | Standard slot-filling models train or finetune on large datasets of carefully-annotated data that is domain specific. |
| Approach: | They propose automatic methods to identify inconsistencies in crowd-annotated data . a slot-filling model can extract the tokens "New York" as a TO LOCATION slot in a query . |
| Outcome: | The proposed methods reveal inconsistencies in data, though there is scope for improvement. |
Data Query Language and Corpus Tools for Slot-Filling and Intent Classification Data (2020.lrec-1)
Copied to clipboard
| Challenge: | Typical machine learning approaches require large amounts of training data . Managing training data can be cumbersome without dedicated tools . |
| Approach: | They propose a toolkit for analyzing slot-filling and intent classification corpora . they propose 'Query Language' for searching such corporan and tools for understanding structure . |
| Outcome: | The proposed toolkit can be used to uncover interesting and surprising insights. |
Iterative Feature Mining for Constraint-Based Data Collection to Increase Data Diversity and Model Robustness (2020.emnlp-main)
Copied to clipboard
Stefan Larson, Anthony Zheng, Anish Mahendran, Rishi Tekriwal, Adrian Cheung, Eric Guldan, Kevin Leach, Jonathan K. Kummerfeld
| Challenge: | Recent work on dialog has found that crowdsourced data can have limited diversity as workers tend to write simple variations from prompts. |
| Approach: | They propose a general approach for guiding workers to write more diverse text by iteratively constraining their writing. |
| Outcome: | The proposed approach improves performance on dialog tasks and improves on existing datasets. |
SynthFix: Adaptive Neuro-Symbolic Code Vulnerability Repair (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) struggle with complex semantic and structural correctness required for automated code repair. |
| Approach: | They propose a hybrid neural-symbolic framework that unifies code synthesis with compiler-informed symbolic feedback to improve LLM-based vulnerability repair. |
| Outcome: | The proposed framework improves code repair accuracy and efficiency over strong SFT and RFT training strategies on the FixJS and CodeFlaws benchmarks. |
On Evaluation of Document Classification with RVL-CDIP (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing document classification benchmarks have label noise, ambiguous documents, and sensitive information. |
| Approach: | They argue that RVL-CDIP is unsuitable for benchmarking document classifiers . they advocate for a new document classification benchmark with ambiguous labels . |
| Outcome: | The RVL-CDIP benchmark is widely used for document classification . the authors argue that its limited scope, presence of errors and lack of diversity make it less than ideal for benchmarking. |
De-Identification of Sensitive Personal Data in Datasets Derived from IIT-CDIP (2024.emnlp-main)
Copied to clipboard
Stefan Larson, Nicole Lima, Santiago Diaz, Amogh Joshi, Siddharth Betala, Jamiu Suleiman, Yash Mathur, Kaushal Prajapati, Ramla Alakraa, Junjie Shen, Temi Okotore, Kevin Leach
| Challenge: | Large volumes of data are becoming increasingly important for training machine learning models for document understanding tasks like classification, information extraction, and visual question answering. |
| Approach: | They propose a data de-identification pipeline that replaces sensitive data with synthetic, but realistic, data that preserves the utility of de-identified documents. |
| Outcome: | The proposed method preserves the utility of the de-identified documents so that they can continue to be used in various document understanding applications. |
EyeMulator: Improving Code Language Models by Mimicking Human Visual Attention (2026.acl-long)
Copied to clipboard
Yifan Zhang, Chen Huang, Yueke Zhang, Jiahao Zhang, Toby Jia-Jun Li, Collin McMillan, Kevin Leach, Yu Huang
| Challenge: | Code Language Models learn attention based on statistical input-output token correlations. |
| Approach: | They propose a model-agnostic technique to align CodeLLM attention with human visual attention without architectural changes. |
| Outcome: | The proposed model outperforms baselines in three languages, with gains of over 30 CodeBLEU points in translation and up to 22 BERTScore points in summarization. |