Papers by Stefan Larson
Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have not studied the performance of intent classifiers against hard-negative out-of-scope utterances. |
| Approach: | They propose to generate hard-negative OOS data using ChatGPT and evaluate them against three benchmark intent classifiers. |
| Outcome: | The proposed method improves classifiers' robustness against hard-negative out-of-scope utterances and general OOS data. |
An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (D19-1)
Copied to clipboard
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, Jason Mars
| Challenge: | Task-oriented dialog systems need to know when a query falls outside their range of supported intents. |
| Approach: | They propose a dataset that includes queries that are out-of-scope and 150 intent classes over 10 domains. |
| Outcome: | The proposed dataset includes queries that are out-of-scope, i.e., queries that do not fall into any of the system’s supported intents. |
Inconsistencies in Crowdsourced Slot-Filling Annotations: A Typology and Identification Methods (2020.coling-main)
Copied to clipboard
| Challenge: | Standard slot-filling models train or finetune on large datasets of carefully-annotated data that is domain specific. |
| Approach: | They propose automatic methods to identify inconsistencies in crowd-annotated data . a slot-filling model can extract the tokens "New York" as a TO LOCATION slot in a query . |
| Outcome: | The proposed methods reveal inconsistencies in data, though there is scope for improvement. |
Data Query Language and Corpus Tools for Slot-Filling and Intent Classification Data (2020.lrec-1)
Copied to clipboard
| Challenge: | Typical machine learning approaches require large amounts of training data . Managing training data can be cumbersome without dedicated tools . |
| Approach: | They propose a toolkit for analyzing slot-filling and intent classification corpora . they propose 'Query Language' for searching such corporan and tools for understanding structure . |
| Outcome: | The proposed toolkit can be used to uncover interesting and surprising insights. |
LSOIE: A Large-Scale Dataset for Supervised Open Information Extraction (2021.eacl-main)
Copied to clipboard
| Challenge: | Open Information Extraction (OIE) systems extract factual propositions into n-ary tuples . current datasets are limited in size and diversity . |
| Approach: | They propose to convert QA-SRL 2.0 dataset to large-scale OIE dataset LSOIE. |
| Outcome: | The proposed dataset is 20 times larger than the next largest human-annotated OIE dataset. |
Iterative Feature Mining for Constraint-Based Data Collection to Increase Data Diversity and Model Robustness (2020.emnlp-main)
Copied to clipboard
Stefan Larson, Anthony Zheng, Anish Mahendran, Rishi Tekriwal, Adrian Cheung, Eric Guldan, Kevin Leach, Jonathan K. Kummerfeld
| Challenge: | Recent work on dialog has found that crowdsourced data can have limited diversity as workers tend to write simple variations from prompts. |
| Approach: | They propose a general approach for guiding workers to write more diverse text by iteratively constraining their writing. |
| Outcome: | The proposed approach improves performance on dialog tasks and improves on existing datasets. |
Outlier Detection for Improved Data Quality and Diversity in Dialog Systems (N19-1)
Copied to clipboard
Stefan Larson, Anish Mahendran, Andrew Lee, Jonathan K. Kummerfeld, Parker Hill, Michael A. Laurenzano, Johann Hauswald, Lingjia Tang, Jason Mars
| Challenge: | Existing methods to detect outliers in text have been neglected in NLP . outlier detection is a problem in dialog systems where text is often no more than a few sentences in length. |
| Approach: | They propose a technique that uses sentence embeddings to detect outliers in short texts using neural sentence embeds and distance-based outlier detection. |
| Outcome: | The proposed technique detects outliers in a corpus of short texts while generating highly diverse corpora that produce more robust intent classification and slot-filling models. |
On Evaluation of Document Classification with RVL-CDIP (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing document classification benchmarks have label noise, ambiguous documents, and sensitive information. |
| Approach: | They argue that RVL-CDIP is unsuitable for benchmarking document classifiers . they advocate for a new document classification benchmark with ambiguous labels . |
| Outcome: | The RVL-CDIP benchmark is widely used for document classification . the authors argue that its limited scope, presence of errors and lack of diversity make it less than ideal for benchmarking. |
De-Identification of Sensitive Personal Data in Datasets Derived from IIT-CDIP (2024.emnlp-main)
Copied to clipboard
Stefan Larson, Nicole Lima, Santiago Diaz, Amogh Joshi, Siddharth Betala, Jamiu Suleiman, Yash Mathur, Kaushal Prajapati, Ramla Alakraa, Junjie Shen, Temi Okotore, Kevin Leach
| Challenge: | Large volumes of data are becoming increasingly important for training machine learning models for document understanding tasks like classification, information extraction, and visual question answering. |
| Approach: | They propose a data de-identification pipeline that replaces sensitive data with synthetic, but realistic, data that preserves the utility of de-identified documents. |
| Outcome: | The proposed method preserves the utility of the de-identified documents so that they can continue to be used in various document understanding applications. |