An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (D19-1)
Copied to clipboard
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, Jason Mars
| Challenge: | Task-oriented dialog systems need to know when a query falls outside their range of supported intents. |
| Approach: | They propose a dataset that includes queries that are out-of-scope and 150 intent classes over 10 domains. |
| Outcome: | The proposed dataset includes queries that are out-of-scope, i.e., queries that do not fall into any of the system’s supported intents. |
Similar Papers
Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have not studied the performance of intent classifiers against hard-negative out-of-scope utterances. |
| Approach: | They propose to generate hard-negative OOS data using ChatGPT and evaluate them against three benchmark intent classifiers. |
| Outcome: | The proposed method improves classifiers' robustness against hard-negative out-of-scope utterances and general OOS data. |
Class Name Guided Out-of-Scope Intent Classification (2024.findings-emnlp)
Copied to clipboard
Chandan Gautam, Sethupathy Parameswaran, Aditya Kane, Yuan Fang, Savitha Ramasamy, Suresh Sundaram, Sunil Sahu, Xiaoli Li
| Challenge: | SCOOS leverages semantic cues embedded in class labels to improve classification accuracy. |
| Approach: | They propose a method to create a compact feature space around class label semantics . they use a shared latent space between ID features and class names to minimize losses . |
| Outcome: | The proposed method outperforms existing methods for out-of-scope intent detection and ID intent classification. |
Benchmarking the Covariate Shift Robustness of Open-world Intent Classification Approaches (2022.aacl-short)
Copied to clipboard
| Challenge: | Existing benchmarks for open-world intent classification focus on the second aspect of the problem. |
| Approach: | They propose two datasets that include utterances useful for evaluating the robustness of open-world models to covariate shift. |
| Outcome: | The proposed datasets provide a more realistic evaluation scenario for open-world intent classification in task-oriented dialog systems. |
Towards Open Environment Intent Prediction (2023.findings-acl)
Copied to clipboard
| Challenge: | Out-of-Domain (OOD) Intent Classification and New Intent Discovering are two tasks in the Task-Oriented Dialogue System. |
| Approach: | They propose a task paradigm to extend Out-of-Domain (OOD) Intent Classification and New Intent Discovering tasks in the Task-Oriented Dialogue System. |
| Outcome: | The proposed scheme improves on existing OOD intent classification and discovery datasets. |
Benchmarking Commercial Intent Detection Services with Practice-Driven Evaluations (2021.naacl-industry)
Copied to clipboard
| Challenge: | Intent detection models require large amounts of labeled data to achieve high accuracy, and in practical scenarios it is more common to find small, unbalanced, and noisy datasets. |
| Approach: | They benchmark intent detection methods on a variety of datasets and found that Watson Assistant's model outperforms other commercial solutions. |
| Outcome: | The proposed model outperforms pretrained language models on a variety of datasets while requiring only a fraction of computational resources and training data. |
Out-of-Scope Intent Detection with Self-Supervision and Discriminative Training (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for out-of-scope intent detection rely on strong assumptions on data distribution and confidence threshold selection. |
| Approach: | They propose a method to train an out-of-scope intent classifier in a fully end-to-end manner by simulating the test scenario in training. |
| Outcome: | The proposed method improves on four benchmark dialogue datasets and improves over state-of-the-art methods. |
Navigating the Unknown: Intent Classification and Out-of-Distribution Detection Using Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Out-of-Distribution (OOD) detection requires great generalization capability . |
| Approach: | They propose a method that is cost-efficient, high-performing, highly robust and versatile enough to be used with smaller LLMs without sacrificing performance. |
| Outcome: | The proposed method is cost-efficient, high-performing, robust, and versatile enough to be used with smaller LLMs without sacrificing performance. |
Generalized Intent Discovery: Learning from Open World Dialogue System (2022.coling-1)
Copied to clipboard
| Challenge: | Existing intent classification models rely on a pre-defined intent set and supervised labels, which is limited in some practical scenarios. |
| Approach: | They propose to extend an IND intent classifier to an open-world intent set including IND and OOD intents. |
| Outcome: | The proposed task can classify IND and OOD intents while discovering new unlabeled OOD types incrementally. |
Improving Out-of-Scope Detection in Intent Classification by Using Embeddings of the Word Graph Space of the Classes (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for intent classification use one-class classification or inverse dictionary. |
| Approach: | They propose to represent class labels as a vector space where word graphs are mapped . they use inverse dictionary to take in account inter-class similarities provided by repeated occurrences . |
| Outcome: | The proposed method beats the state-of-the-art method in the Larson dataset by about 31 percentage points. |
Beyond the Known: Investigating LLMs Performance on Out-of-Domain Intent Detection (2024.lrec-main)
Copied to clipboard
Pei Wang, Keqing He, Yejie Wang, Xiaoshuai Song, Yutao Mou, Jingang Wang, Yunsen Xian, Xunliang Cai, Weiran Xu
| Challenge: | Out-of-domain (OOD) intent detection is crucial for task-oriented dialogue systems. |
| Approach: | They conduct a comprehensive evaluation of large language models (LLMs) under various experimental settings and outline their strengths and weaknesses. |
| Outcome: | The proposed models exhibit strong zero-shot and few-shot capabilities, but is still at a disadvantage compared to models fine-tuned with full resource. |