Benchmarking the Covariate Shift Robustness of Open-world Intent Classification Approaches (2022.aacl-short)
Copied to clipboard
| Challenge: | Existing benchmarks for open-world intent classification focus on the second aspect of the problem. |
| Approach: | They propose two datasets that include utterances useful for evaluating the robustness of open-world models to covariate shift. |
| Outcome: | The proposed datasets provide a more realistic evaluation scenario for open-world intent classification in task-oriented dialog systems. |
Similar Papers
An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (D19-1)
Copied to clipboard
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, Jason Mars
| Challenge: | Task-oriented dialog systems need to know when a query falls outside their range of supported intents. |
| Approach: | They propose a dataset that includes queries that are out-of-scope and 150 intent classes over 10 domains. |
| Outcome: | The proposed dataset includes queries that are out-of-scope, i.e., queries that do not fall into any of the system’s supported intents. |
Benchmarking Commercial Intent Detection Services with Practice-Driven Evaluations (2021.naacl-industry)
Copied to clipboard
| Challenge: | Intent detection models require large amounts of labeled data to achieve high accuracy, and in practical scenarios it is more common to find small, unbalanced, and noisy datasets. |
| Approach: | They benchmark intent detection methods on a variety of datasets and found that Watson Assistant's model outperforms other commercial solutions. |
| Outcome: | The proposed model outperforms pretrained language models on a variety of datasets while requiring only a fraction of computational resources and training data. |
Generalized Intent Discovery: Learning from Open World Dialogue System (2022.coling-1)
Copied to clipboard
| Challenge: | Existing intent classification models rely on a pre-defined intent set and supervised labels, which is limited in some practical scenarios. |
| Approach: | They propose to extend an IND intent classifier to an open-world intent set including IND and OOD intents. |
| Outcome: | The proposed task can classify IND and OOD intents while discovering new unlabeled OOD types incrementally. |
Out-of-Scope Intent Detection with Self-Supervision and Discriminative Training (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for out-of-scope intent detection rely on strong assumptions on data distribution and confidence threshold selection. |
| Approach: | They propose a method to train an out-of-scope intent classifier in a fully end-to-end manner by simulating the test scenario in training. |
| Outcome: | The proposed method improves on four benchmark dialogue datasets and improves over state-of-the-art methods. |
Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation benchmarks focus on pairwise matching, ignoring robustness . current models exhibit frustrating degradation, with a maximum drop of 23.43 F1 score . |
| Approach: | They propose a benchmark that simulates the evaluation of open information extraction models in the real world . they perform experiments on typical models published in the last decade and a representative large language model . |
| Outcome: | The proposed model is rated robust on a knowledge-invariant clique with different syntactic and expressive forms. |
Navigating the Unknown: Intent Classification and Out-of-Distribution Detection Using Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Out-of-Distribution (OOD) detection requires great generalization capability . |
| Approach: | They propose a method that is cost-efficient, high-performing, highly robust and versatile enough to be used with smaller LLMs without sacrificing performance. |
| Outcome: | The proposed method is cost-efficient, high-performing, robust, and versatile enough to be used with smaller LLMs without sacrificing performance. |
Towards Open Environment Intent Prediction (2023.findings-acl)
Copied to clipboard
| Challenge: | Out-of-Domain (OOD) Intent Classification and New Intent Discovering are two tasks in the Task-Oriented Dialogue System. |
| Approach: | They propose a task paradigm to extend Out-of-Domain (OOD) Intent Classification and New Intent Discovering tasks in the Task-Oriented Dialogue System. |
| Outcome: | The proposed scheme improves on existing OOD intent classification and discovery datasets. |
The Open-World Lottery Ticket Hypothesis for OOD Intent Classification (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods of Out-of-Domain intent classification lack confidence in In- and Out- of-domain intents. |
| Approach: | They propose to prune overparameterized models to provide better confidence . they extend the Lottery Ticket Hypothesis to open-world scenarios . |
| Outcome: | The proposed model can be calibrated to distinguish In- and Out-of-domain intents . the model can also improve on open-world scenarios . |
Exploring The Landscape of Distributional Robustness for Question Answering Models (2022.findings-emnlp)
Copied to clipboard
Anas Awadalla, Mitchell Wortsman, Gabriel Ilharco, Sewon Min, Ian Magnusson, Hannaneh Hajishirzi, Ludwig Schmidt
| Challenge: | Existing methods for predicting distributional robustness fail to generalize reliably in a variety of test conditions. |
| Approach: | They conduct a large empirical evaluation to investigate the landscape of distributional robustness in question answering. |
| Outcome: | The proposed methods are more robust to distribution shifts than fully fine-tuned models, and few-shot prompt models exhibit better robustness than few- shot prompt models. |
Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have not studied the performance of intent classifiers against hard-negative out-of-scope utterances. |
| Approach: | They propose to generate hard-negative OOS data using ChatGPT and evaluate them against three benchmark intent classifiers. |
| Outcome: | The proposed method improves classifiers' robustness against hard-negative out-of-scope utterances and general OOS data. |