Papers by Spyros Matsoukas
Active Learning for New Domains in Natural Language Understanding (N19-2)
Copied to clipboard
| Challenge: | Existing approaches to improve the accuracy of new domains are lacking annotated live utterances. |
| Approach: | They propose an algorithm called Majority-CRF that uses an ensemble of classification models to guide the selection of relevant utterances and a sequence labeling model to prioritize informative examples. |
| Outcome: | The proposed algorithm achieves 6.6%-9% error rate reduction and statistically significant improvements on six new domains. |
A Scalable Framework for Learning From Implicit User Feedback to Improve Natural Language Understanding in Large-Scale Conversational AI Systems (2021.emnlp-main)
Copied to clipboard
Sunghyun Park, Han Li, Ameen Patel, Sidharth Mudgal, Sungjin Lee, Young-Bum Kim, Spyros Matsoukas, Ruhi Sarikaya
| Challenge: | Existing methods to improve NLU are laborintensive and expensive. |
| Approach: | They propose a scalable and automatic approach to improving NLU in a large-scale conversational AI system by leveraging implicit user feedback. |
| Outcome: | The proposed framework improves NLU in a large-scale conversational AI system across 10 domains. |
Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogs (2023.acl-industry)
Copied to clipboard
Abishek Komma, Nagesh Panyam Chandrasekarasastry, Timothy Leffel, Anuj Goyal, Angeliki Metallinou, Spyros Matsoukas, Aram Galstyan
| Challenge: | Existing methods to dialog quality estimation focus on evaluating individual turns or collecting dialog-level quality measurements from end users immediately following an interaction. |
| Approach: | They propose a dialog-level annotation workflow called Dialog Quality Annotation . they propose to annotate dialogs for attributes such as goal completion and user sentiment . |
| Outcome: | The proposed model outperforms existing methods for dialog quality estimation . it shows that high-quality human-annotated data is important for dialog-quality evaluation . |
The Alexa Meaning Representation Language (N18-3)
Copied to clipboard
Thomas Kollar, Danielle Berry, Lauren Stuart, Karolina Owczarzak, Tagyoung Chung, Lambert Mathias, Michael Kayser, Bradford Snow, Spyros Matsoukas
| Challenge: | a new meaning representation language for spoken language is introduced for Alexa . AMRL provides a common representation for how people communicate in spoken language . there is no mechanism to represent ambiguity, forcing the choice of a fixed interpretation for ambiguous utterances. |
| Approach: | They introduce a meaning representation for spoken language, the Alexa meaning represent language . they use a spoken language dataset to collect a sample of utterances from eight domains . |
| Outcome: | The proposed representation provides a common representation for spoken language understanding . it supports cross-domain queries, fine-grained types, complex utterances and composition . the proposed representation was released to developers at a trade show in 2016 . |
Joint Turn and Dialogue level User Satisfaction Estimation on Multi-Domain Conversations (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to estimate turn and dialogue level user satisfaction employ hand-crafted features and rely on complex annotation schemes, which reduce generalizability of the trained models. |
| Approach: | They propose to use an adaptive multi-task loss function to minimize hand-crafted features to estimate user satisfaction at turn level from an end user perspective. |
| Outcome: | The proposed model improves on 28 Alexa domains, two dialogue systems and three user groups on a set of user-generated dialogues from 28 Alexia domain and 28 Alexis domains. |
Fast and Scalable Expansion of Natural Language Understanding Functionality for Intelligent Agents (N18-3)
Copied to clipboard
| Challenge: | a recent paper describes efficient deep neural network architectures for expanding natural language capabilities of virtual agents. |
| Approach: | They propose deep neural network architectures that maximize re-use available resources . they use data from Amazon Alexa to accelerate expansion of new natural language domains . |
| Outcome: | The proposed methods increase accuracy in low resource settings and enable rapid development with less data. |