Investigating person-specific errors in chat-oriented dialogue systems (2022.acl-short)
Copied to clipboard
| Challenge: | Errors in general chatbots and chatbot that follow a rough persona have been studied . but those in chatbot based on real people have not been thoroughly investigated . |
| Approach: | They analyze dialogue data of a generation-based chatbot trained from dialogue data . they find errors in attributes and relations can be divided into two levels: self and other . |
| Outcome: | The results show that errors in chatbots can be divided into two types . the correspondence with an existing taxonomy of errors was also investigated . |
Similar Papers
Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systems (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models enable them to follow freeform instructions, including imitating generic or specific demographic personas in conversations. |
| Approach: | They propose to investigate persona biases by experimenting with UNIVERSALPERSONA, a model that incorporates both generic and specific personas. |
| Outcome: | The proposed model systematically measures persona biases in harmful expression and harmful agreement. |
Exploring the Impact of Human Evaluator Group on Chat-Oriented Dialogue Evaluation (2024.lrec-main)
Copied to clipboard
| Challenge: | Evaluator groups such as domain experts, university students, and crowdworkers have been used to assess and compare chat-oriented dialogue systems. |
| Approach: | They analyze the impact of evaluator groups on dialogue system evaluation by testing 4 state-of-the-art dialogue systems using 4 distinct evaluer groups. |
| Outcome: | The proposed evaluations show that the evaluator group impact is not seen for Pairwise, and that it is beneficial for certain metrics. |
A Typology of Errors for User Utterances in Chatbots (2024.lrec-main)
Copied to clipboard
| Challenge: | Non-prescriptive language uses in chatbot communication create challenges for Semantic Parsing (SP). |
| Approach: | They propose a typology based on a sample of non-prescriptive language uses mined from a domain-specific chatbot log and a framework for automatically mapping these errors to the typology. |
| Outcome: | The proposed typology can be extended to include new classes of errors across different domains and user demographics. |
The R-U-A-Robot Dataset: Helping Avoid Chatbot Deception by Detecting User Questions About Human or Non-Human Identity (2021.acl-long)
Copied to clipboard
| Challenge: | We analyze 2,500 phrasings related to the intent of “Are you a robot?” and 2,500 adversarially selected utterances to determine whether systems are non-human. |
| Approach: | They analyze 2,500 phrasings related to the intent of "Are you a robot?" and 2,500 adversarially selected utterances to determine whether systems are non-human. |
| Outcome: | The proposed model and two systems fail to confirm non-human intent, and the proposed model is complex. |
Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluations (2024.lrec-main)
Copied to clipboard
| Challenge: | Personalization is a multifaceted process that requires multiple definitions and varies between individuals. |
| Approach: | They propose to systemically survey the recent landscape of personalized dialogue generation including the datasets employed, methodologies developed, and evaluation metrics applied. |
| Outcome: | The proposed model can generate fluent and coherent responses to human queries in a language-based conversational agent. |
Inconsistent dialogue responses and how to recover from them (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to assess and bolster utterance consistency of chat systems have been shown difficult to detect. |
| Approach: | They propose to use annotators to write dialogue responses and recovery utterances to assess and bolster utteration consistency of chat systems. |
| Outcome: | The proposed dataset significantly improves the detection and resolution of inconsistencies in chat conversations. |
Training Millions of Personalized Dialogue Agents (D18-1)
Copied to clipboard
| Challenge: | Current dialogue systems fail at being engaging for users when trained end-to-end without relying on proactive reengaging scripted strategies. |
| Approach: | They propose a dataset that provides 5 million personas and 700 million person-based dialogues. |
| Outcome: | The proposed dataset provides 5 million personas and 700 million person-based dialogues. |
Getting To Know You: User Attribute Extraction from Dialogues (2020.lrec-1)
Copied to clipboard
| Challenge: | a new method to extract user attributes from dialogues is needed to improve user understanding. |
| Approach: | They propose to leverage dialogues with conversational agents to automatically extract user attributes from dialogues. |
| Outcome: | The proposed model surpasses retrieval and generation baselines on human evaluation. |
ChatGPT Role-play Dataset: Analysis of User Motives and Model Naturalness (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in interactive large language models like ChatGPT have revolutionized various domains, but their behavior in natural and role-play settings remains underexplored. |
| Approach: | They analyze ChatGPT interactions in a normal way and a role-play setting to examine its behavior in conversational settings. |
| Outcome: | The proposed dataset shows that chatGPT behaves in natural and role-play settings with different user motives and model naturalness. |
Recipes for Building an Open-Domain Chatbot (2021.eacl-main)
Copied to clipboard
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, Jason Weston
| Challenge: | Existing work shows that scaling models in the number of parameters and the size of the data they are trained on gives improved results, but other factors are important. |
| Approach: | They propose to build open-domain chatbots that can be scaled to improve their performance . they use a blend of cognitive and cognitive skills to build a model that combines these skills . |
| Outcome: | The proposed models outperform existing approaches in multi-turn dialogue on engagingness and humanness measurements. |