ChatGPT Role-play Dataset: Analysis of User Motives and Model Naturalness (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in interactive large language models like ChatGPT have revolutionized various domains, but their behavior in natural and role-play settings remains underexplored. |
| Approach: | They analyze ChatGPT interactions in a normal way and a role-play setting to examine its behavior in conversational settings. |
| Outcome: | The proposed dataset shows that chatGPT behaves in natural and role-play settings with different user motives and model naturalness. |
Similar Papers
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023.findings-acl)
Copied to clipboard
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, Jimmy Huang
| Challenge: | Currently, the evaluation of large language models (LLMs) such as ChatGPT in academic datasets is difficult due to the difficulty of evaluating the generative outputs produced by this model against the ground truth. |
| Approach: | They evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in academic datasets. |
| Outcome: | The proposed model performs well on 140 tasks and generates 255K responses in these datasets. |
Image-Chat: Engaging Grounded Conversations (2020.acl-main)
Copied to clipboard
| Challenge: | In order for machines to communicate with humans, they must understand the natural things that humans say about the world they live in and respond in kind. |
| Approach: | They propose to fuse a set of neural architectures using image and text representations to achieve this goal. |
| Outcome: | The proposed model performs well on the Image-Chat task and humans prefer it 47.7% of the time. |
Is ChatGPT a General-Purpose Natural Language Processing Task Solver? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in scale have enabled large language models to perform NLP tasks zero-shot . however, it is not known whether ChatGPT can serve as a generalist model that can perform many NLP jobs zero- shot. |
| Approach: | They empirically evaluate ChatGPT's zero-shot learning ability on 20 popular NLP datasets . they find it performs well on many tasks favoring reasoning abilities . |
| Outcome: | The proposed model can perform many NLP tasks zero-shot without adaptation on downstream data. |
Can You Follow Me? Testing Situational Understanding for ChatGPT (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have identified certain SU limitations in non-chatbot Large Language models, but the extent and causes of these limitations are not well understood. |
| Approach: | They propose a synthetic environment for SU testing in chat-oriented models . they test models' ability to track and enumerate environment states . |
| Outcome: | The proposed environment allows for controlled and systematic testing of SU in chat-oriented models, and to better understand underlying causes for performance patterns. |
NatCS: Eliciting Natural Customer Support Dialogues (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing task-oriented dialogue datasets do not reflect the expected characteristics of real customer support conversations. |
| Approach: | They propose to collect real customer service conversations from real conversations . they show that dialogue act annotations provide more effective training data . |
| Outcome: | The proposed approach is more representative of real human-to-human conversations compared to existing dialogue datasets . the proposed approach can be used to facilitate open research in natural dialog systems . |
ChatGPT for Zero-shot Dialogue State Tracking: A Solution or an Opportunity? (2023.acl-short)
Copied to clipboard
Michael Heck, Nurul Lubis, Benjamin Ruppik, Renato Vukovic, Shutong Feng, Christian Geishauser, Hsien-chin Lin, Carel van Niekerk, Milica Gasic
| Challenge: | Recent research on dialog state tracking (DST) focuses on methods that allow few- and zero-shot transfer to new domains or schemas. |
| Approach: | They propose to use schema descriptions to facilitate zero-shot transfer to new domains . they argue that general purpose language models lack the ability to replace specialized systems . |
| Outcome: | The proposed method achieves state-of-the-art in zero-shot DST with in-context learning capabilities. |
Harnessing the Power of Large Language Models for Empathetic Response Generation: Empirical Investigations and Improvements (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Empathetic dialogue is an essential part of building harmonious social relationships and contributes to the development of a helpful AI. |
| Approach: | They propose three methods to improve the performance of large language models (LLMs) they propose semantically similar in-context learning, two-stage interactive generation and combination with the knowledge base. |
| Outcome: | The proposed methods achieve state-of-the-art in automatic and human evaluations and the possibility of GPT-4 simulating human evaluators. |
Exploring the Potential of ChatGPT on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relations (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies have demonstrated ChatGPT's remarkable few-shot, even zero-shot learning abilities when compared to other models. |
| Approach: | They quantitatively evaluate the performance of ChatGPT on inter-sentential relations such as temporal relations, causal relations, and discourse relations. |
| Outcome: | The proposed model performs well on temporal relations, causal relations, and discourse relations. |
Uncovering the Potential of ChatGPT for Discourse Analysis in Dialogue: An Empirical Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models have shown remarkable capability in many downstream tasks, yet their ability to understand discourse structures of dialogues remains less explored. |
| Approach: | They aim to systematically inspect ChatGPT’s performance in two discourse analysis tasks: topic segmentation and discourse parsing. |
| Outcome: | The proposed model can give more reasonable topic structures than human annotations but only linearly parses the hierarchical rhetorical structures. |
Primacy Effect of ChatGPT (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing machine learning models may lead to poor performance in discriminative natural language understanding tasks. |
| Approach: | They propose to use ChatGPT to query large amounts of human-written text to find the answer to a question. |
| Outcome: | The proposed model has a high chance to select labels at earlier positions as the answer. |