PhotoChat: A Human-Human Dialogue Dataset With Photo Sharing Behavior For Joint Image-Text Modeling (2021.acl-long)
Copied to clipboard
| Challenge: | PhotoChat contains 12k dialogues, each of which is paired with a user photo that is shared during the conversation. |
| Approach: | They propose to use PhotoChat to facilitate research on image-text modeling by combining a photo-sharing intent prediction task and a picture retrieval task to retrieve the most relevant photo according to the dialogue context. |
| Outcome: | The proposed tasks achieve 10.4% recall@1 and 58.1% F1 scores, indicating that the proposed dataset presents interesting yet challenging real-world problems. |
Similar Papers
How do people talk about images? A study on open-domain conversations with images. (2022.naacl-srw)
Copied to clipboard
| Challenge: | utterances/conversations are not always related to the given image, and conversation topics diverge within three turns about half of the time. |
| Approach: | They propose to enrich images' image information with captions and object tags to generate more engaging conversations when an image is presented. |
| Outcome: | The proposed enhancements improve the BLEU and Bert Score and increase the diversity and image-relevancy of generated responses to the strong baseline. |
Image-Chat: Engaging Grounded Conversations (2020.acl-main)
Copied to clipboard
| Challenge: | In order for machines to communicate with humans, they must understand the natural things that humans say about the world they live in and respond in kind. |
| Approach: | They propose to fuse a set of neural architectures using image and text representations to achieve this goal. |
| Outcome: | The proposed model performs well on the Image-Chat task and humans prefer it 47.7% of the time. |
Large Language Models can Share Images, Too! (2024.findings-acl)
Copied to clipboard
| Challenge: | Using a zero-shot prompting, large language models can be used to share images in a multi-tasking environment. |
| Approach: | They introduce a dataset that includes enriched annotations and a framework to evaluate LLMs. |
| Outcome: | The proposed framework unlocks image-sharing capability of LLMs in zero-shot prompting, with ChatGPT achieving the best performance. |
LiveChat: A Large-Scale Personalized Dialogue Dataset Automatically Constructed from Live Streaming (2023.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that open-domain dialogue systems are not able to perform well in fast-growing scenarios such as live streaming due to the domain gap between online-post constructed data and those required in downstream conversational tasks. |
| Approach: | They propose to train a conversational agent based on large social media datasets with multiple domains to improve response in live streaming scenarios. |
| Outcome: | The proposed model improves response modeling and addressee recognition in live open-domain scenarios. |
The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue (P19-1)
Copied to clipboard
| Challenge: | Using the PhotoBook dataset, we investigate shared dialogue history accumulating during conversation . human interlocutors are known to collaboratively establish a shared repository of mutual information during a conversation - this common ground is then used to optimise understanding and communication efficiency. |
| Approach: | They propose a data-collection task formulated as a collaborative game prompting two online participants to refer to images utilising both their visual context and previously established referring expressions. |
| Outcome: | The proposed model takes into account shared information accumulated in a reference chain and is important to resolve later descriptions. |
SHARE: Shared Memory-Aware Open-Domain Long-Term Dialogue Dataset Constructed from Movie Script (2025.acl-long)
Copied to clipboard
| Challenge: | Antoine de Saint-Exupéry Memory in dialogue plays a crucial role in building relationships and facilitating the ongoing conversation. |
| Approach: | They propose a long-term dialogue dataset named SHARE that includes shared memories between two individuals. |
| Outcome: | The proposed dataset makes long-term dialogues more engaging and sustainable . it includes summaries of persona information and events of two individuals . |
Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledge (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies focus on image-sharing behavior in singular sessions, leading to limited long-term social interaction. |
| Approach: | They propose a large-scale long-term multi-modal dialogue dataset that generates long-time multi-modity dialogue distilled from ChatGPT and proposed image aligner. |
| Outcome: | The proposed framework generates long-term multi-modal dialogue from ChatGPT and image aligner. |
MMChat: Multi-Modal Chat Dataset on Social Media (2022.lrec-1)
Copied to clipboard
| Challenge: | Incorporating multi-modal contexts in conversation is important for developing engaging dialogue systems. |
| Approach: | They propose a large scale Chinese multi-modal dialogue corpus that contains image-grounded dialogues from real conversations on social media. |
| Outcome: | The proposed model can handle sparsity issues in dialogue generation tasks by incorporating image features. |
NoteChat: A Dataset of Synthetic Patient-Physician Conversations Conditioned on Clinical Notes (2024.findings-acl)
Copied to clipboard
| Challenge: | NoteChat is a cooperative multi-agent framework for generating patient-physician dialogues . evaluator finds it outperforms state-of-the-art models for generating clinical notes . clinical documentation is largely done by physicians at both steps . |
| Approach: | They propose a cooperative multi-agent framework leveraging Large Language Models to generate patient-physician dialogues. |
| Outcome: | The proposed framework outperforms state-of-the-art models for generating clinical notes . it can engage patients directly and help clinical documentation, a leading cause of physician burnout . |
MPCHAT: Towards Multimodal Persona-Grounded Conversation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing research on persona-based dialogue has focused on textual persona that delivers personal facts or personalities, but image modality can reveal the speaker’s personal characteristics and experiences in episodic memory. |
| Approach: | They propose a multimodal persona-based dialogue dataset which extends persona with both text and images to contain episodic memories. |
| Outcome: | The proposed dataset extends persona with text and images to contain episodic memories. |