PragmatiCQA: A Dataset for Pragmatic Question Answering in Conversations (2023.findings-acl)
Copied to clipboard
| Challenge: | Mars? - PragmatiCQA |
| Approach: | Mars? - The Paper . |
| Outcome: | The proposed dataset features 6873 QA pairs that explores pragmatic reasoning in conversations over a diverse set of topics. |
Similar Papers
QAConv: Question Answering on Informative Conversations (2022.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that state-of-the-art pretrained QA systems have limited zero-shot performance and tend to predict our questions as unanswerable. |
| Approach: | They propose a question-answering dataset that uses conversations as a knowledge source. |
| Outcome: | The proposed dataset provides a training and evaluation testbed to facilitate QA on conversations research. |
IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing open-domain QA tasks focus on questions whose answer can be deduced directly from global factual knowledge. |
| Approach: | They propose a dataset where each question is based on a counterfactual presupposition via an "if" clause. |
| Outcome: | The IfQA dataset contains 3,800 questions that were annotated by crowdworkers on relevant Wikipedia passages. |
TWEETQA: A Social Media Focused Question Answering Dataset (P19-1)
Copied to clipboard
Wenhan Xiong, Jiawei Wu, Hong Wang, Vivek Kulkarni, Mo Yu, Shiyu Chang, Xiaoxiao Guo, William Yang Wang
| Challenge: | Social media is becoming an important realtime information source, especially during natural disasters and emergencies. |
| Approach: | They present a large-scale dataset for question answering over social media data . they gather tweets used by journalists and ask human annotators to write questions upon them . |
| Outcome: | The proposed dataset shows that neural models that perform well on formal texts are limited in their performance . the proposed model is still lagging behind human performance with a large margin . |
DoQA - Accessing Domain-Specific FAQs via Conversational QA (2020.acl-main)
Copied to clipboard
| Challenge: | a dataset of 2,437 dialogues and 10,917 QA pairs is used to access domain-specific FAQ information. |
| Approach: | They present a dataset with 2,437 dialogues and 10,917 QA pairs for FAQs . they use the Wizard of Oz method with crowdsourcing to create dialogues using the original post and the original reply. |
| Outcome: | The proposed system can access domain-specific FAQ information without training data. |
ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters (N19-1)
Copied to clipboard
| Challenge: | ComQA dataset captures question phenomena and the diverse ways in which they are formulated. |
| Approach: | They propose a large dataset of real user questions that captures question phenomena and the diverse ways in which they are formulated. |
| Outcome: | The proposed dataset can be a driver of future research on factoid question answering (QA). |
Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges (2025.acl-long)
Copied to clipboard
Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter, Barbara Plank
| Challenge: | linguistics studies how context influences meaning of language and how people use it to convey implied meanings, emotions, and intentions. |
| Approach: | They analyze task designs, data collection methods, evaluation approaches and their relevance to real-world applications. |
| Outcome: | The findings highlight emerging trends, challenges, and gaps in existing benchmarks . the findings will contribute to more nuanced and context-aware NLP models . |
From Chat Logs to Collective Insights: Aggregative Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to analyzing large-scale conversation logs treat interactions as independent, missing critical insights. |
| Approach: | They propose a task that requires models to reason explicitly over thousands of user-chatbot interactions to answer aggregational queries. |
| Outcome: | The proposed task requires models to reason over thousands of user-chatbot interactions to answer aggregational queries such as identifying emerging concerns among demographics. |
ForumSum: A Multi-Speaker Conversation Summarization Dataset (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Abstractive summarization quality has been improved but there is a lack of data for conversation summarizing applications. |
| Approach: | They propose to build a conversation summarization dataset with human written summaries from internet forums. |
| Outcome: | The proposed dataset can be easily expanded to improve conversation summarization applications. |
Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Table Question Answering (TQA) aims to answer natural language questions using tabular data. |
| Approach: | They propose a systematic overview of TQA research using large language models and summarize available benchmarks based on task features. |
| Outcome: | The proposed framework provides a comprehensive overview of the current state of the art in the field of Table Question Answering. |
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023.findings-acl)
Copied to clipboard
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, Jimmy Huang
| Challenge: | Currently, the evaluation of large language models (LLMs) such as ChatGPT in academic datasets is difficult due to the difficulty of evaluating the generative outputs produced by this model against the ground truth. |
| Approach: | They evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in academic datasets. |
| Outcome: | The proposed model performs well on 140 tasks and generates 255K responses in these datasets. |