Show or Tell? Modeling the evolution of request-making in Human-LLM conversations (2026.findings-eacl)
Copied to clipboard
| Challenge: | a new framework to describe request-making segments user input into request content, roles assigned, query-specific context, and task-independent expressions. |
| Approach: | They propose a framework to describe request-making that segments user input into request content, roles assigned, query-specific context, and the remaining task-independent expressions. |
| Outcome: | The proposed framework reveals fundamental and habitual user-LLM interaction patterns beyond individual task completion. |
Similar Papers
User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that asking for direct user feedback can be disruptive . we examine whether incorporating the contents of user feedback improves model performance . |
| Approach: | They analyze user feedback in the user-LLM conversation logs and harvest learning signals from it. |
| Outcome: | The proposed approach can lead to model degradation on two user-LM interaction datasets. |
Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wild (2025.emnlp-main)
Copied to clipboard
| Challenge: | Rather than passively accepting output, users actively refine, explore, and co-construct text. |
| Approach: | They conduct a large scale analysis of user-LLM collaboration behavior with two popular AI assistants, Bing Copilot and WildChat. |
| Outcome: | The proposed models show that a small group of prototypical human AI collaboration behaviors explain a majority of the variation seen in user-LLM interaction. |
Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Human-LLM Dialogue (2026.findings-acl)
Copied to clipboard
Jonathan Ivey, Shivani Kumar, Jiayu Liu, Hua Shen, Sushrita Rakshit, Rohan Raju, Haotian Zhang, Aparna Ananthasubramaniam, Junghwan Kim, Bowen Yi, Dustin Wright, Abraham Israeli, Anders Giovanni Møller, Lechen Zhang, David Jurgens
| Challenge: | Recent work has sought to use large language models to simulate human-human and human-LLM interactions. |
| Approach: | They use a large-scale dataset to generate a paired LLM-LLM and human-LLm dialogues from the WildChat dataset and quantify how well they align with their human counterparts. |
| Outcome: | The proposed models perform similarly in simulating English, Chinese, and Russian dialogues. |
WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback (2026.acl-long)
Copied to clipboard
Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Kumar Jauhar, Sihao Chen, Shan Xia, Hongfei Zhang, Jieyu Zhao, Xiaofeng Xu, Xia Song, Jennifer Neville
| Challenge: | Traditional alignment methods rely on human annotations and are subjective and misalignment with real-world user preferences. |
| Approach: | They propose a framework that leverages in-situ user feedback during conversations with LLMs to create preference datasets automatically. |
| Outcome: | The proposed framework identifies and classifies user feedback to LLM responses between conversation turns and creates examples of preferred and dispreferred responses according to user preferences. |
The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have shown capabilities close to human performance in various analytical tasks. |
| Approach: | They investigate the efficiency and accuracy of Large Language Models in specialized tasks . they integrate LLMs with expert annotators to observe the impact of LLM suggestions . |
| Outcome: | The proposed model improves task completion speed but introduces anchoring bias . the proposed model is not suitable for open-ended analysis, but is capable of handling specialized tasks. |
The Death and Life of Great Prompts: Analyzing the Evolution of LLM Prompts from the Structural Perspective (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has shown that high-quality prompts are essential for LLMs to produce accurate and relevant responses. |
| Approach: | They analyze 10,538 in-the-wild prompts collected from various platforms and develop a framework that decomposes the prompts into eight key components. |
| Outcome: | The proposed framework decomposes 10,538 in-the-wild prompts into eight components. |
Towards Modern Topic Models: A Survey of Taxonomies and Paradigm Shifts from Algorithm-Centric to LLM-Centered Topic Analysis (2026.findings-acl)
Copied to clipboard
| Challenge: | Topic modeling (TM) is a classic unsupervised learning task in the field of natural language processing. |
| Approach: | They propose a new taxonomy that emphasizes the role of LLMs and the design of end-to-end workflows. |
| Outcome: | The proposed taxonomy emphasizes the role of LLMs and the design of end-to-end workflows. |
Navigating Rifts in Human-LLM Grounding: Study and Benchmark (2025.acl-long)
Copied to clipboard
| Challenge: | Language models excel at following instructions but struggle with collaborative aspects of conversation that humans naturally employ. |
| Approach: | They analyze logs from WildChat, MultiWOZ, and Bing Chat to examine grounding challenges . they propose a benchmark to determine when LLMs fail to initiate grounding . |
| Outcome: | The proposed model predicts interactions that fail to ground with users . the proposed model is based on human-human interactions with humans . |
The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions (2023.emnlp-main)
Copied to clipboard
Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, Jiawei Han
| Challenge: | Recent advances in large language models (LLMs) have produced models that exhibit remarkable performance across a variety of NLP tasks. |
| Approach: | They analyze a large-scale collection of user-GPT conversations to identify a significant gap between academic research in NLP and the needs of real-world NLP applications. |
| Outcome: | The proposed model outperforms existing models in a large-scale collection of user-GPT conversations and identifies a significant gap between the tasks that users frequently request from LLMs and the tasks commonly studied in academic research. |
Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown considerable promise for annotation purposes, but questions remain about their ability to capture human label variation (HLV) label variation is genuine disagreement between annotators observed across NLP tasks. |
| Approach: | They investigate how label and explanation variation manifests within and across LLMs with respect to the Natural Language Inference task. |
| Outcome: | The proposed models generate label distributions similar to humans but exhibit distinct, idiosyncratic judgments and disagreement patterns. |