Why Didn’t You Listen to Me? Comparing User Control of Human-in-the-Loop Topic Models (P19-1)
Copied to clipboard
| Challenge: | Informed prior-based methods provide better control than constraints, but constraints yield higher quality topics, but with less control. |
| Approach: | They propose to use constraints and informed prior-based methods to improve user control and topic coherence. |
| Outcome: | The proposed methods improve user control and topic coherence, while constraints yield higher quality topics, but with less control. |
Similar Papers
A User-Centered, Interactive, Human-in-the-Loop Topic Modelling System (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent research has demonstrated the value of user feedback, but there are still issues to consider, such as the difficulty in tracking changes and comparing different models. |
| Approach: | They propose a human-in-the-loop topic modeling system that integrates users’ knowledge into the modelling process, enabling them to refine the model iteratively. |
| Outcome: | The proposed system is based on a series of user studies to assess its performance in progressively more realistic applications. |
Are Neural Topic Models Broken? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation paradigms are often divorced from real-world use . recent results have challenged the validity of the prevailing model evaluation paradigm . |
| Approach: | They show that neural topic models fare worse in both respects compared to an established classical method. |
| Outcome: | The proposed method outperforms the members of the ensemble in both respects. |
Language Models Don’t Know What You Want: Evaluating Personalization in Deep Research Needs Real Users (2026.acl-long)
Copied to clipboard
Nishant Balepur, Malachi Hamada, Varsha Kishore, Sergey Feldman, Amanpreet Singh, Pao Siangliulue, Joseph Chee Chang, Eunsol Choi, Jordan Lee Boyd-Graber, Aakanksha Naik
| Challenge: | Earlier research used real users to push personalization, but easy-to-use judges have been criticized for not adopting online studies. |
| Approach: | They propose a personalized action-following tool that infers a user's research interests and proposes personalized actions for a query. |
| Outcome: | The proposed tool beats baselines in citation metrics and personalized action-following with an online version of MySQA. |
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing metrics for dialog evaluation are trained on human annotations, which is cumbersome to collect. |
| Approach: | They propose to use user sentiment and other information as proxy to measure the quality of previous dialogs. |
| Outcome: | The proposed model is comparable to models trained on human annotated data. |
The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have shown capabilities close to human performance in various analytical tasks. |
| Approach: | They investigate the efficiency and accuracy of Large Language Models in specialized tasks . they integrate LLMs with expert annotators to observe the impact of LLM suggestions . |
| Outcome: | The proposed model improves task completion speed but introduces anchoring bias . the proposed model is not suitable for open-ended analysis, but is capable of handling specialized tasks. |
Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis (2024.eacl-long)
Copied to clipboard
Zongxia Li, Andrew Mao, Daniel Stephens, Pranav Goel, Emily Walpole, Alden Dima, Juan Fung, Jordan Boyd-Graber
| Challenge: | Existing evaluation metrics such as coherence and coherency are inadequate for neural topic models. |
| Approach: | They conduct the first evaluation of neural, supervised and classical topic models in an interactive task-based setting. |
| Outcome: | The proposed model performs better on cluster evaluation metrics and human evaluations than classical models on real-world tasks. |
Opportunities for Human-centered Evaluation of Machine Translation Systems (2022.findings-naacl)
Copied to clipboard
| Challenge: | a new study examines the role of machine translation in larger user-facing systems . a sysadmin and a human factors researcher are developing evaluation tools . |
| Approach: | They argue that machine translation models are embedded in larger user-facing systems . they argue that evaluation at the systems level is still lacking . |
| Outcome: | The proposed model evaluations are based on human-computer interaction models . the authors argue that evaluations should be based more on the entire system . |
Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas (2024.findings-emnlp)
Copied to clipboard
Salvatore Giorgi, Tingting Liu, Ankit Aich, Kelsey Isman, Garrick Sherman, Zachary Fried, João Sedoc, Lyle Ungar, Brenda Curtis
| Challenge: | Large language models (LLMs) are increasingly being used in human-centered social scientific tasks, such as data annotation, synthetic data creation, and engaging in dialog. |
| Approach: | They propose to prompt LLMs with human-like personas and ask them to answer as if they were a specific human, either explicitly, with exact demographics, political beliefs, and lived experiences, or implicitly via names prevalent in specific populations. |
| Outcome: | The proposed model is based on explicit, explicit, and implicit personas, and fails to show implicit biases. |
Hey Siri. Ok Google. Alexa: A topic modeling of user reviews for smart speakers (D19-55)
Copied to clipboard
| Challenge: | Using coherence scores to choose topics, we test whether the results help us to understand user interests and concerns. |
| Approach: | They analyze user reviews from Best Buy US website for smart speakers to determine whether they provide useful information for product analysis. |
| Outcome: | The proposed models capture brand performance and differences and differentiate the market into two distinct groups with different properties. |
Contextualized Topic Coherence Metrics (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing topic models that estimate the interpretability of topics are difficult to compare due to their nature as unsupervised models. |
| Approach: | They propose to use contextualized topic coherence metrics to simulate human-centered coherency evaluation while maintaining the efficiency of other automated methods. |
| Outcome: | The proposed metrics better reflect human judgment on topics extracted from short text collections by avoiding highly scored topics that are meaningless to humans. |