OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistants (2024.emnlp-main)
Copied to clipboard
Jaspreet Ranjit, Brihi Joshi, Rebecca Dorn, Laura Petry, Olga Koumoundouros, Jayne Bottarini, Peichen Liu, Eric Rice, Swabha Swayamdipta
| Challenge: | a large-scale analysis of millions of tweets on homelessness is challenging to understand at scale. |
| Approach: | They propose a framing typology: Online Attitudes Towards Homelessness (OATH) They use large language models to analyze millions of tweets to find patterns in public attitudes . |
| Outcome: | The proposed model speeds up annotations while incurring a 3 point performance reduction compared to existing classifiers . |
Similar Papers
PairScale: Analyzing Attitude Change with Pairwise Comparisons (2025.findings-naacl)
Copied to clipboard
| Challenge: | a text-based framework for measuring attitudes in communities is proposed . the framework uses both implicit and explicit evidence in language to characterize attitudes . |
| Approach: | They propose a text-based framework for measuring attitudes in communities toward issues of interest using language. |
| Outcome: | The proposed framework is validated by examining attitudes on two high-profile issues in the u.s. |
Modeling Ideological Salience and Framing in Polarized Online Groups with Graph Neural Networks and Structured Sparsity (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to detect ideological divides in social media rely on knowing in advance the political orientation of text . fascist and mainstream are among the most polarized concepts in reddit in 2019 . |
| Approach: | They propose a minimally supervised method that leverages the network structure of online discussion forums to detect polarized concepts. |
| Outcome: | The proposed framework captures temporal ideological dynamics such as right-wing and left-wing radicalization using graph neural networks and sparsity learning. |
The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models (2024.findings-emnlp)
Copied to clipboard
Bolei Ma, Xinpeng Wang, Tiancheng Hu, Anna-Carolina Haensch, Michael Hedderich, Barbara Plank, Frauke Kreuter
| Challenge: | Recent advances in Large Language Models have sparked interest in validating human-like cognitive-behavioral traits. |
| Approach: | They examine whether LLM outputs reflect human-like cognitive-behavioral traits . they find that measuring AOVs embedded within LLMs remains opaque . |
| Outcome: | The proposed model can be used to evaluate human-like cognitive-behavioral traits . the proposed model could be used in writing assistants and other applications . |
Modeling Framing in Immigration Discourse on Social Media (2021.naacl-main)
Copied to clipboard
| Challenge: | Using a dataset of immigration-related tweets, we examine how ordinary people on social media frame political issues. |
| Approach: | They propose to use a dataset of immigration-related tweets labeled for multiple framing typologies from political communication theory to analyze framers. |
| Outcome: | The proposed model enables comparisons between different types of frames on social media and a dataset of immigration-related tweets. |
The Computational Anatomy of Humility: Modeling Intellectual Humility in Online Public Discourse (2024.emnlp-main)
Copied to clipboard
| Challenge: | enhancing the quality of online public discourse requires promoting foundational human virtues, such as “intellectual humility” (IH) . discourse on social media rewards forgetting our virtuous selves, embedding users within echo chambers and causing negative affect towards those who hold different beliefs. |
| Approach: | They propose to use a codebook to measure "intellectual humility" they manually validated the codebook and used it to develop LLM-based models . |
| Outcome: | The proposed model achieves a Macro-F1 score of 0.64 across labels and 0.70 when predicting IH/IA/Neutral at the coarse level. |
Aligning Language Models to User Opinions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Personality is a defining feature of human beings, shaped by a complex interplay of demographic characteristics, moral principles, and social experiences. |
| Approach: | They use public opinion surveys to model past user opinions in addition to user demographics and ideology to achieve up to 7 points accuracy gains in predicting public opinions from survey questions. |
| Outcome: | The proposed model achieves 7 points accuracy gains in predicting public opinions from public opinion surveys across a broad set of topics. |
Discovering Language Model Behaviors with Model-Written Evaluations (2023.findings-acl)
Copied to clipboard
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan
| Challenge: | Prior work creates evaluations with crowdwork or existing data sources, which are not always available. |
| Approach: | They generate evaluations automatically with language models (LMs) using crowdwork or existing data sources to find out how they behave . |
| Outcome: | The results show that large LMs repeat back a dialog user’s preferred answer and express greater desire to pursue concerning goals like resource acquisition and goal preservation. |
Modeling Frames in Argumentation (D19-1)
Copied to clipboard
| Challenge: | In argumentation, framing is used to emphasize a specific aspect of a topic while concealing others. |
| Approach: | They propose an unsupervised method for framing arguments into non-overlapping frames . authors propose a corpus of 12, 326 debate-portal arguments organized along the frames of debates' topics . |
| Outcome: | The proposed method outperforms baselines on the argumentation task by 0.28 points. |
Improving and Assessing the Fidelity of Large Language Models Alignment to Online Communities (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown promise in representing individuals and communities, but evaluating their fidelity remains a challenge. |
| Approach: | They propose a framework for aligning large language models with online communities via instruction-tuning and comprehensively evaluating alignment across various aspects of language. |
| Outcome: | The proposed framework shows that it can be used to create high-fidelity representations of people and communities. |
The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection (2025.findings-naacl)
Copied to clipboard
Tomáš Horych, Christoph Mandl, Terry Ruas, Andre Greiner-Petter, Bela Gipp, Akiko Aizawa, Timo Spinde
| Challenge: | Recent research suggests using Large Language Models (LLMs) to automate the annotation process, reducing these costs while maintaining data quality. |
| Approach: | They propose to use Large Language Models to automate annotation process and train classifiers on large datasets. |
| Outcome: | The proposed model outperforms all of the annotator LLMs on two media bias benchmark datasets (BABE and BASIL) while maintaining data quality. |