Evaluation of LLMs-based Hidden States as Author Representations for Psychological Human-Centered NLP Tasks (2025.findings-naacl)
Copied to clipboard
| Challenge: | Many human-centered NLP tasks focus on assessing human-attributes of a user based on their language. |
| Approach: | They evaluate different ways of representing documents and users using different LM and HuLM architectures to predict task outcomes as dynamically changing states and averaged trait-like user-level attributes. |
| Outcome: | The proposed representations predict valence, arousal, empathy, and distress as well as trait-like user-level attributes. |
Similar Papers
Systematic Evaluation of Auto-Encoding and Large Language Model Representations for Capturing Author States and Traits (2025.findings-acl)
Copied to clipboard
Khushboo Singh, Vasudha Varadarajan, Adithya V Ganesan, August Håkan Nilsson, Nikita Soni, Syeda Mahwish, Pranav Chitale, Ryan L. Boyd, Lyle Ungar, Richard N Rosenthal, H. Schwartz
| Challenge: | Large Language Models (LLMs) are increasingly used in human-centered applications, yet their ability to model diverse psychological constructs is not well understood. |
| Approach: | They evaluated a range of Transformer-LMs to predict psychological variables across five major dimensions: affect, substance use, mental health, sociodemographics, and personality. |
| Outcome: | The models predict affect, substance use, mental health, sociodemographics, and personality across five major dimensions. |
Discovering Language Model Behaviors with Model-Written Evaluations (2023.findings-acl)
Copied to clipboard
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan
| Challenge: | Prior work creates evaluations with crowdwork or existing data sources, which are not always available. |
| Approach: | They generate evaluations automatically with language models (LMs) using crowdwork or existing data sources to find out how they behave . |
| Outcome: | The results show that large LMs repeat back a dialog user’s preferred answer and express greater desire to pursue concerning goals like resource acquisition and goal preservation. |
Layered Insights: Generalizable Analysis of Human Authorial Style by Leveraging All Transformer Layers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to authorship attribution model only learn from the output layer of pre-trained transformers, ignoring representations learned at other layers. |
| Approach: | They propose a model that leverages the various linguistic representations learned at different layers of pre-trained transformer-based models to model the authorship attribution task more effectively. |
| Outcome: | The proposed model performs better on out-of-domain and in-domain scenarios, while ignoring representations learned at other layers. |
Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing research suggests that contextual representations of large language models exhibit subpar performance in downstream tasks, struggling to fully capture the semantic nuances of words. |
| Approach: | They investigate the bottom-up evolution of lexical semantics for a popular LLM . they probing its hidden states at the end of each layer using a contextualized word identification task . |
| Outcome: | The proposed model is able to encode lexical semantics in lower layers while achieving weaker induction in higher layers. |
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) provide a data-centric solution to alleviate limitations of real-world data with synthetic data generation. |
| Approach: | They propose a generic workflow for LLM-driven synthetic data generation. |
| Outcome: | The proposed workflows highlight gaps in existing research and outline avenues for future studies. |
The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have shown capabilities close to human performance in various analytical tasks. |
| Approach: | They investigate the efficiency and accuracy of Large Language Models in specialized tasks . they integrate LLMs with expert annotators to observe the impact of LLM suggestions . |
| Outcome: | The proposed model improves task completion speed but introduces anchoring bias . the proposed model is not suitable for open-ended analysis, but is capable of handling specialized tasks. |
Beyond Fixed Psychological Personas: State Beats Trait, but Language Models are State-Blind (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing persona datasets capture only trait, and ignore impact of state. |
| Approach: | They use a Reddit dataset to study user interactions with language models . they find that existing persona datasets capture only trait and ignore impact of state . |
| Outcome: | The proposed dataset decomposes variance and finds that LLMs are state-blind . the reward models react to user state, but inconsistently, the authors say . |
Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks (2025.coling-main)
Copied to clipboard
| Challenge: | Existing work uses large language models (LLMs) to evaluate natural language process tasks, but there are shortcomings in current LLMs. |
| Approach: | They examine the alignment between LLM evaluators and human annotators by comparing conventional and alignment tasks with different evaluation criteria. |
| Outcome: | The proposed models excel in general criteria, such as fluency, but face challenges with complex criteria, including numerical reasoning. |
Modeling, Evaluating, and Embodying Personality in LLMs: A Survey (2025.findings-emnlp)
Copied to clipboard
Iago Alves Brito, Julia Soares Dollis, Fernanda Bufon Färber, Pedro Schindler Freire Brasil Ribeiro, Rafael Teixeira Sousa, Arlindo Rodrigues Galvão Filho
| Challenge: | This survey provides a comprehensive overview of the LLM-driven personality scenario. |
| Approach: | This survey provides a comprehensive overview of the LLM-driven personality scenario. |
| Outcome: | The proposed taxonomy analyzes the limitations of existing methods and identifies key research gaps. |
Justice in Judgment: Unveiling (Hidden) Bias in LLM-assisted Peer Reviews (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that large language models carry implicit biases across race, gender, and religion . prior studies documented such biase based on text generation and classification tasks . |
| Approach: | They investigate bias in large language models by controlling metadata on author metadata . authors found affiliation bias favoring authors from highly ranked institutions . |
| Outcome: | The proposed model favors authors from highly ranked institutions, the authors show . the model also favors author affiliations from highly-ranked institutions . |