Papers by Jongwook Han
Quantifying Data Contamination in Psychometric Evaluations of LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies have raised concerns about data contamination from psychometric inventories . however, there is no systematic attempt to quantify the extent of data contamination . |
| Approach: | They propose a framework to measure data contamination in psychometric evaluations of Large Language Models by item memorization, evaluation memorisation and target score matching. |
| Outcome: | The proposed framework evaluates item memorization, evaluation memorisation, and target score matching in 21 models from major families and four widely used psychometric inventories. |
Don’t Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models (2026.acl-long)
Copied to clipboard
| Challenge: | Small language models struggle with tool-use tasks, particularly in selecting appropriate tools and identifying correct parameters. |
| Approach: | They propose a training-free method that leverages peakedness to align schemas with pretraining knowledge to rename tool components. |
| Outcome: | Experiments on MetaTool and RoTBench show that PA-Tool significantly improves tool-use accuracy without retraining. |
Value Portrait: Assessing Language Models’ Values through Psychometrically and Ecologically Valid Items (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks rely on human annotations that are vulnerable to value-related biases. |
| Approach: | They propose a value portrait benchmark that uses items that capture real-life user-LLM interactions and a rated item based on its similarity to their own thoughts to determine reliability. |
| Outcome: | The proposed framework improves the relevance of assessment results to real-world LLM usage by allowing human subjects to rate items with similarity to their own thoughts and derived correlations between these ratings and the subjects’ actual value scores. |
PVP: An Image Dataset for Personalized Visual Persuasion with Persuasion Strategies, Viewer Characteristics, and Persuasiveness Ratings (2025.acl-long)
Copied to clipboard
| Challenge: | Visual persuasion uses visual elements to influence cognition and behaviors . lack of comprehensive data sets connect persuasiveness of images with personal information . |
| Approach: | They propose to use a dataset to connect persuasiveness with personal information . they find psychological characteristics enhance the generation and evaluation of persuasive images . |
| Outcome: | The proposed dataset provides persuasiveness scores of images evaluated by human annotators along with demographic and psychological characteristics. |