Papers by Vinay Samuel
Towards Data Contamination Detection for Modern Large Language Models: Limitations, Inconsistencies, and Oracle Challenges (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for detecting data contamination in large language models have limitations and limitations . data contamination occurs when test or evaluation data is exposed to the model during its training phases . |
| Approach: | They evaluate five different methods for detecting data contamination in large language models . they find that current methods have non-trivial limitations in their assumptions and practical applications . |
| Outcome: | The proposed methods have non-trivial limitations and difficulties in detecting contamination . the authors highlight the complexity of contamination detection in advanced LLMs . |
CIE: Controlling Language Model Text Generations Using Continuous Signals (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to control language models with intent are brittle and hard to scale. |
| Approach: | They propose to use a set of LMs to fine-tune to expect a control vector that is interpolated between a "low" and a 'high' token embedding. |
| Outcome: | The proposed method can be finetuned to expect a control vector that is interpolated between a “low” and a ‘high” token embedding. |
ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction (2024.findings-acl)
Copied to clipboard
Henry Zou, Vinay Samuel, Yue Zhou, Weizhi Zhang, Liancheng Fang, Zihe Song, Philip Yu, Cornelia Caragea
| Challenge: | Existing datasets for attribute value extraction focus on explicit attribute values while neglecting the implicit ones. |
| Approach: | They present a multimodal dataset for implicit attribute value extraction that includes AVE and multimodality. |
| Outcome: | The proposed dataset includes 68k training and 1.6k testing data across five domains. |
PersonaGym: Evaluating Persona Agents and LLMs (2025.findings-emnlp)
Copied to clipboard
Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik R Narasimhan, Vishvak Murahari
| Challenge: | Persona agents are LLM agents conditioned to act according to an assigned persona . evaluating how faithfully these agents adhere to their personas remains a challenge . |
| Approach: | a new study evaluates persona agents' ability to act according to an assigned persona . a persona agent's person score is a human-aligned automatic metric that can be used to evaluate a model . |
| Outcome: | a new evaluation framework and a human-aligned automatic metric show that persona agents can perform better. |
Can LLMs Augment Low-Resource Reading Comprehension Datasets? Opportunities and Challenges (2024.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive zero-shot performance on a wide range of NLP tasks. |
| Approach: | They propose to use large language models to augment extractive reading comprehension datasets by fine-tuning their annotations and comparing their performance to human annotators. |
| Outcome: | The proposed model can be used to augment extractive reading comprehension datasets. |