Papers by Naihao Deng
Benchmarking and Improving LLM Robustness for Personalized Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluations focus on whether a model’s responses align with a user’s preferences, but factuality is an important yet overlooked dimension. |
| Approach: | They propose a scalable framework for evaluating robustness of large language models in personalization and a new dataset, PERGData. |
| Outcome: | The proposed framework improves robustness by 25% across models. |
Beyond Screenshots: Evaluating VLMs’ Understanding of UI Animations (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies of Vision Language Models (VLMs) for UI understanding have focused primarily on static screenshots, leaving it unclear how well these models handle dynamic UI animations. |
| Approach: | They evaluate UI animation models' ability to perceive animation effects and interpret animation meaning . they use motion, context, and perceptual cues to probe factors affecting VLM performance . |
| Outcome: | The proposed model can detect primitive motion, but its interpretation is inconsistent . the proposed model is based on 300 annotated UI animation videos . |
Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to generate text in mental health are limiting, but they are effective for many tasks. |
| Approach: | They propose a task-adaptive tokenizer that allows for the integration of task-specific tokens into the pre-trained model's tokenization step. |
| Outcome: | The proposed tokenization approach improves generation performance on psychological question-answering tasks in Chinese and English while using 60% fewer tokens. |
Chumor 2.0: Towards Better Benchmarking Chinese Humor Understanding from (Ruo Zhi Ba) (2025.findings-acl)
Copied to clipboard
Ruiqi He, Yushu He, Longju Bai, Jiarui Liu, Zhenjie Sun, Zenghao Tang, He Wang, Hanchen Xia, Rada Mihalcea, Naihao Deng
| Challenge: | Existing studies on humor in non-English languages lack culturally nuanced humor in other languages. |
| Approach: | They construct a Chinese humor explanation dataset using a reddit-like platform . they test ten LLMs and find they are significantly better than existing LLM models . |
| Outcome: | The proposed dataset is the first and largest Chinese humor explanation dataset. |
CliniDial: A Naturally Occurring Multimodal Dialogue Dataset for Team Reflection in Action During Clinical Operation (2025.findings-acl)
Copied to clipboard
| Challenge: | Prior studies have shown that sufficient collaboration is the key factor that determines the outcome of an operation. |
| Approach: | They propose to model the communication between team members during an operation using audio data and physiology signals from two camera angles. |
| Outcome: | The proposed model is based on existing frameworks and invites future effort on developing methods that can deal with real-world clinical data. |
Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expect (2022.coling-1)
Copied to clipboard
| Challenge: | text-to-SQL is a language processing and database-based language processing (NLP) task is to convert natural utterances into SQL queries and its practical application is to build natural language interfaces to database systems. |
| Approach: | They propose to conduct a systematic survey of text-to-SQL to examine the challenges and potential future directions. |
| Outcome: | The proposed system converts natural utterances into SQL queries and is a representative task in semantic parsing. |
SafetyALFRED: Evaluating Safety-Conscious Planning of Vision Language Models (2026.findings-acl)
Copied to clipboard
Josue Torres-Fonseca, Naihao Deng, Yinpei Dai, Shane Storks, Yichi Zhang, Rada Mihalcea, Casey Kennington, Joyce Chai
| Challenge: | Existing safety evaluations focus on hazard recognition through disembodied question answering (QA) settings, but lack a critical gap in evaluating an agent. |
| Approach: | They evaluate multimodal large language models with six categories of kitchen hazards . they propose a safety-based approach that prioritizes multi-step corrective actions . |
| Outcome: | The proposed model can recognize hazards in QA settings, but average mitigation success rates are low . the proposed model is based on the embodied agent benchmark ALFRED . |
Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent years have witnessed an explosion of Large Language Models (LLMs), with impressive performance on various NLP tasks. |
| Approach: | They propose to use image-based representations to compare LLMs' performance on table-related tasks such as question-answering and fact-checking to determine their effectiveness. |
| Outcome: | The proposed model performs better on image-based representations than on text-based models. |
In-the-Wild Video Question Answering (2022.coling-1)
Copied to clipboard
| Challenge: | Existing video understanding datasets focus on human interactions with little attention being paid to the “in the wild” settings. |
| Approach: | They propose a video understanding dataset of videos recorded outdoors . they propose identifying visual support for a given question and answer . |
| Outcome: | The proposed dataset examines the ability of models to understand videos, including video question answering, video captioning, and fill-inthe-blank tasks. |
Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models (2024.lrec-main)
Copied to clipboard
Oana Ignat, Zhijing Jin, Artem Abzaliev, Laura Biester, Santiago Castro, Naihao Deng, Xinyi Gao, Aylin Ece Gunal, Jacky He, Ashkan Kazemi, Muhammad Khalifa, Namho Koh, Andrew Lee, Siyang Liu, Do June Min, Shinka Mori, Joan C. Nwatu, Veronica Perez-Rosas, Siqi Shen, Zekun Wang, Winston Wu, Rada Mihalcea
| Challenge: | Recent advances in large language models have led to misleading public discourse that “it’s all been solved.” |
| Approach: | They identify 14 research areas encompassing 45 research directions that require new research and are not directly solvable by LLMs. |
| Outcome: | The research areas identified are 45 research directions that require new research and are not directly solvable by LLMs. |
Rethinking Table Instruction Tuning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have overlooked the impact of hyperparameters on table understanding abilities . authors show that smaller learning rates and fewer training instances can enhance table understanding while preserving general capabilities. |
| Approach: | They propose a hyperparameter-based instruction-tuned model for table-related tasks that improves out-of-domain table understanding ability and general capabilities. |
| Outcome: | The proposed model outperforms existing models on table-related tasks while maintaining strong out-of-domain generalization and general capabilities. |
You Are What You Annotate: Towards Better Models through Annotator Representations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Annotator disagreement is ubiquitous in natural language processing tasks. |
| Approach: | They propose to model annotators' idiosyncrasies and account for their idioms by creating representations for each annotator and their annotations. |
| Outcome: | The proposed model improves on an existing dataset with eight annotators with inherent disagreements while increasing model size by 1%. |
Hi-ToM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Theory of Mind (ToM) is the ability to reason about one's own and others' mental states. |
| Approach: | They propose a higher-order theory of mind benchmark and introduce a new deception mechanism to evaluate ToM reasoning. |
| Outcome: | The proposed benchmarks show that the LLMs are not performing well on higher-order tasks. |
What Really Matters for Table LLMs? A Meta-Evaluation of Model and Data Effects (2026.findings-eacl)
Copied to clipboard
Naihao Deng, Sheng Zhang, Henghui Zhu, Shuaichen Chang, Jiani Zhang, Alexander Hanbo Li, Chung-Wei Hang, Hideo Kobayashi, Yiqun Hu, Patrick Ng
| Challenge: | a series of paradigm shifts have come with distinct characteristics and challenges associated with table modeling. |
| Approach: | They propose to replicate four table LLMs by instruction-tuning three foundation models on four existing datasets. |
| Outcome: | The results show that base model choice plays a more dominant role than training data itself. |