Papers by Raoyuan Zhao
Do We Know What LLMs Don’t Know? A Study of Consistency in Knowledge Probing (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for probing knowledge gaps in large language models are inconsistent and inconsistent. |
| Approach: | They propose a process based on input variations and quantitative metrics to evaluate probing methods that are inconsistent on knowledge gaps. |
| Outcome: | The proposed process exposes two dimensions of inconsistency in knowledge gap probing. |
Large Reasoning Models Are (Not Yet) Multilingual Latent Reasoners (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work shows that large reasoning models arrive at the correct answer before completing textual reasoning steps, indicating the presence of latent reasoning. |
| Approach: | They conduct a systematic investigation of multilingual latent reasoning in large reasoning models across 11 languages. |
| Outcome: | The proposed model arrive at the correct answer before completing the reasoning steps, indicating the presence of latent reasoning. |
Evaluating Robustness of Large Language Models Against Multilingual Typographical Errors (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly deployed in multilingual, real-world applications where user inputs introduce typographical errors. |
| Approach: | They propose a multilingual typo generation algorithm that simulates human-like errors based on language-specific keyboard layouts and typing behavior. |
| Outcome: | The proposed model can generate the correct answer ("500") under typos in English, German, and Russian. |
SynthEval: Hybrid Behavioral Testing of NLP Models with Synthetic Evaluation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing frameworks for benchmarking in NLP often overestimate performance . however, manually creating a variety of test types requires significant human labor . |
| Approach: | They propose a framework that leverages large language models to generate a wide range of test types . they first generate sentences via LLMs and then identifies challenging examples . |
| Outcome: | The proposed framework overestimates performance on two classification tasks. |
A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent work has examined final-answer accuracy in multilingual settings, but the behavior of thinking traces, i.e., the intermediate steps that lead to the final answer, remains underexplored. |
| Approach: | They propose to measure language compliance, answer accuracy, and answer consistency when LRMs are explicitly instructed or prompt-hacked to think in a target language. |
| Outcome: | The proposed model improves in English and other high-resource languages while relying on traces to varying degrees. |
MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are used globally across many languages, but their English-centric pretraining raises concerns about cross-lingual disparities for cultural awareness . |
| Approach: | They introduce an automatic multilingual framework for evaluating cultural awareness in large language models across languages, regions, and topics. |
| Outcome: | The framework evaluates open-ended text generation, capturing how models express culturally grounded knowledge in natural language. |
What’s the Difference? Supporting Users in Identifying the Effects of Prompt and Model Changes Through Token Patterns (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods for prompting for large language models have limitations such as being labor-intensive or lacking insights. |
| Approach: | They propose a new approach that automatically distinguishes between random variations and systematic differences in language model outputs by using token patterns. |
| Outcome: | The proposed method combines both automation and human analysis to provide new insights into established prompt data. |