Papers by Jiahao Ying

C-ICL: Contrastive In-context Learning for Information Extraction (2024.findings-emnlp)

Copied to clipboard

Ying Mo, Jiahao Liu, Jian Yang, Qifan Wang, Shun Zhang, Jingang Wang, Zhoujun Li

Challenge:	Existing methods for in-context learning with large language models focus on using correct or negative examples, ignoring the potential value of incorrect or negative samples.
Approach:	They propose a few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations.
Outcome:	The proposed technique outperforms previous few-shot in-context learning methods on a broad spectrum of related tasks.

Intuitive or Dependent? Investigating LLMs’ Behavior Style to Conflicting Prompts (2024.acl-long)

Copied to clipboard

Jiahao Ying, Yixin Cao, Kai Xiong, Long Cui, Yidong He, Yongbin Liu

Responsible & Trustworthy NLP

Challenge:	Extensive experiments with seven Large Language Models reveal their varying behaviors.
Approach:	They investigate the behaviors of Large Language Models when faced with conflicting prompts versus their internal memory.
Outcome:	Extensive experiments with seven LLMs reveal their varying behaviors.

Disentangling Language and Culture for Evaluating Multilingual Large Language Models (2025.acl-long)

Copied to clipboard

Jiahao Ying, Wei Tang, Yiran Zhao, Yixin Cao, Yu Rong, Wenxuan Zhang

Language Models

Challenge:	Extensive evaluations of large language models (LLMs) are conducted on a wide range of models, revealing a notable cultural-linguistic synergy phenomenon, where models exhibit better performance when questions are culturally aligned with the language.
Approach:	They propose a Dual Evaluation Framework to comprehensively assess the multilingual capabilities of large language models by decomposing evaluation along dimensions of linguistic medium and cultural context.
Outcome:	The proposed framework allows for a nuanced analysis of LLMs’ ability to process questions within both native and cross-cultural contexts cross-lingually.

LLMs-as-Instructors: Learning from Errors Toward Automating Model Improvement (2024.findings-emnlp)

Copied to clipboard

Jiahao Ying, Mingbao Lin, Yixin Cao, Wei Tang, Bo Wang, Qianru Sun, Xuanjing Huang, Shuicheng Yan

Challenge:	Using advanced Large Language Models, instructors can improve training of smaller models by analyzing their own model's errors.
Approach:	They propose a framework that leverages advanced Large Language Models to enhance training of smaller target models.
Outcome:	The proposed framework outperforms ChatGPT on multiple benchmarks and shows that it improves on both in-domain and out-of-domain benchmarks.

Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots (2025.findings-naacl)

Copied to clipboard

Chengyue Wu, Zhixuan Liang, Yixiao Ge, Qiushan Guo, Zeyu Lu, Jiahao Wang, Ying Shan, Ping Luo

Multimodality

Challenge:	Multi-modal Large Language Models have shown remarkable progress in visual contexts, yet their ability to convert visual figures into executable code remains underexplored.
Approach:	They propose to use a set of visual coding metrics to assess MLLMs' visual . pass rate, text-match ratio, and GPT-4V rating judgement to assess the quality of generated code and rendered images.
Outcome:	The proposed benchmark includes 132 high-quality matplotlib plots across six plot types, as well as 150 and 86 plots from Python’s and R’s plotly libraries respectively, totaling 368 plots.

SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia (2025.findings-naacl)

Copied to clipboard

Chaoqun Liu, Wenxuan Zhang, Jiahao Ying, Mahani Aljunied, Anh Tuan Luu, Lidong Bing

Multilinguality

Challenge:	Large Language Models (LLMs) have shown remarkable performance across various English benchmarks, including both human exam datasets such as MMLU and instruction-following datasets.
Approach:	They introduce two new benchmarks to evaluate the capabilities of Large Language Models in Southeast Asian (SEA) application scenarios.
Outcome:	The proposed benchmarks show that they can discern LLM performance on SEA language tasks compared to their translated benchmarks.

LLaMA Pro: Progressive LLaMA with Block Expansion (2024.acl-long)

Copied to clipboard

Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, Ping Luo

Multimodality

Challenge:	Existing studies have demonstrated that pre-trained LLMs are limited in certain domains, such as programming, mathematics, biomedical, or finance.
Approach:	They propose a new post-pretraining method with an expansion of Transformer blocks to tune the expanded blocks using only new corpus, efficiently and effectively improving the model’s knowledge while mitigating forgetting.
Outcome:	The proposed model outperforms existing models in programming and math and its instruction-following counterpart LLaMA Pro-8.3B in general tasks, programming, and mathematics.

A + B: A General Generator-Reader Framework for Optimizing LLMs to Unleash Synergy Potential (2024.findings-acl)

Copied to clipboard

Wei Tang, Yixin Cao, Jiahao Ying, Bo Wang, Yuyue Zhao, Yong Liao, Peng Zhou

Information Retrieval

Challenge:	Retrieval-Augmented Generation (RAG) is an effective solution to supplement necessary knowledge to large language models.
Approach:	They propose a "generate-then-read" pipeline to replace retrieval stage with generation from the LLM itself.
Outcome:	The proposed framework outperforms single models in the base and chat versions and addresses safety and helpfulness post-adaptation challenges.

CompTab: A Comprehensive Benchmark for Real-World TableQA with Complex Reasoning and Irregular Tables (2026.acl-long)

Copied to clipboard

Zhen Yang, Wei Du, Jie Wang, Wenze Zhou, Xiangfeng Meng, Zhengyang Wang, Suping Sun, Ziwei Du, Haodong Zou, Jie Chen, Yongbin Liu, Shicheng Tan, Jiahao Ying, Shu Zhao

Multimodality

Challenge:	Existing benchmarks focus on well-structured tables and fail to reflect irregular structures and complex reasoning commonly encountered in real-world scenarios.
Approach:	They propose a benchmark to evaluate TableQA under complex reasoning and irregular table conditions.
Outcome:	The proposed framework improves generalization and realism of large language models under complex and irregular table conditions.

QRMeM: Unleash the Length Limitation through Question then Reflection Memory Mechanism (2024.findings-emnlp)

Copied to clipboard

Bo Wang, Heyan Huang, Yixin Cao, Jiahao Ying, Wei Tang, Chong Feng

Question Answering

Challenge:	Existing methods for processing large textual content face insufficient adaptation to task-specific needs and missing multi-segmentation relationships.
Approach:	They propose a question then reflection memory mechanism which integrates a dual-structured memory pool and a structured graph guidance to facilitate a reflective trial-and-error approach for navigating and identifying relevant segments.
Outcome:	The proposed model achieves superior performance on multiple-choice questions and multi-doc QA.

EvoWiki: Evaluating LLMs on Evolving Knowledge (2025.acl-long)

Copied to clipboard

Wei Tang, Yixin Cao, Yang Deng, Jiahao Ying, Bo Wang, Yizhe Yang, Yuyue Zhao, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang, Yong Liao

Challenge:	Existing knowledge evolution benchmarks are static and fail to capture the evolving nature of LLMs and knowledge.
Approach:	They propose an evolving dataset that categorizes information into stable, evolved, and uncharted states.
Outcome:	The proposed dataset is auto-updatable and enables evaluation of continuously changing knowledge and newly released LLMs.