Papers by Bingsheng Yao
Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language Explanations (2023.acl-long)
Copied to clipboard
| Challenge: | Human-annotated labels and explanations are critical for training explainable NLP models. |
| Approach: | They propose a metric that measures the usefulness of an explanation for model performance at both fine-tuning and inference. |
| Outcome: | The proposed metric can evaluate the quality of human-annotated explanations, while Simulatability falls short. |
It is AI’s Turn to Ask Humans a Question: Question-Answer Pair Generation for Children’s Story Books (2022.acl-long)
Copied to clipboard
| Challenge: | Existing question answering (QA) techniques are created mainly to answer questions asked by humans, but in educational applications, teachers often need to decide what questions to ask . |
| Approach: | They propose to use a fairytale-themed storybook as input to generate QA pairs that can test a student's comprehension skills. |
| Outcome: | The proposed system outperforms state-of-the-art QAG baseline systems and builds an interactive story-telling application for the future real-world deployment. |
Fantastic Questions and Where to Find Them: FairytaleQA – An Authentic Dataset for Narrative Comprehension (2022.acl-long)
Copied to clipboard
Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Li, Nora Bradford, Branda Sun, Tran Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, Mark Warschauer
| Challenge: | Existing QA datasets rarely distinguish fine-grained reading skills, such as the understanding of varying narrative elements. |
| Approach: | They propose to use FairytaleQA to generate 10,580 questions based on 278 children-friendly stories to assess model's fine-grained learning skills. |
| Outcome: | The proposed dataset consists of 10,580 questions derived from 278 children-friendly stories, covering seven types of narrative elements or relations. |
StorySparkQA: Expert-Annotated QA Pairs with Real-World Knowledge for Children’s Story-Based Learning (2024.emnlp-main)
Copied to clipboard
Jiaju Chen, Yuxuan Lu, Shao Zhang, Bingsheng Yao, Yuanzhe Dong, Ying Xu, Yunyao Li, Qianwen Wang, Dakuo Wang, Yuling Sun
| Challenge: | Existing story reading systems fail to capture the nuances of how education experts think when conducting interactive story reading activities. |
| Approach: | They propose to use existing question-answering (QA) datasets to capture experts' annotations and thinking process to construct a story-based annotation framework. |
| Outcome: | The proposed framework captures experts’ annotations and thinking process and can be used to generate 5, 868 expert-annotated QA pairs with real-world knowledge. |
Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecture (2023.findings-emnlp)
Copied to clipboard
Bingsheng Yao, Ishan Jindal, Lucian Popa, Yannis Katsis, Sayan Ghosh, Lihong He, Yuxuan Lu, Shashank Srivastava, Yunyao Li, James Hendler, Dakuo Wang
| Challenge: | Existing low-resource learning techniques focus on label annotation while neglecting the natural language explanation of a data point. |
| Approach: | They propose a novel architecture that leverages an explanation-generation model to produce explanations guided by human explanations and a prediction model that utilizes generated explanations toward prediction faithfully. |
| Outcome: | The proposed architecture produces explanations guided by human explanations, a prediction model that utilizes generated explanations toward prediction faithfully, and a data diversity-based AL sampling strategy that benefits from the explanation annotations. |
Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent advances in open-domain question answering (ODQA) have led to human-level performance on many datasets. |
| Approach: | They provide a comprehensive and quantitative analysis about the difficulty of book QA . they compare the results of their research with extensive ODQA experiments . |
| Outcome: | The proposed model outperforms existing models on event-oriented questions on the NarrativeQA dataset. |
LLMs for Now, Fine-Tuning for Later: An Ensemble Approach to Data Drift in Domain-Specific Tasks (2026.acl-srw)
Copied to clipboard
Yuxuan Lu, Bingsheng Yao, Shao Zhang, Yisi Sang, Yun Wang, Hansu Gu, Peng Zhang, Tun Lu, Toby Jia-Jun Li, Dakuo Wang
| Challenge: | Deploying machine learning models in domain-specific scenarios is challenged by data drift and the scarcity of expert annotations. |
| Approach: | They propose a system that combines an LLM, an AL-assisted compact model and an automatic switch module to assist the active learning process. |
| Outcome: | The proposed system achieves 96–98% switch accuracy and outperforms both models used alone. |
Large Language Models with Temporal Reasoning for Longitudinal Clinical Summarization and Prediction (2025.findings-emnlp)
Copied to clipboard
Maya Kruse, Shiyue Hu, Nicholas Derby, Yifu Wu, Samantha Stonbraker, Bingsheng Yao, Dakuo Wang, Elizabeth M. Goldberg, Yanjun Gao
| Challenge: | Recent advances in large language models have shown potential in clinical text summarization, but their ability to handle long patient trajectories with multi-modal data spread across time remains underexplored. |
| Approach: | They evaluate open-source large language models, their Retrieval Augmented Generation variants and chain-of-thought prompting on long-context clinical summarization and prediction. |
| Outcome: | The proposed models can synthesize structured and unstructured EHR data while reasoning over temporal coherence. |
Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data (2026.acl-long)
Copied to clipboard
Yuxuan Lu, Jing Huang, Yan Han, Bingsheng Yao, Sisong Bei, Yaochen Xie, Yisi Sang, Qi He, Dakuo Wang
| Challenge: | Recent research shows that LLM Agents can generate “believable” human behaviors via prompt-only methods, leaving open questions of whether they can accurately generate step-by-step actions in multi-turn interaction tasks. |
| Approach: | They propose to use shopping data to evaluate LLMs' ability to accurately generate step-by-step actions in a multi-turn interaction task. |
| Outcome: | The proposed model achieves 17.26% action generation accuracy and 33.86% F1 score on final purchase prediction, representing improvements of 5.4% and 13.85% over baselines. |
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022.emnlp-demos)
Copied to clipboard
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo F . R. Ribeiro, Lewis Tunstall, Li Zhang, Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou
| Challenge: | Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. |
| Approach: | They propose to use the Generation, Evaluation, and Metrics Benchmark to integrate new evaluation methods into existing evaluations. |
| Outcome: | The proposed evaluation infrastructure bridges the gap between the advantages of leaderboards and in-depth and evolving evaluations by allowing model developers to benefit from each other's work. |
More Samples or More Prompts? Exploring Effective Few-Shot In-Context Learning for LLMs with In-Context Sampling (2024.findings-naacl)
Copied to clipboard
Bingsheng Yao, Guiming Chen, Ruishi Zou, Yuxuan Lu, Jiachen Li, Shao Zhang, Yisi Sang, Sijia Liu, James Hendler, Dakuo Wang
| Challenge: | Existing studies on LLM prompting focus on selecting a better set of data samples inside one single prompt input, but why not design and leverage multiple ICL prompts together to further improve the LLM’s performance? |
| Approach: | They propose a low-resource LLM prompting technique to optimize the construction of multiple ICL prompt inputs to produce confident predictions. |
| Outcome: | The proposed technique can produce confident predictions by optimizing the construction of multiple ICL prompt inputs on four NLI datasets and one QA dataset. |
A Corpus for Commonsense Inference in Story Cloze Test (2022.lrec-1)
Copied to clipboard
| Challenge: | Story Cloze Test (SOTA) models can achieve over 90% accuracy on predicting the last sentence, but high accuracy can be achieved by merely using surface-level features. |
| Approach: | They constructed a human-labeled and human-verified commonsense knowledge inference dataset using data from 1871 stories and three human workers labeled each story. |
| Outcome: | The proposed models can achieve 90% accuracy on predicting the last sentence, but they don't perform well on new and more challenging tasks. |
Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performance (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) provide superior summarization quality, but their high computational resource requirements limit practical use applications. |
| Approach: | They evaluate 19 small language models for news summarization across 2,000 news samples . they find that top-performing models achieve comparable results to those of 70B LLMs . |
| Outcome: | The proposed models achieve comparable results to 70B LLMs while generating more concise summaries. |
Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation (2026.acl-long)
Copied to clipboard
Jiaju Chen, Yuxuan Lu, Xiaojie Wang, Huimin Zeng, Jing Huang, Jiri Gesi, Ying Xu, Bingsheng Yao, Dakuo Wang
| Challenge: | Existing "LLM-as-a-judge" evaluation frameworks are limited by persona descriptions and are not generalizable to other tasks. |
| Approach: | They propose a framework that can automatically construct multiple evaluator personas with distinct dimensions from relevant text documents and instantiate LLM agents with the persona. |
| Outcome: | The proposed framework can believably simulate human evaluators . it extracts stakeholders' diverse perspectives from the provided research papers and constructs personas for the agents . |
‘Don’t Get Too Technical with Me’: A Discourse Structure-Based Framework for Automatic Science Journalism (2023.emnlp-main)
Copied to clipboard
| Challenge: | Science journalism is the production of journalistic content covering scientific topics that are not covered in the scientific literature. |
| Approach: | They propose to use a dataset to generate a scientific paper's tuples, a summary snippet and a novel technical framework to integrate a paper' s discourse structure with its metadata to guide generation. |
| Outcome: | The proposed system outperforms baseline methods in elaborating a content plan meaningful for the target audience, simplifying the information selected, and producing a coherent final report in a layman’s style. |
Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents (2025.findings-acl)
Copied to clipboard
| Challenge: | Role-Playing Agents (RPAs) are increasingly popular due to diverse task requirements and agent designs. |
| Approach: | They propose an evidence-based evaluation design guideline for LLM-based RPAs based on agent attributes, task attributes, and evaluation metrics. |
| Outcome: | The proposed evaluation design guideline is based on a systematic review of 1,676 papers published between Jan. 2021 and Dec. 2024. |