Papers by Jiao Sun
Investigating the Benefits of Free-Form Rationales (2022.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that crowdsourced rationales provide additional background knowledge to models . a qualitative study shows generated rationale is not as useful for humans as crowdsourced ones . |
| Approach: | They investigate whether crowdsourced rationales provide additional background knowledge to models . they find that ECQA rationale provides additional background information to understand a decision . |
| Outcome: | The results show that ECQA rationales provide additional background knowledge to understand a decision . compared to crowdsourced rationale, generated rationale is not as useful for humans . |
Effective and Efficient Query-aware Snippet Extraction for Web Search (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to extract webpage snippets ignore contextual information of webpages, which may be sub-optimal. |
| Approach: | They propose a query-aware webpage snippet extraction method called DeepQSE that captures contextual information of webpages. |
| Outcome: | The proposed method can significantly improve the performance of DeepQSE without affecting its performance. |
TableVista: Benchmarking Multimodal Table Reasoning under Visual and Structural Complexity (2026.findings-acl)
Copied to clipboard
Zheyuan Yang, Liqiang Shang, Junjie Chen, Xun Yang, Chenglong Xu, Bo Yuan, Chenyuan Jiao, Yaoru Sun, Yilun Zhao
| Challenge: | TableVista evaluates multimodal table reasoning under visual and structural complexity . current models struggle to maintain reasoning consistency when structural complexity combined with visually integrated presentations. |
| Approach: | They propose a benchmark for evaluating multimodal table reasoning under visual and structural complexity. |
| Outcome: | The proposed model performs poorly on visual and structural complexity. |
WebClipper: Efficient Evolution of Web Agents with Graph-based Trajectory Pruning (2026.acl-long)
Copied to clipboard
Junjie Wang, Zequn Xie, Dan Yang, Jie Feng, Yue Shen, Duolin Sun, Meixiu Long, Yihan Jiao, Zhehao Tan, Jian Wang, Peng Wei, Jinjie Gu
| Challenge: | Open-source web agents rely on long tool-call trajectories with cyclic reasoning loops and exploration of unproductive branches. |
| Approach: | They propose a framework that compresses web agent trajectories via graph-based pruning. |
| Outcome: | The proposed framework reduces tool-call rounds by 20% while improving accuracy and efficiency while maintaining the same level of performance as existing models. |
DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedback (2025.naacl-long)
Copied to clipboard
Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Herrmann, Sjoerd Van Steenkiste, Ranjay Krishna, Cyrus Rashtchian
| Challenge: | Text-to-Image models (T2I) still struggle to produce images that are both aesthetically pleasing and faithful to the user’s input text. |
| Approach: | They propose a training algorithm that trains T2I models to be faithful to the input text. |
| Outcome: | The proposed model improves both the semantic alignment and aesthetic appeal of two diffusion-based T2I models, evidenced by multiple benchmarks (+1.7% on TIFA, +2.9% on DSG1K, +3.4% on VILA aesthetic). |
ExPUNations: Augmenting Puns with Keywords and Explanations (2022.emnlp-main)
Copied to clipboard
Jiao Sun, Anjali Narayan-Chen, Shereen Oraby, Alessandra Cervone, Tagyoung Chung, Jing Huang, Yang Liu, Nanyun Peng
| Challenge: | Puns add the challenge of fusing commonsense and world knowledge with the ability to interpret lexical-semantic ambiguity. |
| Approach: | They propose to augment existing datasets with detailed crowdsourced annotations of puns, keywords and fine-grained funniness ratings to challenge current models' ability to understand and generate humor. |
| Outcome: | The proposed tasks include explanation generation to aid with pun classification and keyword-conditioned pun generation to challenge state-of-the-art models' ability to understand and generate humor. |
HIRAG: Hierarchical-Thought Instruction-Tuning Retrieval-Augmented Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | In-depth research on the specific capabilities needed by the RAG generation model is lacking, leading to inconsistent document quality and retrieval system imperfections. |
| Approach: | They propose that RAG models should possess three progressively hierarchical abilities: (1) Filtering: the ability to select relevant information; (2) Combination: the capability to combine semantic information across paragraphs; (3) RAG-specific reasoning: the capacity to further process external knowledge using internal knowledge. |
| Outcome: | Experiments show that the proposed method significantly improves the model’s open-book examination capability on datasets such as RGB, PopQA, MuSiQue, HotpotQA, and PubmedQA. |
AESOP: Paraphrase Generation with Adaptive Syntactic Control (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for paraphrase generation use fixed syntactic structures for all input sentences. |
| Approach: | They propose to add syntactical control to a pretrained language model to generate fluent paraphrases using a retrieval-based selection module. |
| Outcome: | The proposed model achieves state-of-the-art on semantic preservation and syntactic conformation on two benchmark datasets with ground-truth syntaktic control from human-annotated exemplars. |
“Kelly is a Warm Person, Joseph is a Role Model”: Gender Biases in LLM-Generated Reference Letters (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are an effective tool to assist individuals in writing documents. |
| Approach: | They examine gender biases in large language models (LLMs)-generated reference letters . they find that models are biased because they are hallucinated . |
| Outcome: | The proposed model-generated reference letters are evaluated on 2 popular LLMs- ChatGPT and Alpaca. |
Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermark (2023.acl-long)
Copied to clipboard
Wenjun Peng, Jingwei Yi, Fangzhao Wu, Shangxi Wu, Bin Bin Zhu, Lingjuan Lyu, Binxing Jiao, Tong Xu, Guangzhong Sun, Xing Xie
| Challenge: | Large language models (LLMs) have demonstrated exceptional abilities in both text understanding and generation. |
| Approach: | They propose an Embedding Watermark method that implants backdoors on embeddings to protect copyright of large language models. |
| Outcome: | The proposed method protects the copyright of large language models without compromising service quality while minimizing the adverse impact on the original embeddings’ utility. |
Evaluating Large Language Models on Controlled Generation Tasks (2023.emnlp-main)
Copied to clipboard
Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, Xuezhe Ma
| Challenge: | Recent studies have looked into the ability of large language models in various benchmark tasks, including question generation, reading comprehension, multilingual and etc. However, few studies investigate the controllability of large languages. |
| Approach: | They propose to compare large language models with state-of-the-start finetuned smaller models to find that large language model controls are comparable to smaller models. |
| Outcome: | The proposed model can meet hard constraints and perform better than state-of-the-art models. |
EventPlus: A Temporal Event Understanding Pipeline (2021.naacl-demos)
Copied to clipboard
Mingyu Derek Ma, Jiao Sun, Mu Yang, Kung-Hsiang Huang, Nuan Wen, Shikhar Singh, Rujun Han, Nanyun Peng
| Challenge: | Event information is a type of common sense knowledge that helps people understand how stories evolve and provides predictive hints for future events. |
| Approach: | They propose a temporal event understanding pipeline that integrates state-of-the-art components. |
| Outcome: | The proposed pipeline can be easily adapted to other domains, including biomedical domains. |
On Measures of Biases and Harms in NLP (2022.findings-aacl)
Copied to clipboard
Sunipa Dev, Emily Sheng, Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Jiin Kim, Akihiro Nishi, Nanyun Peng, Kai-Wei Chang
| Challenge: | Recent studies show that natural language processing (NLP) technologies propagate societal biases about demographic groups associated with attributes such as gender, race, and nationality. |
| Approach: | They propose a framework for harms and questions to help practitioners understand biases . they propose measurable measures to detect and mitigate biased groups . |
| Outcome: | The proposed framework provides a framework for harms and questions for practitioners to answer to guide the development of bias measures. |
Men Are Elected, Women Are Married: Events Gender Bias on Wikipedia (2021.acl-short)
Copied to clipboard
| Challenge: | Disproportional event distributions can manifest and amplify social stereotypes . researchers have been using NLP tools to analyze corpora for various tasks on online platforms. |
| Approach: | They propose to scrape a corpus of career and personal life descriptions with demographic information from 10,412 celebrities to facilitate the study. |
| Outcome: | The proposed model detects gender biases in a corpus of career and personal life descriptions and calibrates the results using strategically generated templates. |
From Shijing to English and German: Resources and Evaluation for LLM Translation of Early Chinese Poetry (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) show promise in literary translation, but their performance in poetry remains unexplored. |
| Approach: | They propose a framework that integrates knowledge-driven, rule-based, and LLM-as-judge metrics into a Shijing corpus . their code, lexical KB, and corpus reconstruction protocols are available at https://github.com/ML-KULeuven/ShijingLLMTrans. |
| Outcome: | The proposed framework achieves higher human correlation than traditional metrics and high statistical stability. |
ESTER: A Machine Reading Comprehension Dataset for Reasoning about Event Semantic Relations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent event-centric reading comprehension datasets focus mostly on event arguments or temporal relations. |
| Approach: | They propose a machine reading comprehension dataset that leverages natural language queries to reason about the five most common event semantic relations. |
| Outcome: | The proposed dataset shows that current SOTA systems achieve 22.1%, 63.3% and 83.5% for token-based exact-match, **F1** and event-based **HIT@1** scores. |
SkillVerse : Assessing and Enhancing LLMs with Tree Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | Language models evolve to tackle complex, multifaceted tasks, requiring granular evaluations . recent studies have focused on leaderboard and benchmark results, but limited interpretability makes it difficult to compare strengths and weaknesses of models. |
| Approach: | They propose an unsupervised tree-structured diagnosis framework for understanding model proficiency in specific abilities with an LLM as a judge. |
| Outcome: | The proposed framework improves model in-context learning and predicts model weaknesses with a 55% success rate compared to the framework without SkillVerse. |
GroupRank: A Groupwise Paradigm for Effective and Efficient Passage Reranking with LLMs (2026.findings-acl)
Copied to clipboard
Meixiu Long, Duolin Sun, Dan Yang, Yihan Jiao, Lei Liu, Jiahai Wang, Binbin Hu, Yue Shen, Jie Feng, Zhehao Tan, Junjie Wang, Lianzhen Zhong, Jian Wang, Peng Wei, Jinjie Gu
| Challenge: | Existing rankers excel in lexical-matching scenarios, while they struggle with complex queries requiring deep reasoning. |
| Approach: | They propose a new paradigm that balances flexibility and context awareness to unlock the full potential of groupwise reranking. |
| Outcome: | The proposed approach achieves a state-of-the-art 65.2 NDCG@10 on BRIGHT and surpasses baselines by 2.1 points on R2MED while delivering a 6.4 inference speedup. |
Towards Robust NLG Bias Evaluation with Syntactically-diverse Prompts (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Past studies have shown biases in natural language generation systems but there has been little work on evaluating the bias evaluation approaches. |
| Approach: | They propose a method for evaluating biases in natural language generation systems by paraphrasing syntactic prompts with different syntaktic structures and paraphrazing them to evaluate demographic bias. |
| Outcome: | The proposed method is more robust and shows that some syntactic structures prompt more toxic content while others could prompt less biased generation. |
Dialect-robust Evaluation of Generated Text (2023.acl-long)
Copied to clipboard
Jiao Sun, Thibault Sellam, Elizabeth Clark, Tu Vu, Timothy Dozat, Dan Garrette, Aditya Siddhant, Jacob Eisenstein, Sebastian Gehrmann
| Challenge: | Existing evaluation metrics that are not robust to dialect variation are difficult to measure for many groups of users and can penalize systems for producing text in lower-resource dialects. |
| Approach: | They propose a dialect-robust evaluation metric that produces the same score for system outputs that share the same semantics but are expressed in different dialects. |
| Outcome: | The proposed method significantly improves dialect robustness while preserving the correlation between automated metrics and human ratings. |
Alternated Training with Synthetic and Authentic Data for Neural Machine Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to synthesizing data in NMT focus on leveraging monolingual data in training. |
| Approach: | They propose alternated training with synthetic and authentic data to improve NMT models' performance. |
| Outcome: | The proposed approach improves Chinese-English and German-English translation tasks over strong baselines. |
Context-Situated Pun Generation (2022.emnlp-main)
Copied to clipboard
Jiao Sun, Anjali Narayan-Chen, Shereen Oraby, Shuyang Gao, Tagyoung Chung, Jing Huang, Yang Liu, Nanyun Peng
| Challenge: | a new task for context-situated pun generation uses a given context to generate puns . human evaluation shows that 69% of top retrieved pun words can be used to generate context-based puns. |
| Approach: | They propose a task where puns are generated based on contextual keywords and pun words. |
| Outcome: | The proposed system generates successful puns 31% of the time given a plausible tuple of context words and pun pairs. |