Papers by Crystina Zhang
Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models exhibit positional bias in how they use context, which affects listwise ranking. |
| Approach: | They propose a method to marginalize out different list orders in the prompt to produce an order-independent ranking with less positional bias. |
| Outcome: | The proposed method improves on five datasets in sorting and passage reranking by 34-52% . it marginalizes out different list orders in the prompt to produce an order-independent ranking . |
The Role of Mixed-Language Documents for Multilingual Large Language Model Pretraining (2026.acl-long)
Copied to clipboard
Jiandong Shao, Raphael Tang, Crystina Zhang, Karin Sevegnani, Pontus Stenetorp, Jianfei Yang, Yao Lu
| Challenge: | Existing research suggests that multilingual large language models can achieve impressive cross-lingual understanding despite largely monolingual pretraining. |
| Approach: | They compare a monolingual-only corpus with a standard web corpus that removes all multilingual documents and then retrain the models from scratch under controlled conditions. |
| Outcome: | The results show that removing bilingual data causes translation performance to drop 56% in BLEU, whereas code-switching contributes minimally. |
CELI: Simple yet Effective Approach to Enhance Out-of-Domain Generalization of Cross-Encoders. (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing cross-encoders do not capture all information into the [CLS] token . Xiong et al., 2021) find that the out-of-domain approach is less robust. |
| Approach: | They introduce a cross-encoder with late interaction that incorporates a late interaction layer into existing models. |
| Outcome: | The proposed method improves BEIR by 5% without compromising in-domain effectiveness or search latency. |
“Knowing When You Don’t Know”: A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation (2024.findings-emnlp)
Copied to clipboard
Nandan Thakur, Luiz Bonifacio, Crystina Zhang, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Boxing Chen, Mehdi Rezagholizadeh, Jimmy Lin
| Challenge: | Prior work on RAG grounds Large Language Models to reduce factual hallucinations lacks a comprehensive evaluation of different language families. |
| Approach: | They propose a human-annotated dataset for evaluating LLM robustness in RAG . they find that most models struggle to balance the two capacities . |
| Outcome: | The proposed dataset includes both a non-relevant and a relevant subset. |
Words Worth a Thousand Pictures: Measuring and Understanding Perceptual Variability in Text-to-Image Generation (2024.emnlp-main)
Copied to clipboard
Raphael Tang, Crystina Zhang, Lixinyu Xu, Yao Lu, Wenyan Li, Pontus Stenetorp, Jimmy Lin, Ferhan Ture
| Challenge: | Current diffusion models do not cover recent models, thus we curate three test sets for evaluation. |
| Approach: | They propose a human-calibrated measure of variability in a set of images bootstrapped from existing image-pair perceptual distances. |
| Outcome: | The proposed model outperforms nine baselines by 18 points in accuracy and matches graded human judgements 78% of the time. |
FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culture (2024.emnlp-main)
Copied to clipboard
Wenyan Li, Crystina Zhang, Jiaang Li, Qiwei Peng, Raphael Tang, Li Zhou, Weijia Zhang, Guimin Hu, Yifei Yuan, Anders Søgaard, Daniel Hershcovich, Desmond Elliott
| Challenge: | FoodieQA is a manually curated, fine-grained image-text dataset capturing the intricate features of food cultures across various regions in China. |
| Approach: | They evaluate vision–language Models and large language models on unseen food images and corresponding questions. |
| Outcome: | The proposed dataset evaluates vision–language Models and large language models on unseen food images and corresponding questions. |
BrowseComp-Plus: A Fair and Disentangled Evaluation Benchmark for Deep Search Agents (2026.acl-long)
Copied to clipboard
Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Sahel Sharifymoghaddam, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Hosna Oyarhoseini, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, Jimmy Lin
| Challenge: | Existing benchmarks for deep search agents rely on blackbox web search APIs . dynamic and opaque web APIs hinder reproducibility and fair comparisons - authors . |
| Approach: | They propose a benchmark that employs a fixed corpus for controlled retrieval for deep search agents. |
| Outcome: | The new benchmark shows that agents that combine large language models with retrieval tools excel at complex, reasoning-intensive queries. |
Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Using LLMs to identify false negatives improves retrieval and reranker models by 0.7-1.4 points on BEIR and by 1.7-1.8 points on AIR-Bench evaluation. |
| Approach: | They use a simple, cost-effective approach to identify and relabel false negatives in training datasets. |
| Outcome: | The proposed approach improves retrieval models by 0.7-1.4 points on BEIR and by 1.7-1.8 points on AIR-Bench evaluation. |
Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts? (2025.findings-naacl)
Copied to clipboard
| Challenge: | a recent study shows that human understanding of text depends on general semantic concepts of words that are robust to their superficial forms. |
| Approach: | They evaluate the accuracy of multilingual multilingual language models based on subword-level semantics . they form "semantic tokens" by merging semantically similar subwords and embeddings based upon the results . |
| Outcome: | The proposed models are able to make predictions on multilingual tasks with different tokenizers and model sizes. |