Papers by Chris Bryan
Real-Time Visual Feedback to Guide Benchmark Creation: A Human-and-Metric-in-the-Loop Workflow (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent research has shown that language models exploit ‘artifacts’ in benchmarks to solve tasks, rather than learning them, leading to inflated model performance. |
| Approach: | They propose a benchmark creation paradigm for NLP that focuses on guiding crowdworkers and provides realtime visual feedback to improve sample quality. |
| Outcome: | The proposed paradigm decreases effort, frustration, mental, and temporal demands of crowdworkers and analysts, while increasing the performance of both user groups. |
Multilingual Retrieval Augmented Generation for Culturally-Sensitive Tasks: A Benchmark for Cross-lingual Robustness (2025.findings-acl)
Copied to clipboard
Bryan Li, Fiona Luo, Samar Haider, Adwait Agashe, Siyu Li, Runqi Liu, Miranda Muqing Miao, Shriya Ramakrishnan, Yuan Yuan, Chris Callison-Burch
| Challenge: | Existing approaches to retrieval-augmented generated (RAG) can be useful in multilingual settings, but they also introduce biases in the retrieved documents. |
| Approach: | They propose a dataset of territorial disputes paired with retrieved Wikipedia documents in 49 languages to evaluate cross-lingual robustness. |
| Outcome: | The proposed paradigm helps mitigate hallucinations of large language models (LLMs). |
PAXQA: Generating Cross-lingual Question Answering Examples at Training Scale (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing question answering systems rely on large, high-quality training data. |
| Approach: | They propose a synthetic data generation method which decomposes cross-lingual QA into two stages . they apply a question generation model to the English side and annotation projection to translate both questions and answers. |
| Outcome: | The proposed method outperforms existing methods on cross-lingual QA datasets. |
This Land is Your, My Land: Evaluating Geopolitical Bias in Language Models through Territorial Disputes (2024.naacl-long)
Copied to clipboard
| Challenge: | Pretrained large language models may answer differently in different languages . this contrasts with a multilingual human, who would likely answer consistently . |
| Approach: | They propose a dataset of territorial disputes which includes multiple-choice questions in 49 languages . they propose metrics to quantify bias and consistency in responses across different languages based on their data . |
| Outcome: | The proposed model recalls certain knowledge inconsistently when asked in different languages. |