Papers by Kevin Robinson
MiTTenS: A Dataset for Evaluating Gender Mistranslation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on gender mistranslation in translation systems have highlighted the problem . a dataset of 26 languages is presented to measure the extent of such errors . |
| Approach: | They propose a dataset that measures the extent of gender mistranslation in translation systems . they use handcrafted passages that target known failure patterns and synthetically generated passages . |
| Outcome: | The proposed dataset covers 26 languages from a variety of language families and scripts, including several traditionally under-represented in digital resources. |
Bias in Language Models: Beyond Trick Tests and Towards RUTEd Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | Standard bias benchmarks are used for large language models to measure the association between social attributes and single-word outputs. |
| Approach: | They adapt three standard bias metrics of next-word prediction to measure gender-occupation bias and develop an analogous RUTEd evaluation in three contexts of real-world LLM use. |
| Outcome: | The proposed benchmarks are robust to lengthening model outputs via a more realistic user prompt in the domain of gender-occupation bias. |
AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are rapidly becoming more and more popular, but dealing with the potential harms associated with their deployment in real-world scenarios is still an open research question. |
| Approach: | They propose an automated approach for automated generation of adversarial evaluation datasets to test the safety of LLM generations on new downstream applications. |
| Outcome: | AART generates evaluation datasets with high diversity of content characteristics critical for effective adversarial testing. |
A Pretrainer’s Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity (2024.naacl-long)
Copied to clipboard
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, Daphne Ippolito
| Challenge: | a large number of pretraining data design practices are under-documented, authors say . authors: strong performance of modern language models depends on selfsupervised pretraining . |
| Approach: | They propose to pretrain models on data curated at different collection times . they find temporal shift between evaluation data and pretraining data leads to performance degradation . |
| Outcome: | The results validate, quantify, and expose many undocumented intuitions about text pretraining . authors say this practice has outperformed other models in the field . |