Papers by Parker Riley
Finding Replicable Human Evaluations via Stable Ranking Probability (2024.naacl-long)
Copied to clipboard
| Challenge: | a recent study shows that human evaluation is the best way to rank natural language generation systems . human raters can exhibit different behaviors when rating outputs, causing ranking to be unstable . stability is the degree to which a specific evaluation methodology produces the same system ranking when repeated. |
| Approach: | They propose to evaluate results through the lens of stability: stability is the degree to which a specific evaluation methodology produces the same system ranking when repeated. |
| Outcome: | The proposed model is based on a dataset of multi-segment translations rated by multiple professionals . human raters can exhibit different behaviors when rating NLG outputs, the study shows . |
FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation (2023.tacl-1)
Copied to clipboard
Parker Riley, Timothy Dozat, Jan A. Botha, Xavier Garcia, Dan Garrette, Jason Riesa, Orhan Firat, Noah Constant
| Challenge: | a new dataset and evaluation benchmark for Few-shot Region-aware Machine Translation is presented . FRMT is a type of style-targeted translation that uses labeled training data to perform tasks. |
| Approach: | They propose a dataset and evaluation benchmark for Few-shot Region-aware Machine Translation. |
| Outcome: | The proposed model is based on two translations from English into Portuguese and Mandarin Chinese. |
Outside Computation with Superior Functions (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing generalized inside-outside algorithm would violate Strong Exponential Time Hypothesis (SETH) |
| Approach: | They propose a framework for efficient outside computation that would yield a sub-exponential time algorithm for SAT, violating the Strong Exponential Time Hypothesis (SETH). |
| Outcome: | The proposed algorithm would violate the Strong Exponential Time Hypothesis (SETH) . |
Orthographic Features for Bilingual Lexicon Induction (P18-2)
Copied to clipboard
| Challenge: | Recent embedding-based methods do not take advantage of orthographic features, such as edit distance, which can be helpful for pairs of related languages. |
| Approach: | They propose to use orthographic features to integrate orthographic induction into embedding methods . they use document-aligned data instead of a seed dictionary to learn bilingual embedds . |
| Outcome: | This work extends embedding-based methods to incorporate orthographic features . it shows that the methods can learn bilingual embeddables in low-resource languages . |
TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted Restyling (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text style transfer require style-labeled training data, but use only labeled data at inference time. |
| Approach: | They propose a method that uses readily-available unlabeled text to train style transfer . they use a style vector to condition a decoder to perform style transfer using unlabelled text . |
| Outcome: | The proposed method is competitive on sentiment transfer, even compared to models trained fully on labeled data. |
XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages (2023.findings-emnlp)
Copied to clipboard
Sebastian Ruder, Jonathan Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Jean-Michel Sarr, Xinyi Wang, John Wieting, Nitish Gupta, Anna Katanova, Christo Kirov, Dana Dickinson, Brian Roark, Bidisha Samanta, Connie Tao, David Adelani, Vera Axelrod, Isaac Caswell, Colin Cherry, Dan Garrette, Reeve Ingle, Melvin Johnson, Dmitry Panteleev, Partha Talukdar
| Challenge: | Existing datasets are often informed by established research directions in the NLP community. |
| Approach: | They propose a benchmark to evaluate the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks. |
| Outcome: | The proposed benchmark evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks. |
Enhancing Human Evaluation in Machine Translation with Comparative Judgement (2025.acl-long)
Copied to clipboard
| Challenge: | Human evaluation is crucial for assessing rapidly evolving language models but is influenced by annotator proficiency and task design. |
| Approach: | They evaluate three annotation setups to integrate comparative judgment into human annotation for machine translation. |
| Outcome: | The proposed approach improves inter-annotator agreement and stability of the annotations. |
MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation (2026.acl-long)
Copied to clipboard
| Challenge: | a critical component of machine translation model development is evaluating model quality. |
| Approach: | They propose a two-stage version of the current translation evaluation paradigm (MQM) they propose re-annotation, which uses raters to review and edit annotations . |
| Outcome: | The proposed method improves annotation quality by finding errors missed in the first pass. |
WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects (2025.findings-acl)
Copied to clipboard
Daniel Deutsch, Eleftheria Briakou, Isaac Rayburn Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, Markus Freitag
| Challenge: | In order to evaluate large language models (LLMs), it is important to collect benchmark datasets in order to assess their multilingual performance. |
| Approach: | They extend the WMT24 dataset to cover 55 languages by collecting new human-written references and post-edits for 46 new languages/dialects. |
| Outcome: | The proposed dataset covers 55 languages and provides best-performing MT systems in all 55 languages. |
Translationese as a Language in “Multilingual” NMT (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work examines the impact of translationese in machine translation evaluation using the WMT evaluation campaign. |
| Approach: | They propose to use a sentence-level classifier to distinguish translationese from original target text to generate a machine translation model that can produce more natural outputs at test time. |
| Outcome: | The proposed model produces more natural outputs at test time, yielding gains in human evaluation scores on accuracy and fluency. |