Papers by Thibault Sellam
SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation (2023.emnlp-main)
Copied to clipboard
Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, Ankur Parikh
| Challenge: | evaluating the quality of generated text is a difficult problem for large language models. |
| Approach: | They propose a dataset for multilingual, multifaceted summarization evaluation. |
| Outcome: | The proposed dataset can be used to train multilingual summarization systems . it shows that the dataset performs well on the out-of-domain meta-evaluation benchmarks TRUE and mFACE . |
A Multilingual View of Unsupervised Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Empirically, we show that our approach results in higher BLEU scores over state-of-the-art unsupervised models on the WMT’14 English-French, WMT'16 English-German, and WMT‘16 English–Romanian datasets in most directions. |
| Approach: | They propose a probabilistic framework for multilingual neural machine translation that encompasses supervised and unsupervised setups, focusing on unsupervised translation. |
| Outcome: | The proposed framework achieves higher BLEU scores than state-of-the-art unsupervised models on the WMT’14 English-French, WMT'16 English-German, and WMT‘16 English–Romanian datasets in most directions. |
Learning Compact Metrics for MT (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in machine translation and multilingual text generation have led researchers to adopt trained metrics such as COMET or BLEURT, which treat evaluation as a regression problem and use representations from multilingual pre-trained models such as XLM-RoBERTa or mBERT. |
| Approach: | They propose to use multilingual model capacity to improve model performance by transferring knowledge from one teacher to multiple students trained on related languages. |
| Outcome: | The proposed model yields 10.5% improvement over vanilla fine-tuning and reaches 92.6% of RemBERT’s performance using only a third of its parameters. |
BLEURT: Learning Robust Metrics for Text Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Text generation has made significant advances, but evaluation metrics have lagged behind. |
| Approach: | They propose a learning evaluation metric for English based on BERT . BLEURT can model human judgment with a few thousand possibly biased training examples . |
| Outcome: | The proposed model can model human judgment with a few thousand potentially biased training examples. |
Dialect-robust Evaluation of Generated Text (2023.acl-long)
Copied to clipboard
Jiao Sun, Thibault Sellam, Elizabeth Clark, Tu Vu, Timothy Dozat, Dan Garrette, Aditya Siddhant, Jacob Eisenstein, Sebastian Gehrmann
| Challenge: | Existing evaluation metrics that are not robust to dialect variation are difficult to measure for many groups of users and can penalize systems for producing text in lower-resource dialects. |
| Approach: | They propose a dialect-robust evaluation metric that produces the same score for system outputs that share the same semantics but are expressed in different dialects. |
| Outcome: | The proposed method significantly improves dialect robustness while preserving the correlation between automated metrics and human ratings. |
Reward Gaming in Conditional Text Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work has used reward functions learned from human annotations to align conditional text generation models with desired behaviors. |
| Approach: | They propose to use reinforcement learning to train conditional text generation models with reward functions learned from human annotations to align outputs with desired behaviors. |
| Outcome: | The proposed framework improves the quality of generated summaries by using saliency and faithfulness metrics. |