Papers by Vikas Raunak
Leveraging GPT-4 for Automatic Translation Post-Editing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural Machine Translation models still require translation post-editing to rectify errors and enhance quality under critical settings. |
| Approach: | They use GPT-4 to automatically post-edit NMT outputs across several language pairs . they show that GPT4 is adept at translation post- editing, producing meaningful edits . |
| Outcome: | The proposed translation post-editor improves on state-of-the-art language models on English-Chinese, English-German, Chinese-English and German-English language pairs. |
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022.emnlp-demos)
Copied to clipboard
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo F . R. Ribeiro, Lewis Tunstall, Li Zhang, Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou
| Challenge: | Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. |
| Approach: | They propose to use the Generation, Evaluation, and Metrics Benchmark to integrate new evaluation methods into existing evaluations. |
| Outcome: | The proposed evaluation infrastructure bridges the gap between the advantages of leaderboards and in-depth and evolving evaluations by allowing model developers to benefit from each other's work. |
Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translation (2025.acl-short)
Copied to clipboard
| Challenge: | Memorization of noisy training data creates unexpected failure modes in neural machine translation models, thus presenting a reliability risk when deploying them in the real world. |
| Approach: | They propose a modification to sequence-level knowledge distillation (SeqKD) that intervenes in SeqKd to reduce memorization and hallucinations. |
| Outcome: | The proposed modification reduces memorization and hallucinations in the student model . |
SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window (2024.naacl-short)
Copied to clipboard
| Challenge: | prevailing evaluation methods for machine translation metrics are at the sentence-level . but there are many linguistic phenomena that cannot be translated without context . |
| Approach: | They propose a metric that leverages a moving window that slides over a document to feed it into a quality estimation model. |
| Outcome: | The proposed metric obtains significantly higher pairwise system accuracy than its sentence-level baseline. |
Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks (2021.naacl-main)
Copied to clipboard
| Challenge: | ESPnet framework exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks. |
| Approach: | They propose a framework that exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks. |
| Outcome: | The proposed framework outperforms the state-of-the-art on speech translation tasks by +6 and +3 BLEU on the two test sets of Fisher-CallHome and +4 BLUE on the English-German and English-French test sets. |
SALTED: A Framework for SAlient Long-tail Translation Error Detection (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Traditional machine translation metrics are insensitive to the long tail of behavioral problems. |
| Approach: | They propose a specification-based framework for behavioral testing of NMT models . they use high-precision detectors that flag errors between a source sentence and output . |
| Outcome: | The proposed framework provides a reliable view of problems that were previously invisible. |
Dissecting In-Context Learning of Translations in GPT-3 (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on fewshot prompting in Large Language Models has focused on selecting the few-shot samples for prompting. |
| Approach: | They propose a method to add demonstration attributes to prompting in machine translations by perturbations of high-quality in-domain demonstrations. |
| Outcome: | The proposed method improves upon the zero-shot translation performance of GPT-3, even making it competitive with few-shot prompted translations. |
The Curious Case of Hallucinations in Neural Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) suffers from well known pathologies such as coverage, mistranslation of named entities, etc. |
| Approach: | They propose a theory that explains hallucinations under source perturbation and a method that generates hallucines under corpus-level noise without any source perturbations. |
| Outcome: | The proposed hypothesis is validated by a corpus-level noise analysis and is validateable in other datasets. |
Finding Memo: Extractive Memorization in Constrained Sequence Generation Tasks (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Memorization presents a challenge for constrained Natural Language Generation tasks . previous studies focused on counterfactual memorization, linking it to hallucinations . |
| Approach: | They propose an algorithm for extractive memorization in constrained sequence generation tasks . they propose to elicit non-memorized translations of memorized samples from the same model . |
| Outcome: | The proposed algorithm could be leveraged to mitigate memorization in the model through finetuning. |
Do GPTs Produce Less Literal Translations? (2023.acl-short)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are general-purpose language models capable of many natural language generation or understanding tasks. |
| Approach: | They investigate how LLMs differ qualitatively from standard Neural Machine Translation models by measuring literalness and monotonicity. |
| Outcome: | The proposed models achieve close to state-of-the-art translation performance under few-shot prompting . the results are backed up by human evaluations and a newer MT quality metrics . |
On Long-Tailed Phenomena in Neural Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art Neural Machine Translation models struggle with generating low-frequency tokens, tackling which remains a major challenge. |
| Approach: | They propose a loss function to better adapt model training to structural dependencies of conditional text generation by incorporating inductive biases of beam search into the training process. |
| Outcome: | The proposed method leads to significant gains over cross-entropy across different language pairs, especially on the generation of low-frequency words. |