Papers by Vitaly Nikolaev
SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation (2023.emnlp-main)
Copied to clipboard
Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, Ankur Parikh
| Challenge: | evaluating the quality of generated text is a difficult problem for large language models. |
| Approach: | They propose a dataset for multilingual, multifaceted summarization evaluation. |
| Outcome: | The proposed dataset can be used to train multilingual summarization systems . it shows that the dataset performs well on the out-of-domain meta-evaluation benchmarks TRUE and mFACE . |
Planning with Learned Entity Prompts for Abstractive Summarization (2021.tacl-1)
Copied to clipboard
| Challenge: | a simple but flexible mechanism is used to ground the generation of abstractive summaries. |
| Approach: | They propose a mechanism to learn an intermediate plan to ground the generation of abstractive summaries. |
| Outcome: | The proposed model outperforms state-of-the-art methods for faithfulness on CNN and BillSum. |
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022.emnlp-demos)
Copied to clipboard
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo F . R. Ribeiro, Lewis Tunstall, Li Zhang, Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou
| Challenge: | Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. |
| Approach: | They propose to use the Generation, Evaluation, and Metrics Benchmark to integrate new evaluation methods into existing evaluations. |
| Outcome: | The proposed evaluation infrastructure bridges the gap between the advantages of leaderboards and in-depth and evolving evaluations by allowing model developers to benefit from each other's work. |
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020.tacl-1)
Copied to clipboard
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, Jennimaria Palomaki
| Challenge: | Existing models for multilingual modeling are based on a set of typological features that are used to express meaning in languages such as English. |
| Approach: | They present a question-answer-typed question-referenced dataset that covers 11 typologically diverse languages with 204K question-and-answered pairs. |
| Outcome: | The proposed dataset covers 11 typologically diverse languages with 204K question-answer pairs. |
TaTA: A Multilingual Table-to-Text Dataset for African Languages (2023.findings-emnlp)
Copied to clipboard
Sebastian Gehrmann, Sebastian Ruder, Vitaly Nikolaev, Jan Botha, Michael Chavinda, Ankur Parikh, Clara Rivera
| Challenge: | Existing data-to-text generation datasets are limited to English and a small number of other languages. |
| Approach: | They create the first large multilingual table-to-text dataset with a focus on African languages. |
| Outcome: | The proposed dataset includes 8,700 examples in nine languages including four African languages and a zero-shot test language. |
The Morpho-syntactic Annotation of Animacy for a Dependency Parser (L18-1)
Copied to clipboard
| Challenge: | Animacy is a feature found in nouns such as 'gender', 'number' and 'case' that improves parser accuracy. |
| Approach: | They propose an annotation scheme and parser results for the animacy feature in Russian and Arabic, morphologically rich languages, using the universal dependency framework. |
| Outcome: | The proposed scheme and parser improve on the animacy feature in Russian and Arabic, and the results show that the feature is more accurate than other features found in nouns, namely, 'gender', , and 'number' |
Improving homograph disambiguation with supervised machine learning (L18-1)
Copied to clipboard
| Challenge: | a new system for text-to-speech synthesis uses rule-based homograph disambiguation . a simple application of machine learning produces significant improvements in homograph ambiguity . |
| Approach: | They propose a rule-based homograph disambiguation system for text-to-speech synthesis at Google . they compare it to a new system which performs disambiguations using classifiers trained on labeled data . |
| Outcome: | The proposed system is more accurate than hand-written rules or machine learning alone. |