Papers by Jekaterina Novikova
Lexical Features Are More Vulnerable, Syntactic Features Have More Predictive Power (D19-55)
Copied to clipboard
| Challenge: | Existing metrics to quantify lexical diversity have been proposed. |
| Approach: | They propose to examine how generic language characteristics are impacted by text alterations. |
| Outcome: | The proposed models show that lexical features are more sensitive to text modifications than syntactic ones. |
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022.emnlp-demos)
Copied to clipboard
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo F . R. Ribeiro, Lewis Tunstall, Li Zhang, Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou
| Challenge: | Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. |
| Approach: | They propose to use the Generation, Evaluation, and Metrics Benchmark to integrate new evaluation methods into existing evaluations. |
| Outcome: | The proposed evaluation infrastructure bridges the gap between the advantages of leaderboards and in-depth and evolving evaluations by allowing model developers to benefit from each other's work. |
RankME: Reliable Human Ratings for Natural Language Generation (N18-2)
Copied to clipboard
| Challenge: | Existing studies have shown that human evaluation for natural language generation often suffers from inconsistent user ratings. |
| Approach: | They propose a rank-based magnitude estimation method which combines continuous scales and relative assessments to improve the reliability of human ratings. |
| Outcome: | The proposed method significantly improves the reliability and consistency of human ratings compared to traditional evaluation methods. |
Detecting cognitive impairments by agreeing on interpretations of linguistic features (N19-1)
Copied to clipboard
| Challenge: | Linguistic features have shown promising applications for detecting cognitive impairments. |
| Approach: | They propose a framework to classify after reaching agreements between modalities by using linguistic features to divide linguistic subsets into subset and let neural networks learn low-dimensional representations that agree with each other. |
| Outcome: | The proposed framework outperforms existing classifiers using all of the 413 linguistic features. |