Papers with Gemini-Pro
MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks (2024.naacl-long)
Copied to clipboard
Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram
| Challenge: | Several new LLMs have been introduced necessitating their evaluation on non-English languages. |
| Approach: | They perform a thorough evaluation of the non-English capabilities of SoTA LLMs by comparing them on the same set of multilingual datasets. |
| Outcome: | The proposed model outperforms models on multilingual datasets on 22 languages including low-resource African languages. |
On the Reliability of Psychological Scales on Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on examining Large Language Models’ characteristics from a psychological standpoint, acknowledging the necessity of understanding their behavioral characteristics. |
| Approach: | They propose to examine the reliability of personality tests to LLMs by using psychological scales. |
| Outcome: | The proposed model can represent diverse personalities with specific prompt instructions. |
TCProF:Time-Complexity Prediction SSL Framework (2025.naacl-long)
Copied to clipboard
| Challenge: | determining the precise time complexity of a code is theoretically undecidable . determining time complexity is a challenging task in programming efficiency analysis . |
| Approach: | They propose a time-complexity prediction SSL framework that uses code snippets labeled with their time complexity classes to predict code time. |
| Outcome: | The proposed approach improves performance by 60% over self-training methods. |
HAIC: Improving Human Action Understanding and Generation with Better Captions for Multi-modal Large Language Models (2025.acl-long)
Copied to clipboard
Xiao Wang, Jingyun Hua, Weihong Lin, Yuanxing Zhang, Fuzheng Zhang, Jianlong Wu, Di Zhang, Liqiang Nie
| Challenge: | Existing studies have shown that high-quality video captions can improve MLLMs' performance on videos involving human actions. |
| Approach: | They propose a data annotation pipeline to collect videos featuring clear human actions from the Internet and annotate them in a standardized caption format that uses human attributes to distinguish individuals. |
| Outcome: | The proposed pipeline combines two datasets to evaluate human action understanding. |
Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models (2024.emnlp-main)
Copied to clipboard
Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varshney, Chitta Baral
| Challenge: | Existing logical reasoning evaluation benchmarks focus on simplistic single-step or multi-step reasoning with limited set of inference rules. |
| Approach: | They propose to use a multi-step logical reasoning evaluation dataset to measure their ability for human-like multi- step logical thinking. |
| Outcome: | The proposed dataset covers three logic types including propositional, first-order, and non-monotonic logic with various inference rules and depths. |