Papers by Mohith Kulkarni
Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models (2024.emnlp-main)
Copied to clipboard
Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varshney, Chitta Baral
| Challenge: | Existing logical reasoning evaluation benchmarks focus on simplistic single-step or multi-step reasoning with limited set of inference rules. |
| Approach: | They propose to use a multi-step logical reasoning evaluation dataset to measure their ability for human-like multi- step logical thinking. |
| Outcome: | The proposed dataset covers three logic types including propositional, first-order, and non-monotonic logic with various inference rules and depths. |
Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter? (2024.emnlp-main)
Copied to clipboard
Nemika Tyagi, Mihir Parmar, Mohith Kulkarni, Aswin Rrv, Nisarg Patel, Mutsumi Nakamura, Arindam Mitra, Chitta Baral
| Challenge: | Existing studies evaluate only the final predicted answer of a puzzle, without providing any finer metrics to evaluate them. |
| Approach: | They propose to use a grid-based evaluation dataset to evaluate LLMs' reasoning abilities and a new error taxonomy to evaluate their reasoning chains. |
| Outcome: | The proposed model outperforms existing prompting methods on a wide range of natural language understanding tasks previously thought to be exclusive to humans. |