Papers by Srini Narayanan
A Benchmark for Reasoning with Spatial Prepositions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Spatial reasoning is a fundamental building block of human cognition . large language models (LLMs) are not on par with advanced aspects of human cognitive domains . |
| Approach: | They propose a benchmark to assess inferential properties of statements with spatial prepositions . they use prompt engineering to test the performance of two large language models . |
| Outcome: | The proposed benchmark shows that none of the models reaches human performance. |
UGIF-DataSet: A New Dataset for Cross-lingual, Cross-modal Sequential actions on the UI (2024.findings-naacl)
Copied to clipboard
| Challenge: | Identifying the right help document, understanding instructions from the document, and using them to resolve the issue at hand is challenging. |
| Approach: | They propose to use help documents to create step-by-step tutorials overlaid on the phone UI to overcome challenges in retrieval, parsing, and grounding in multilingual-multimodal setting. |
| Outcome: | The proposed dataset contains 4,184 tasks across 8 languages and shows that the end-to-end completion rate drops from 48% in English to 32% for other languages. |
MiQA: A Benchmark for Inference on Metaphorical Questions (2022.aacl-short)
Copied to clipboard
| Challenge: | a benchmark is proposed to assess the capability of large language models to reason with conventional metaphors. |
| Approach: | They propose to assess the capability of large language models to reason with conventional metaphors. |
| Outcome: | The proposed benchmark compares pre-trained models on binary-choice tasks with human models . the results show that human models perform better on the largest model, compared to small models based on the same task . |