Papers by Raj Shah
LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing (2024.emnlp-main)
Copied to clipboard
Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Cheng Jiayang, Zhaowei Wang, Ying Su, Raj Shah, Ruohao Guo, Jing Gu, Haoran Li, Kangda Wei, Zihao Wang, Lu Cheng, Surangika Ranathunga, Meng Fang, Jie Fu, Fei Liu, Ruihong Huang, Eduardo Blanco, Yixin Cao, Rui Zhang, Philip Yu, Wenpeng Yin
| Challenge: | a comparative analysis of paper (meta-)reviews by large language models (LLMs) aims to identify and distinguish LLMs from human activities . |
| Approach: | They present a comparative analysis to identify and distinguish LLM activities from human activities. |
| Outcome: | The proposed analysis aims to improve recognition of instances when someone implicitly uses LLMs for reviewing activities. |
KG-MuLQA: A Framework for KG-based Multi-Level QA Extraction and Long-Context LLM Evaluation (2026.acl-long)
Copied to clipboard
Nikita Tatarinov, Vidhyakshaya Kannan, Haricharana Srinivasa, Arnav Raj, Harpreet Singh Anand, Varun Singh, Aditya Luthra, Ravij Lade, Agam Shah, Sudheer Chava
| Challenge: | KG-MulQA extracts QA pairs at multiple complexity levels along three key dimensions: multi-hop retrieval, set operations, and answer plurality. |
| Approach: | They propose a framework that extracts QA pairs at multiple complexity levels along three key dimensions: multi-hop retrieval, set operations, and answer plurality. |
| Outcome: | The framework extracts QA pairs at multiple complexity levels along key dimensions . it enables fine-grained assessment of model performance across controlled difficulty levels. |
Development of Cognitive Intelligence in Pre-trained Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show evidence for emergent cognitive abilities in Large Pre-trained Language Models (PLMs). Prior research into emergental cognitive abilities of PLMs has been path-independent to model training. |
| Approach: | They use four task categories to examine the alignment of ten popular families of PLMs and evaluate their performance to the developmental trajectories of children's thinking. |
| Outcome: | The results show that the models are more aligned to children's thinking than previous studies. |
When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain (2022.emnlp-main)
Copied to clipboard
Raj Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, Diyi Yang
| Challenge: | Pre-trained language models have shown impressive performance on a variety of tasks and domains. |
| Approach: | They propose a domain specific financial LANGuage model which uses financial keywords and phrases for better masking. |
| Outcome: | The proposed model outperforms existing models on a variety of tasks and domains. |
Multi-Level Feedback Generation with Large Language Models for Empowering Novice Peer Counselors (2024.acl-long)
Copied to clipboard
| Challenge: | Existing mechanisms of providing feedback rely on human supervision . existing mechanisms of delivering feedback largely rely only on human oversight . |
| Approach: | They propose to leverage large language models to provide contextualized feedback to peer counselors . they construct a publicly available dataset with detailed feedback annotations of 400 conversations . |
| Outcome: | The proposed method minimizes the risk of potentially harmful and low-quality feedback generation. |
Numeric Magnitude Comparison Effects in Large Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Prior research on the representational capabilities of LLMs evaluates whether they show human-level performance. |
| Approach: | They ask how well popular LLMs capture the magnitudes of numbers from a behavioral lens. |
| Outcome: | The proposed model captures the magnitudes of numbers from a behavioral lens. |