Papers by Shruti Singh
The Inefficiency of Language Models in Scholarly Retrieval: An Experimental Walk-through (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing work does not critically analyze the scientific language models to the best of our knowledge. |
| Approach: | They evaluate scientific language models in handling short-query texts and textual neighbors by leveraging perturbations to generate textual neighbor classes. |
| Outcome: | The proposed model is ineffective for retrieving documents for short-query texts under the most relaxed conditions. |
LEGOBench: Scientific Leaderboard Generation Benchmark (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a growing number of papers make it difficult to stay informed about the latest state-of-the-art research. |
| Approach: | They propose a benchmark to evaluate systems that generate scientific leaderboards . they use 22 years of submission data on arXiv and 11k machine learning leaderboard data on paperswithcode . |
| Outcome: | The proposed model shows significant performance gaps in the LEGOBench model . the model is based on a language model and four graph-based leaderboard generation task configuration . |
TweeNLP: A Twitter Exploration Portal for Natural Language Processing (2021.acl-demo)
Copied to clipboard
| Challenge: | Currently, Twitter curates 19,395 tweets from various NLP conferences and general NLP discussions. |
| Approach: | They propose to integrate tweets pertaining to research papers with the NLPExplorer scientific literature search engine to organize Twitter's natural language processing data. |
| Outcome: | The proposed system curates 19,395 tweets from various NLP conferences and general discussions. |
SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers (2024.emnlp-main)
Copied to clipboard
| Challenge: | SciDQA is a dataset for question-answering that challenges language models to deeply understand scientific articles. |
| Approach: | They propose a new dataset for reading comprehension that challenges language models to deeply understand scientific articles consisting of 2,937 QA pairs. |
| Outcome: | The SciDQA dataset is based on 2,937 QA pairs and decontextualizes the content, tracks the source document across different versions, and incorporates a bibliography for multi-document question-answering. |
SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature (2025.emnlp-main)
Copied to clipboard
David Wadden, Kejian Shi, Jacob Morrison, Alan Li, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, Doug Downey, Hannaneh Hajishirzi, Arman Cohan
| Challenge: | ScIRIFF is the only entirely expert-written instruction-following dataset for scientific literature understanding . it features complex instructions with long input contexts, detailed task descriptions, and structured outputs. |
| Approach: | They present a dataset of 137K instruction-following instances for training and evaluation . they finetuned large language models using a mix of general domain and ScIRIFF instructions . |
| Outcome: | The proposed dataset shows that on nine out-of-distribution held-out tasks, the model performs better than baselines trained on general domain instructions. |