Papers by Amanpreet Singh
SciRepEval: A Multi-Format Benchmark for Scientific Document Representations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for evaluating scientific document representations fail to capture the diversity of relevant tasks. |
| Approach: | They propose a benchmark for training and evaluating scientific document representations that includes 24 challenging and realistic tasks across four formats: classification, regression, ranking and search. |
| Outcome: | The proposed model outperforms existing models by over 2 points absolute. |
PaperMage: A Unified Toolkit for Processing, Representing, and Manipulating Visually-Rich Scientific Documents (2023.emnlp-demo)
Copied to clipboard
Kyle Lo, Zejiang Shen, Benjamin Newman, Joseph Chang, Russell Authur, Erin Bransom, Stefan Candra, Yoganand Chandrasekhar, Regan Huff, Bailey Kuehl, Amanpreet Singh, Chris Wilhelm, Angele Zamarron, Marti A. Hearst, Daniel Weld, Doug Downey, Luca Soldaini
| Challenge: | Existing tools for working with scientific documents are limited and documents are often in difficult-to-use PDF formats. |
| Approach: | They propose an open-source Python toolkit for analyzing and processing visually-rich scientific documents. |
| Outcome: | PaperMage provides turn-key recipes for common scientific document processing use-cases. |
Ai2 Scholar QA: Organized Literature Synthesis with Attribution (2025.acl-demo)
Copied to clipboard
Amanpreet Singh, Joseph Chee Chang, Dany Haddad, Aakanksha Naik, Jena D. Hwang, Rodney Kinney, Daniel S Weld, Doug Downey, Sergey Feldman
| Challenge: | Ai2 Scholar QA is a free online scientific question answering application . it uses retrieval-augmented generation to answer complex scientific questions . many of these systems are expensive to use and closed-source . |
| Approach: | They propose a retrieval-augmented generation-based scientific question answering application . it uses a Python package and an interactive web app to make the entire pipeline public . they compare it with other similar question-answering applications . |
| Outcome: | The proposed system outperforms other systems on a recent scientific QA benchmark. |
Dynabench: Rethinking Benchmarking in NLP (2021.naacl-main)
Copied to clipboard
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, Adina Williams
| Challenge: | Dynabench is an open-source platform for dynamic dataset creation and model benchmarking. |
| Approach: | They propose an open-source platform for dynamic dataset creation and model benchmarking. |
| Outcome: | The proposed platform can be used to create models that fail on simple challenges and falter in real-world scenarios. |
SpecNFS: A Challenge Dataset Towards Extracting Formal Models from Natural Language Specifications (2022.lrec-1)
Copied to clipboard
Sayontan Ghosh, Amanpreet Singh, Alex Merenstein, Wei Su, Scott A. Smolka, Erez Zadok, Niranjan Balasubramanian
| Challenge: | Existing methods for building formal semantic representations of specification texts are laborious and error-prone. |
| Approach: | They propose to use SpecIR to model sentences appearing in NFS specification documents as IF-THEN statements and introduce a representation language to parse them. |
| Outcome: | The proposed models achieve an F1 score of only 60.5 and 33.3 when using a state-of-the-art language model. |
Embedding Recycling for Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on embedding recycling have not adequately account for overhead costs. |
| Approach: | They propose to reuse contextualized embeddings from previous runs to speed training and inference of future ones. |
| Outcome: | The proposed technique speeds training and inference with no impact on accuracy. |
LMUNIT: Fine-grained Evaluation with Natural Language Unit Tests (2025.findings-emnlp)
Copied to clipboard
Jon Saad-Falcon, Rajan Pathe Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, Shikib Mehri
| Challenge: | Using natural language unit tests, language models are costly and noisy, and automated metrics provide only coarse, difficult-to-interpret signals. |
| Approach: | They propose a paradigm that decomposes response quality into explicit, testable criteria and a unified scoring model, LMUnit, which combines multi-objective training across preferences, direct ratings, and natural language rationales. |
| Outcome: | The proposed paradigm significantly improves inter-annotator agreement and enables more effective LLM development workflows. |
Language Models Don’t Know What You Want: Evaluating Personalization in Deep Research Needs Real Users (2026.acl-long)
Copied to clipboard
Nishant Balepur, Malachi Hamada, Varsha Kishore, Sergey Feldman, Amanpreet Singh, Pao Siangliulue, Joseph Chee Chang, Eunsol Choi, Jordan Lee Boyd-Graber, Aakanksha Naik
| Challenge: | Earlier research used real users to push personalization, but easy-to-use judges have been criticized for not adopting online studies. |
| Approach: | They propose a personalized action-following tool that infers a user's research interests and proposes personalized actions for a query. |
| Outcome: | The proposed tool beats baselines in citation metrics and personalized action-following with an online version of MySQA. |
TOPICAL: TOPIC Pages AutomagicaLly (2024.naacl-demo)
Copied to clipboard
| Challenge: | Topic pages aggregate useful information about an entity or concept into a single concise article. |
| Approach: | They propose a web app that generates topic pages for biomedical entities on demand . they use large language models and retrieval-augmented generation to generate high-quality topics . |
| Outcome: | The proposed method is based on a human evaluation of 150 biomedical topics . it uses large language models and retrieval-augmented generation (RAG) |