Papers by John Salvador
Benchmarking LLMs on Semantic Overlap Summarization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are the most capable text generation models in a variety of tasks and fields. |
| Approach: | They benchmark Large Language Models (LLMs) on SOS and introduce PrivacyPolicyPairs (3P) a dataset of 135 high-quality privacy policy documents is used to evaluate the model. |
| Outcome: | The proposed dataset complements existing resources and broadens domain coverage. |
LLMs as Meta-Reviewers’ Assistants: A Case Study (2025.naacl-long)
Copied to clipboard
Eftekhar Hossain, Sanjeev Kumar Sinha, Naman Bansal, R. Alexander Knipper, Souvika Sarkar, John Salvador, Yash Mahajan, Sri Ram Pavan Kumar Guttikonda, Mousumi Akter, Md. Mahadi Hassan, Matthew Freestone, Matthew C. Williams Jr., Dongji Feng, Santu Karmaker
| Challenge: | Meta-reviews are a critical step in the overall scientific peer-reviewed process, which focuses on understanding the consensus of expert opinions on a scholarly work and making informed judgments on its scientific merit. |
| Approach: | They propose to use large language models to generate a controlled multi-perspective-summary (MPS) of their opinions to help meta-reviewers better comprehend multiple experts' perspectives. |
| Outcome: | The proposed model can help meta-reviewers better comprehend multiple experts’ perspectives by generating a controlled multi-perspective-summary (MPS) of their opinions. |
Large Language Models for IT Automation Tasks: Are We There Yet? (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks rely on synthetic tasks that fail to capture the needs of practitioners who use IT automation tools. |
| Approach: | They evaluate 14 open-source and 3 proprietary LLMs and find that GPT-4.1-Mini achieves the best pass@10 rate of 23.9%, while Claude-3.5-Sonnet achieves best pass @1 performance. |
| Outcome: | The evaluated LLMs perform poorly in 126 tasks and show that they lack state reconciliation capabilities and lack module knowledge. |