Papers by Joykirat Singh
Exposing the Achilles’ Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations focus on final accuracy, neglecting the critical aspect of reasoning capabilities. |
| Approach: | They propose to evaluate LLMs’ abilities to detect and correct reasoning mistakes by using rule-based methods and smaller language models. |
| Outcome: | The proposed model outperforms existing models such as GPT-4o and GPT4 in both accuracy and accuracy, but lacks data contamination and memorization concerns. |
EROS:Entity-Driven Controlled Policy Document Summarization (2024.lrec-main)
Copied to clipboard
| Challenge: | a privacy policy is a crucial component of any organization that allows it to legally collect, process, store, and/or distribute personal data. |
| Approach: | They propose to use a policy-document summarization dataset to enforce the summaries to include critical privacy-related entities and organization’s rationale in collecting those entities. |
| Outcome: | The proposed model improves over baselines and qualitatively evaluates the proposed model on human and qualitative data. |
PromptWizard: Optimizing Prompts via Task-Aware, Feedback-Driven Self-Evolution (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have transformed AI across diverse domains, with prompting being central to their success in guiding model outputs. |
| Approach: | They propose a framework for discrete prompt optimization that generates human-readable prompts using feedback-driven critique and synthesis process. |
| Outcome: | The proposed framework improves prompt quality across 45 tasks and reduces API calls, token usage and overall cost. |
Data-scarce Behavior Editing of Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Prior studies show that noisy neural circuitries coexist with generalizable abilities within LLMs. |
| Approach: | a new method is proposed to improve the generalizability of large-scale web-based text models . a TaRot method is based on learnable rotation matrices optimized for Bayesian optimization . |
| Outcome: | a new method for task adaptation improves on multiple classification and generation tasks . it improves upon zero- and few-shot performance, with average improvements of 14% and 15% . |