Papers by Kamalika Das
Learning to Search Effective Example Sequences for In-Context Learning (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods address these factors in isolation, overlooking their interdependencies. Existing approaches focus on sequence selection, while focusing on the sequence of examples. |
| Approach: | They propose a method that considers key factors involved in sequence selection and incrementally builds the sequence. |
| Outcome: | Experiments across various datasets and language models show that the proposed method significantly reduces the search space and improves performance. |
Gradient-guided Attention Map Editing: Towards Efficient Contextual Hallucination Mitigation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) often experience “contextual hallucination” where they prioritize self-generated content over input context, leading to a disregard for pertinent details. |
| Approach: | They propose a method that dynamically adjusts attention maps to enhance contextual relevance by using a trained classifier to identify attention maps likely to induce hallucinations. |
| Outcome: | The proposed approach reduces hallucinations across open-source models on summarization and open-book QA tasks. |
SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for hallucination detection rely on self-consistency check alone . prominent LMs exhibit a tendency to produce exceedingly confident, but erroneous, assertions . |
| Approach: | They propose a sampling-based method that expands on the principle of self-consistency checking to detect hallucinations at question-level and model-level. |
| Outcome: | The proposed method outperforms the state of the art in detecting non-factual and factual statements across multiple question-answering and open-domain generation benchmarks. |
Synthetic Knowledge Ingestion: Towards Knowledge Refinement and Injection for Enhancing Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models capture factual knowledge across a wide range of domains, but refining their capabilities on previously seen knowledge remains a challenge. |
| Approach: | They propose a synthetic knowledge ingestion method that leverages fine-grained synthesis and interleaved generation to construct high-quality data representations from raw knowledge sources. |
| Outcome: | The proposed method outperforms baseline methods on question-answering tasks spanning finance, biomedicine, and open-generation domains. |
The Behavior Gap: Evaluating Zero-shot LLM Agents in Complex Task-Oriented Dialogs (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that LLM-based agents struggle to perform in zero-shot scenarios. |
| Approach: | They propose a framework to quantify the behavior gap between AI agents and human experts . they propose to examine discrepancies in dialog acts, tool usage, and knowledge utilization . |
| Outcome: | The proposed framework measures the behavior gap between AI agents and human experts on task-oriented dialogs. |
Heuristic-based Search Algorithm in Automatic Instruction-focused Prompt Optimization: A Survey (2025.findings-acl)
Copied to clipboard
Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley A. Malin, Sricharan Kumar
| Challenge: | Recent advances in Large Language Models (LLMs) have led to remarkable achievements across a variety of NLP tasks. |
| Approach: | They propose a taxonomy of automatic prompt optimization methods that explore and improve prompts with minimal human oversight. |
| Outcome: | The proposed methods can explore and improve prompts with minimal human oversight. |
SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Large language models have a tendency to make confidently wrong predictions, highlighting the need for uncertainty quantification (UQ) . previous studies focused on aleatoric uncertainty, but the full spectrum of uncertainties, including epistemic, remains inadequately explored. |
| Approach: | They propose a method to quantify uncertainty in large language models (LLMs) they use a set of perturbations and an aggregation module to generalize the method. |
| Outcome: | The proposed method improves model uncertainty calibration and reduces expected calibration error by 50% on average. |
SEE: Strategic Exploration and Exploitation for Cohesive In-Context Prompt Optimization (2025.acl-long)
Copied to clipboard
Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley A. Malin, Sricharan Kumar
| Challenge: | Existing approaches separate the optimization of prompt instructions and in-context learning examples, leading to incohesive, suboptimal results. |
| Approach: | They propose a framework that refines both prompt instructions and in-context learning examples. |
| Outcome: | The proposed framework outperforms state-of-the-art prompt optimization methods on 35 benchmark tasks. |
Survival of the Safest: Towards Secure Prompt Optimization through Interleaved Multi-Objective Evolution (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Experimental evaluations across diverse benchmark datasets affirm ‘s efficacy in delivering high performance and notably enhancing safety and security compared to single-objective methods. |
| Approach: | They propose a multi-objective prompt optimization framework that integrates semantic, feedback, and crossover mutations to effectively traverse the prompt landscape. |
| Outcome: | The proposed framework enhances both performance and security in large language models while keeping computational demands low. |
Do You Know What You Are Talking About? Characterizing Query-Knowledge Relevance For Reliable Retrieval Augmented Generation (2024.emnlp-main)
Copied to clipboard
Zhuohang Li, Jiaxin Zhang, Chao Yan, Kamalika Das, Sricharan Kumar, Murat Kantarcioglu, Bradley Malin
| Challenge: | Language models suffer from poor interpretability and transparency, as well as the intrinsic risk of hallucination and misinformation. |
| Approach: | They propose a statistical framework that assesses how well a query can be answered by an RAG system by capturing the relevance of knowledge. |
| Outcome: | The proposed framework assesses how well a query can be answered by an RAG system by capturing the relevance of knowledge. |
RIMRULE: Improving Tool-Using Language Agents via MDL-Guided Rule Learning (2026.acl-long)
Copied to clipboard
Xiang Gao, Yuguang Yao, Qi Zhang, Kaiwen Dong, Avinash Baidya, Ruocheng Guo, Hilaf Hasson, Kamalika Das
| Challenge: | Large language models (LLMs) struggle to use tools reliably in domain-specific settings. |
| Approach: | They propose a neuro-symbolic approach to adapt large language models to task-specific tools . they propose reusable rules that are distilled from failure traces and injected into the prompt . |
| Outcome: | Experiments show that the proposed approach outperforms prompting-based adaptation methods and complements finetuning. |
Divide-Conquer-Reasoning for Consistency Evaluation and Automatic Improvement of Large Language Models (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for evaluating the quality and consistency of text generated by Large Language Models are not effective. |
| Approach: | They propose a divide-conquer-reasoning approach to evaluate LLM-generated texts using a split-and-conquers evaluator and an automatic metric converter to facilitate this approach. |
| Outcome: | The proposed framework outperforms state-of-the-art methods by a large margin on multiple benchmarks and reduces 90% of output inconsistencies in one iteration. |