Papers by Haidar Khan
ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition (2025.acl-demo)
Copied to clipboard
| Challenge: | Recent studies highlight the effectiveness of game-based evaluations for Large Language Models. |
| Approach: | They propose a dynamic, competition-based evaluation framework for Large Language Models that leverages competitive games. |
| Outcome: | The framework leverages competitive games to evaluate models in large language models. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
Low-Resource Compositional Semantic Parsing with Concept Pretraining (2023.eacl-main)
Copied to clipboard
| Challenge: | Semantic parsing is a key role in voice assistants by mapping natural language to structured meaning representations. |
| Approach: | They propose an architecture to perform domain adaptation automatically with only a small amount of metadata about the new domain and without any new training data. |
| Outcome: | The proposed architecture outperforms existing models in low-resource settings. |
When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards (2024.acl-long)
Copied to clipboard
Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan AlRashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, M Saiful Bari, Haidar Khan
| Challenge: | Existing leaderboards are often taken at face value, but this is costly . a recent study shows that minor perturbations to the benchmark result in rankings up to 8 positions. |
| Approach: | They propose to use a *hybrid* scoring method for answer selection for large language models . they find that minor perturbations to the benchmark result in rankings changes . |
| Outcome: | The proposed model is a hybrid scoring method, the authors argue . the proposed model could be used to improve the performance of large language models . |
Compressing Transformer-Based Semantic Parsing Models using Compositional Code Embeddings (2020.findings-emnlp)
Copied to clipboard
Prafull Prakash, Saurabh Kumar Shashidhar, Wenlong Zhao, Subendhu Rongali, Haidar Khan, Michael Kayser
| Challenge: | Existing task-oriented semantic parsing models use BERT or RoBERTa as pretrained encoders. |
| Approach: | They propose to learn compositional code embeddings to greatly reduce the sizes of BERT and RoBERTa encoders. |
| Outcome: | The proposed model reduces the size of BERT and RoBERTa encoders while maintaining performance. |
Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning (2023.acl-short)
Copied to clipboard
Mustafa Ozdayi, Charith Peris, Jack FitzGerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, Rahul Gupta
| Challenge: | Large Language Models memorize significant portions of training data, which poses privacy risk. |
| Approach: | They propose a prompt-tuning approach to control the extraction rates of memorized content in large language models. |
| Outcome: | The proposed techniques yield 9.3% increase in extraction rate compared to baseline model . the proposed defense achieves 97.7% reduction with a perplexity increase of 16.9% . |
Controlled Data Generation via Insertion Operations for NLU (2022.naacl-industry)
Copied to clipboard
| Challenge: | a new approach to annotate live traffic is emerging to be cost-effective and efficient . manual data annotation is expensive and not preferred for meeting customer privacy expectations . |
| Approach: | They propose a targeted synthetic data generation technique by inserting tokens into a given semantic signature. |
| Outcome: | The proposed approach achieves the same accuracy as training with all available data on a voice assistant dataset. |