Papers by Vishvak Murahari
QualEval: Qualitative Evaluation for Model Improvement (2024.naacl-long)
Copied to clipboard
Vishvak Murahari, Ameet Deshpande, Peter Clark, Tanmay Rajpurohit, Ashish Sabharwal, Karthik Narasimhan, Ashwin Kalyan
| Challenge: | Quantitative evaluation metrics are inadequate for large language models due to complexity of tasks and cannot provide actionable diagnostics. |
| Approach: | They propose a quantitative evaluation tool called QualEval that uses automated qualitative evaluation as a vehicle for model improvement. |
| Outcome: | The proposed method improves the performance of the Llama 2 model by 15% compared to baselines. |
MUX-PLMs: Data Multiplexing for High-throughput Language Models (2023.findings-emnlp)
Copied to clipboard
Vishvak Murahari, Ameet Deshpande, Carlos Jimenez, Izhak Shafran, Mingqiu Wang, Yuan Cao, Karthik Narasimhan
| Challenge: | MUX-PLMs are high-throughput pre-trained language models that can be fine-tuned for any downstream task to yield high-performance. |
| Approach: | They propose to train language models with data multiplexing to achieve 2x/5x inference speedup . they use multiplexers to entangle and disentangle inputs to achieve the same performance . |
| Outcome: | MUX-PLMs achieve 2x/5x inference speedup with 1-4 % drop on broad suite of tasks. |
Toxicity in chatgpt: Analyzing persona-assigned language models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown incredible capabilities and transcended the natural language processing community. |
| Approach: | They evaluate toxicity in over half a million generations of ChatGPT by assigning it a persona . they find that outputs engage in incorrect stereotypes, harmful dialogue, hurtful opinions . |
| Outcome: | a new study shows that assigning a persona to a chatbot can increase toxicity in half a million generations. |
PruMUX: Augmenting Data Multiplexing with Model Compression (2023.findings-acl)
Copied to clipboard
| Challenge: | Prior work has investigated methods like model pruning, knowledge distillation, and data multiplexing to increase model throughput without sacrificing accuracy. |
| Approach: | They propose to combine structured pruning and data multiplexing methods to increase model throughput without sacrificing accuracy. |
| Outcome: | The proposed method achieves 7.5-29.5X throughput improvement over a BERT-base model with accuracy threshold from 80% to 74%. |
PersonaGym: Evaluating Persona Agents and LLMs (2025.findings-emnlp)
Copied to clipboard
Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik R Narasimhan, Vishvak Murahari
| Challenge: | Persona agents are LLM agents conditioned to act according to an assigned persona . evaluating how faithfully these agents adhere to their personas remains a challenge . |
| Approach: | a new study evaluates persona agents' ability to act according to an assigned persona . a persona agent's person score is a human-aligned automatic metric that can be used to evaluate a model . |
| Outcome: | a new evaluation framework and a human-aligned automatic metric show that persona agents can perform better. |
Improving Generative Visual Dialog by Answering Diverse Questions (D19-1)
Copied to clipboard
| Challenge: | Prior work on training generative Visual Dialog models with reinforcement learning has shown that this improvement saturates and starts degrading after a few rounds of interaction, and does not lead to a better Visual Dialog model. |
| Approach: | They propose a Q-Bot-A-Bot image-guessing game that allows Q-BOT to ask diverse questions, thus reducing repetitions and enabling A-BOTT to explore a larger state space during RL. |
| Outcome: | The proposed approach improves Q-Bot-A-Bot image-guessing performance but degrades after a few rounds of interaction and does not lead to a better Visual Dialog model. |
C-STS: Conditional Semantic Textual Similarity (2023.emnlp-main)
Copied to clipboard
Ameet Deshpande, Carlos Jimenez, Howard Chen, Vishvak Murahari, Victoria Graf, Tanmay Rajpurohit, Ashwin Kalyan, Danqi Chen, Karthik Narasimhan
| Challenge: | Semantic textual similarity (STS) is a cornerstone task in natural language processing, but it is inherently ambiguous. |
| Approach: | They propose a task called conditional STS which measures similarity conditioned on an aspect elucidated in natural language. |
| Outcome: | The proposed task reduces subjectivity and ambiguity and enables fine-grained similarity evaluation using diverse conditions. |