Papers by Karthik Narasimhan
QualEval: Qualitative Evaluation for Model Improvement (2024.naacl-long)
Copied to clipboard
Vishvak Murahari, Ameet Deshpande, Peter Clark, Tanmay Rajpurohit, Ashish Sabharwal, Karthik Narasimhan, Ashwin Kalyan
| Challenge: | Quantitative evaluation metrics are inadequate for large language models due to complexity of tasks and cannot provide actionable diagnostics. |
| Approach: | They propose a quantitative evaluation tool called QualEval that uses automated qualitative evaluation as a vehicle for model improvement. |
| Outcome: | The proposed method improves the performance of the Llama 2 model by 15% compared to baselines. |
Self-Attention Networks Can Process Bounded Hierarchical Languages (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models that can process formal languages with hierarchical structure are limited in their performance. |
| Approach: | They propose to use a subset of Dyck-k with depth bounded by D to train self-attention networks. |
| Outcome: | The proposed model can process Dyck-(k, D) with depth bounded by D, which better captures the hierarchical structure of natural language. |
When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work on multilingual language models has demonstrated their capacity for cross-lingual zero-shot transfer on downstream tasks. |
| Approach: | They conduct a large-scale empirical study to isolate the effects of various linguistic properties by measuring zero-shot transfer between four different natural languages. |
| Outcome: | The proposed model exhibits decent cross-lingual zero-shot transfer, with no significant differences in word order and embedding alignment. |
MUX-PLMs: Data Multiplexing for High-throughput Language Models (2023.findings-emnlp)
Copied to clipboard
Vishvak Murahari, Ameet Deshpande, Carlos Jimenez, Izhak Shafran, Mingqiu Wang, Yuan Cao, Karthik Narasimhan
| Challenge: | MUX-PLMs are high-throughput pre-trained language models that can be fine-tuned for any downstream task to yield high-performance. |
| Approach: | They propose to train language models with data multiplexing to achieve 2x/5x inference speedup . they use multiplexers to entangle and disentangle inputs to achieve the same performance . |
| Outcome: | MUX-PLMs achieve 2x/5x inference speedup with 1-4 % drop on broad suite of tasks. |
Toxicity in chatgpt: Analyzing persona-assigned language models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown incredible capabilities and transcended the natural language processing community. |
| Approach: | They evaluate toxicity in over half a million generations of ChatGPT by assigning it a persona . they find that outputs engage in incorrect stereotypes, harmful dialogue, hurtful opinions . |
| Outcome: | a new study shows that assigning a persona to a chatbot can increase toxicity in half a million generations. |
PruMUX: Augmenting Data Multiplexing with Model Compression (2023.findings-acl)
Copied to clipboard
| Challenge: | Prior work has investigated methods like model pruning, knowledge distillation, and data multiplexing to increase model throughput without sacrificing accuracy. |
| Approach: | They propose to combine structured pruning and data multiplexing methods to increase model throughput without sacrificing accuracy. |
| Outcome: | The proposed method achieves 7.5-29.5X throughput improvement over a BERT-base model with accuracy threshold from 80% to 74%. |
Guiding Attention for Self-Supervised Learning with Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that self-attention patterns in trained models contain a majority of non-linguistic regularities. |
| Approach: | They propose a technique to allow efficient self-supervised learning with bi-directional Transformers by using an auxiliary loss function to guide attention heads to conform to such patterns. |
| Outcome: | The proposed method achieves state-of-the-art in low-resource settings and is agnostic to pre-training objectives. |
Referral Augmentation for Zero-Shot Information Retrieval (2024.findings-acl)
Copied to clipboard
| Challenge: | Referral-augmented retrieval improves zero-shot document retrieval in a variety of tasks . prior work shows sparse models struggle to reconcile with dense models . |
| Approach: | They propose a technique that concatenates document indices with referrals from other documents that cite or link to the given document. |
| Outcome: | The proposed technique outperforms generative text expansion techniques on structured tasks and improves on ACL paper retrieval. |
LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks (2025.coling-industry)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is a popular technique for parameter-efficient fine-tuning of Large Language Models. |
| Approach: | They propose to combine LoRA modules to achieve skill composition . they propose to use concatenation of LoRAs to optimize weights for different LoRA training . |
| Outcome: | The proposed model outperforms existing models and data- merging techniques on math-word problems and domain-specialized corpora. |
InstructEval: Systematic Evaluation of Instruction Selection Methods (2024.findings-naacl)
Copied to clipboard
| Challenge: | In-context learning (ICL) performs tasks by prompting a large language model using an instruction and a small set of annotated examples. |
| Approach: | They develop an ICL evaluation suite to evaluate the performance of popular instruction selection methods. |
| Outcome: | The proposed evaluation suite compares instruction selection methods over five metrics relevant to ICL. |
CARETS: A Consistency And Robustness Evaluative Test Suite for VQA (2022.acl-long)
Copied to clipboard
| Challenge: | Existing VQA tests lack superficial correlations and other weaknesses, which lead to optimistic evaluations when considering accuracy alone. |
| Approach: | They introduce a system to measure consistency and robustness of modern VQA models through a series of six fine-grained capability tests. |
| Outcome: | The proposed test suite evaluates six modern VQA systems and identifies several actionable weaknesses in model comprehension. |
Keep CALM and Explore: Language Models for Action Generation in Text-based Games (2020.emnlp-main)
Copied to clipboard
| Challenge: | Text-based games present a unique challenge for autonomous agents to operate in natural language and handle enormous action spaces. |
| Approach: | They propose a Contextual Action Language Model (CALM) to generate a compact set of action candidates at each game state. |
| Outcome: | The proposed model achieves a 69% improvement in average game score on unsupervised games . the proposed model is competitive with or better than other models that have access to ground truth admissible actions on half of the games tested . |
Can Rationalization Improve Robustness? (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing models that generate rationales before making predictions can ignore noise or adversarially added text by simply masking it out of the generated rationale. |
| Approach: | They propose to use a 'rationalizethen-predict' framework to generate subsets of input to generate rationales and then make predictions using them. |
| Outcome: | The proposed models improve robustness over AddText attacks while struggling in certain scenarios. |
Reading and Acting while Blindfolded: The Need for Semantics in Text Game Agents (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent work has used text-based games as a testbed for developing autonomous agents that operate using natural language. |
| Approach: | They propose an inverse dynamics decoder to regularize representation space and encourage exploration to reduce the amount of semantic information available to a learning agent. |
| Outcome: | The proposed model achieves high scores even in the absence of language semantics on Zork I . |
Robust and Interpretable Grounding of Spatial References with Relation Networks (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for understanding spatial references in text are vulnerable to noise in input text or state observations. |
| Approach: | They propose a text-conditioned relation network with a cross-modal attention module to capture fine-grained spatial relations between entities and a model that is robust and interpretable. |
| Outcome: | The proposed model improves performance on three tasks with a 17% improvement in predicting goal locations and a 15% improvement in robustness compared to state-of-the-art systems. |
C-STS: Conditional Semantic Textual Similarity (2023.emnlp-main)
Copied to clipboard
Ameet Deshpande, Carlos Jimenez, Howard Chen, Vishvak Murahari, Victoria Graf, Tanmay Rajpurohit, Ashwin Kalyan, Danqi Chen, Karthik Narasimhan
| Challenge: | Semantic textual similarity (STS) is a cornerstone task in natural language processing, but it is inherently ambiguous. |
| Approach: | They propose a task called conditional STS which measures similarity conditioned on an aspect elucidated in natural language. |
| Outcome: | The proposed task reduces subjectivity and ambiguity and enables fine-grained similarity evaluation using diverse conditions. |
Improving Dialog Systems for Negotiation with Personality Modeling (2021.acl-long)
Copied to clipboard
| Challenge: | In this paper, we introduce a framework for generating strategic dialog inspired by the idea of incorporating a theory of mind (ToM) into machines. |
| Approach: | They propose a probabilistic formulation to encapsulate the opponent's personality type during both learning and inference. |
| Outcome: | The proposed model achieves 20% higher dialog agreement rate compared to baselines on a mixed population of opponents. |
Universal Adversarial Attacks with Natural Triggers for Text Classification (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent work has demonstrated the vulnerability of modern text classifiers to universal adversarial attacks, which are input-agnostic sequences of words added to text processed by classifier. |
| Approach: | They propose a gradient-based search that aims to maximize the downstream classifierâs prediction loss by using an adversarially regularized autoencoder to generate triggers and propose heuristics to spot such attacks. |
| Outcome: | The proposed algorithms reduce model accuracy while being less identifiable than prior models as per automatic detection metrics and human-subject studies. |