Papers by Shikhar Murty
Sneaking Syntax into Transformer Language Models with Tree Regularization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for incorporating syntactic inductive biases into transformers are limited . we introduce auxiliary loss function that converts bracketing decisions into differentiable orthogonality constraints on vector hidden states. |
| Approach: | They propose to introduce syntactic inductive biases into transformer circuits through a structured regularizer. |
| Outcome: | The proposed approach could unlock more robust and data-efficient learning in transformer language models . it integrates seamlessly with the standard LM objective, requiring no architectural changes. |
Probabilistic Embedding of Knowledge Graphs with Box Lattice Measures (P18-1)
Copied to clipboard
| Challenge: | Structured embeddings based on regions, densities, and orderings have gained popularity for their inductive bias towards the essential asymmetries inherent in problems such as image captioning. |
| Approach: | They propose a box lattice and accompanying probability measure to capture negative correlations over arbitrary concepts. |
| Outcome: | The proposed model can capture anti-correlation and even disjoint concepts while learning from and predicting calibrated uncertainty. |
Grokking of Hierarchical Structure in Vanilla Transformers (2023.acl-short)
Copied to clipboard
| Challenge: | a recent study has shown that neural sequence models like transformers can generalize hierarchically when training for extended periods. |
| Approach: | They show that transformers can learn to generalize hierarchically after long training periods . they call this phenomenon structural grokking, which exhibits inverted U-shaped scaling in model depth . |
| Outcome: | The proposed model generalizes better than both very deep and very shallow models on multiple datasets. |
On Measuring the Intrinsic Few-Shot Hardness of Datasets (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has shown that few-shot learning is successful for pre-trained models, but there is no concrete understanding of when and why it is successful. |
| Approach: | They propose a simple metric that estimates few-shot hardness for a given dataset . they propose metric which exploits feature-space invariances between training and test samples . |
| Outcome: | The proposed metric better accounts for few-shot hardness compared to existing notions and is 8-100x faster to compute. |
Pseudointelligence: A Unifying Lens on Language Model Evaluation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies claim that language models surpass human performance on new benchmarks within a few years. |
| Approach: | They propose a framework for model evaluation that casts as a dynamic interaction between a model and a learned evaluator. |
| Outcome: | The proposed framework can be used to reason about two case studies in language model evaluation, and analyze existing evaluation methods. |
DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference (2021.naacl-main)
Copied to clipboard
| Challenge: | Meta-learning has not yet succeeded in NLP due to the lack of a well-defined task distribution . meta-learners tend to overfit their adaptation mechanism and datasets are heterogeneous . |
| Approach: | They propose a method for decomposing datasets into Reasoning Categories to form additional high quality tasks. |
| Outcome: | The proposed method improves the accuracy of meta-learners by 1.5-4% across four few-shot NLI problems. |
Fixing Model Bugs with Natural Language Patches (2022.emnlp-main)
Copied to clipboard
| Challenge: | a growing body of research focused on using language to give instructions, supervision and even inductive biases to models instead of relying exclusively on labeled examples. |
| Approach: | They explore natural language patches that provide corrective feedback at the right level of abstraction. |
| Outcome: | The proposed model improves accuracy on real data by 1–4 accuracy points on different slices of a sentiment analysis dataset and F1 by 7 points on a relation extraction dataset. |
ExpBERT: Representation Engineering with Natural Language Explanations (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models with explanations use semantic parsers to interpret explanations, but they can only parse low-level statements like "'wife' appears between o 1 and 2 " |
| Approach: | They propose to use natural language explanations to specify inductive biases . they use BERT fine-tuned on MultiNLI to "interpret" the explanations with respect to the input sentence . |
| Outcome: | The proposed method matches a baseline with less labeled data and improves on the baseline with the same amount of labele. |
Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have not evaluated whether probing accuracy predicts syntactic outcomes. |
| Approach: | They evaluate 32 open-weight transformer models and find that probing fails to predict outcomes of targeted syntax evaluations across English linguistic phenomena. |
| Outcome: | The proposed model does not predict syntactic outcomes on English linguistic phenomena. |
Pushdown Layers: Encoding Recursive Structure in Transformer Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pushdown Layers model recursive state via stack tape that tracks estimated depths of tokens in incremental parsing . pushdown layers are drop-in replacement for standard self-attention . recursion is a key component of many aspects of intelligent behavior, authors say . |
| Approach: | They propose a self-attention layer that models recursive state via a stack tape . Pushdown Layers is a drop-in replacement for standard self- attention . |
| Outcome: | The proposed self-attention layer improves on parse tasks with a recursive-state model . it can model recursion using a stack tape that tracks estimated depths of tokens . |
Iterative Search for Weakly Supervised Semantic Parsing (N19-1)
Copied to clipboard
| Challenge: | Recent work has focused on training semantic parsers via weak supervision from denotations alone. |
| Approach: | They propose an iterative training algorithm that alternates between searching for consistent logical forms and maximizing the marginal likelihood of the retrieved ones. |
| Outcome: | The proposed algorithm outperforms the previous best systems on WikiTableQuestions and Cornell Natural Language Visual Reasoning (NLVR) iteratively train models that provide guidance to subsequent models to search for logical forms of increasing complexity, thus dealing with spuriousness. |
Hierarchical Losses and New Resources for Fine-grained Entity Typing and Linking (P18-1)
Copied to clipboard
| Challenge: | Existing methods to incorporate hierarchical information into knowledge bases have yielded little benefit. |
| Approach: | They propose methods to integrate hierarchical information using real bilinear mappings . they also propose two new datasets containing wide and deep hierarchies . |
| Outcome: | The proposed methods improve on flat predictions and fine-grained entity typing on FIGER dataset. |