Papers by Vivek Srikumar
Comprehensive Supersense Disambiguation of English Prepositions and Possessives (P18-1)
Copied to clipboard
Nathan Schneider, Jena D. Hwang, Vivek Srikumar, Jakob Prange, Austin Blodgett, Sarah R. Moeller, Aviram Stern, Adi Bitan, Omri Abend
| Challenge: | Frequent prepositions like for are maddeningly polysemous, their interpretation depends especially on the object of the preposition. |
| Approach: | They propose a new annotation scheme, corpus, and task for the disambiguation of prepositions and possessives in English. |
| Outcome: | The proposed annotations are comprehensive with respect to types and tokens of these markers and use broadly applicable supersense classes rather than fine-grained dictionary definitions. |
LLM-Symbolic Integration for Robust Temporal Tabular Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for temporal tabular question answering are inconsistent and fail to provide the variability needed to thoroughly evaluate models. |
| Approach: | TEMPTABQA-C uses a synthetic dataset and symbolic representation to generate and execute SQL queries. |
| Outcome: | TEMPTABQA-C improves on previous methods for temporal tabular question answering . incorporating adaptive fewshot prompting with tailored examples improves performance . lack of robustness, scalability, and interpretable solutions is key obstacle . |
Promptly Predicting Structures: The Return of Inference (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for structured prediction rely on large labeled datasets. Existing approaches for structured predictions require detailed annotation guidelines about the task, the label set, and the interactions between labels. |
| Approach: | They propose a framework for constructing zero- and few-shot linguistic structure predictors using structural constraints and combinatorial inferences. |
| Outcome: | The proposed framework can be extended to build zero- and few-shot label predictors on two structured prediction tasks and five datasets. |
Augmenting Neural Networks with First-order Logic (P19-1)
Copied to clipboard
| Challenge: | Existing paradigms for training neural networks require large datasets, a paper argues . we present a framework for introducing declarative knowledge to neural networks . |
| Approach: | They propose a framework for introducing declarative knowledge to neural networks . they compile logical statements into graphs that augment a network without extra learnable parameters or manual redesign. |
| Outcome: | The proposed framework improves on three tasks, especially in low-data regimes. |
DirectProbe: Studying Representations without Classifiers (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches for probing opaque representations often use training classifiers and use the accuracy, mutual information, or complexity as a proxy for the representation’s goodness. |
| Approach: | They propose a heuristic that directly studies the geometry of a representation by building upon the notion of 'version space' they argue that doing so can be unreliable because different representations may need different classifiers . |
| Outcome: | Experiments with linguistic tasks and contextualized embeddings show that even without training classifiers, DirectProbe can shine lights on how an embeddable space represents labels and anticipate the classifier performance for the representation. |
Structured Tuning for Semantic Role Labeling (2020.acl-main)
Copied to clipboard
| Challenge: | Recent neural network-driven semantic role labeling systems have shown impressive improvements in F1 scores. |
| Approach: | They propose a framework to tune models using softened constraints only at training time. |
| Outcome: | The proposed framework outperforms the baseline model with minimal training time and consistent improvements under low-resource scenarios. |
Elaboration-Generating Commonsense Question Answering at Scale (2023.acl-long)
Copied to clipboard
| Challenge: | elaborations are generated using language models that generate background knowledge that helps improve performance . human evaluations show that the quality of the generated ellaborations is high . |
| Approach: | They propose to finetune smaller language models to generate useful intermediate context . they compare a language model with an answer predictor and generate elaborations . human evaluations show that the quality of the generated ellaborations is high . |
| Outcome: | The proposed framework outperforms other models on commonsense questions on four commons sense benchmarks. |
Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that tabular reasoning models use spurious correlations and focus on false evidence or ignore it altogether. |
| Approach: | They propose a task where models need to extract evidence and then inference labels . they crowdsource evidence row labels and develop unsupervised evidence extraction strategies . |
| Outcome: | The proposed approach outperforms baseline models on the inference task using only the automatically extracted evidence as the premise. |
In-Context Example Ordering Guided by Label Distributions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Prior work has shown that ICL is sensitive to different natural language instructions and different orderings of in-context examples. |
| Approach: | They propose two principles for in-context example ordering guided by model’s probability predictions. |
| Outcome: | The proposed model outperforms baseline models on 13 text classification datasets and nine autoregressive LLMs with 700M to 13B parameters. |
A Logic-Driven Framework for Consistency of Neural Models (D19-1)
Copied to clipboard
| Challenge: | Recent advances in NLP have improved performance on benchmarks such as GLUE . however, tracking performance on a leaderboard is not sufficient to characterize model quality . |
| Approach: | They propose a framework for constraining neural models using logic rules to regularize them away from inconsistency. |
| Outcome: | The proposed framework can be used on natural language inference and is compatible with off-the-shelf learning schemes without model redesign. |
Visual Interrogation of Attention-Based Models for Natural Language Inference and Machine Comprehension (D18-2)
Copied to clipboard
| Challenge: | Neural networks models have gained popularity due to their state-of-the-art performance but lack of interpretability hinders their deployment and refinement. |
| Approach: | They propose a visual analytic library that provides a user with a customizable visual anallytic environment. |
| Outcome: | The proposed visualization library provides an interactive environment in which the user can investigate and interrogate the relationships between input, model internals and output predictions. |
Whispers of Doubt Amidst Echoes of Triumph in NLP Robustness (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to measure robustness are problematic, and out-of-domain evaluations are no longer relevant. |
| Approach: | They examine models of different sizes spanning different architectural choices and pretraining objectives. |
| Outcome: | The results show that not all out-of-domain tests provide insight into robustness . merely scaling models does not make them adequately robust . |
UNQOVERing Stereotyping Biases via Underspecified Questions (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Language embeddings have been shown to have stereotyping biases, but how these biase affecting downstream question answering models remains unexplored. |
| Approach: | They propose a general framework to probe biases through underspecified questions by building minimal context and building minimal questions. |
| Outcome: | The proposed framework isolates two types of reasoning errors and identifies stereotyping biases in gender, nationality, ethnicity, and religion classes. |
Verifying Annotation Agreement without Multiple Experts: A Case Study with Gujarati SNACS (2023.findings-acl)
Copied to clipboard
| Challenge: | a small fraction of the about 7,000 languages of the world have datasets or linguistic tools . linguistic datasets are a foundation of NLP research, but they are not always reliable . authors propose weak verifiers to help estimate dataset quality . |
| Approach: | They propose four weak verifiers to help estimate dataset quality . they propose to use Gujarati as a low-resource language to test for dataset quality. |
| Outcome: | The proposed methods concur with a double-annotation study in Gujarati. |
Learning Constraints for Structured Prediction Using Rectifier Networks (2020.acl-main)
Copied to clipboard
| Challenge: | Various natural language processing tasks require domain expertise to design good constraints. |
| Approach: | They propose a framework for learning constraints in a network of linear inequalities over the output variables. |
| Outcome: | The proposed framework can be used to learn constraints from data on natural language processing tasks. |
Logic-driven Indirect Supervision: An Application to Crisis Counseling (2023.acl-long)
Copied to clipboard
Mattia Medina Grespan, Meghan Broadbent, Xinyao Zhang, Katherine Axford, Brent Kious, Zac Imel, Vivek Srikumar
| Challenge: | Text-based crisis counseling services are increasingly adopted by people seeking confidential mental health support. |
| Approach: | They propose an inexpensive method that exploits declaratively stated structural dependencies between both levels of annotation to improve utterance modeling. |
| Outcome: | The proposed method improves utterance modeling by 3.5% over a strong multitask baseline. |
Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on compression prioritizes preserving perplexity, which is analogous to training loss. |
| Approach: | They examine the impact of model compression along four dimensions: degeneration harm, representational harm, dialect bias, and language modeling and downstream task performance. |
| Outcome: | The proposed compression methods can lead to unexpected consequences, the authors show . quantization preserves bias while pruning degrades quickly. |
INFOTABS: Inference on Tables as Semi-structured Data (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for text understanding lack human-parity across a wide array of reasoning skills. |
| Approach: | They propose an extension of the natural language inference task to include semi-structured tabulated text . they propose a semi-structural, multi-domain and heterogeneous nature of the premises that are tables extracted from Wikipedia info-boxes. |
| Outcome: | The proposed model outperforms baseline models on the GLUE benchmark suite. |
A Closer Look at How Fine-tuning Changes BERT (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained contextualized representations are used to analyze information in NLP . however, how fine-tuning changes the underlying embedding space is less studied . |
| Approach: | They propose to use probing techniques to analyze how fine-tuning changes the embedding space of pre-trained contextualized representations. |
| Outcome: | The proposed model improves classification performance by increasing the distances between examples associated with different labels. |
Incorporating External Knowledge to Enhance Tabular Reasoning (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing models struggle with tabular inference due to contextualized embeddings of text. |
| Approach: | They propose easy and effective modifications to how information is presented to a model for tabular inference. |
| Outcome: | The proposed modifications significantly improve tabular inference performance on large datasets. |
Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals (2024.findings-emnlp)
Copied to clipboard
Yanai Elazar, Bhargavi Paranjape, Hao Peng, Sarah Wiegreffe, Khyathi Chandu, Vivek Srikumar, Sameer Singh, Noah Smith
| Challenge: | Existing studies have found that datasets with paired inputs are prone to spurious correlations, resulting in models trained only on those outperform chance. |
| Approach: | They propose a counterfactual attentiveness test to measure reliance on spurious correlations by replacing part of the input with its counterpart from a different example. |
| Outcome: | The proposed method improves models' attentiveness on ten datasets spanning four tasks: natural language inference, reading comprehension, paraphrase detection, and visual & language reasoning. |
Learning Semantic Role Labeling from Compatible Label Sequences (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work has shown that cross-task interaction helps, but only explored multitask learning so far. |
| Approach: | They propose a framework that jointly models VerbNet and PropBank labels as one sequence and enforcing Semlink constraints during decoding improves the overall F1 . |
| Outcome: | The proposed model outperforms the prior best in-domain model by 3.5 (VerbNet) and 0.8 (PropBank). |
Putting Words in BERT’s Mouth: Navigating Contextualized Vector Spaces with Pseudowords (2021.emnlp-main)
Copied to clipboard
| Challenge: | a new technique for exploring contextualized vector space is proposed . masked prediction of a word in a sentence allows controlled exploration of the space . |
| Approach: | They propose a method for exploring regions around individual points in a contextualized vector space . they use a static embedding to induce a "pseudoword" vector and masked prediction of a word . |
| Outcome: | The proposed method investigates the geometry of the contextualized space around individual instances of a word . it uses a static embedding to induce a contextualized "pseudoword" vector . |
TempTabQA: Temporal Question Answering for Semi-Structured Tables (2023.emnlp-main)
Copied to clipboard
| Challenge: | Semi-structured data often include temporal information about entities, either implicitly or explicitly. |
| Approach: | They present a dataset that includes 11,454 question-answer pairs from Wikipedia Infobox tables spanning more than 90 distinct domains. |
| Outcome: | The proposed dataset can be used as a benchmark to improve models for temporal reasoning on semi-structured tables. |
Is My Model Using the Right Evidence? Systematic Probes for Examining Evidence-Based Tabular Reasoning (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing models that claim to reason about evidence should avoid spurious patterns . tabular inputs are well-suited for the study—they admit systematic probes . |
| Approach: | They propose to use tabular data to test whether models can reason about evidence . they show that a RoBERTa-based model fails to reason on the following counts . |
| Outcome: | The proposed model fails to reason on tabular data on the following counts . the model is over-sensitive to annotation artifacts and ignores relevant parts of the evidence . |
X-Fact: A New Benchmark Dataset for Multilingual Fact Checking (2021.acl-short)
Copied to clipboard
| Challenge: | Several fact-checking initiatives, such as PolitiFact, expend manual labor to investigate and determine the truthfulness of viral statements. |
| Approach: | They propose a multilingual dataset for factual verification of naturally existing claims . they use a benchmark to evaluate the multilingual models . |
| Outcome: | The proposed model achieves an F-score of around 40%, suggesting it is a challenging benchmark for multilingual fact-checking models. |
Don’t Retrain, Just Rewrite: Countering Adversarial Perturbations by Rewriting Text (2023.acl-long)
Copied to clipboard
| Challenge: | ATINTER model can be used to rewrite adversarial inputs to make them non-adversarial . if undefended, model should maintain good task performance and effectively mitigate adversarials . |
| Approach: | They propose a model that intercepts adversarial inputs and learns to rewrite them . they show that it provides better adversarial robustness than existing defense approaches . |
| Outcome: | The proposed model improves adversarial robustness without compromising task accuracy on a sentiment classification dataset. |
Observing Dialogue in Therapy: Categorizing and Forecasting Behavioral Codes (P19-1)
Copied to clipboard
| Challenge: | a new study examines the role of dialogue observers in psychotherapy . the model is based on motivational interviewing, which is effective for treating addictions . |
| Approach: | They propose to model MI behavioral codes for therapists by an observer . they propose to use the observer to forecast therapist and client MI behavioral code . |
| Outcome: | The proposed model outperforms baseline models for both tasks and reveals tradeoffs in performance. |
OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to mitigate stereotypical biases by linear projection are too aggressive . existing methods remove bias, but they also erase valuable information from word embeddings . |
| Approach: | They propose a bias-mitigating method that disentangles biased associations between concepts instead of removing concepts wholesale. |
| Outcome: | The proposed method disentangles biased associations between concepts rather than eliminating concepts wholesale. |
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations (2023.acl-long)
Copied to clipboard
Valentina Pyatkin, Jena D. Hwang, Vivek Srikumar, Ximing Lu, Liwei Jiang, Yejin Choi, Chandra Bhagavatula
| Challenge: | Changing contexts can flip the moral judgment of an action. |
| Approach: | They propose an interactive system that learns to ask clarification questions to elicit salient contexts of a social or moral situation. |
| Outcome: | The proposed system generates more relevant, informative and defeasible questions compared to baselines. |
Supertagging the Long Tail with Tree-Structured Decoding of Complex Categories (2021.tacl-1)
Copied to clipboard
| Challenge: | Combinatory Categorial Grammar (CCG) parsers operate as a pipeline with a large search space of complex 'supertags' . |
| Approach: | They propose to use CCG supertags to generate CCG categories that have never been seen in training and to use tree-structured prediction to account for their internal structure. |
| Outcome: | The proposed model recovers a fraction of the long-tail supertags while approximating the state of the art in overall tag accuracy with fewer parameters. |
CogCompNLP: Your Swiss Army Knife for NLP (L18-1)
Copied to clipboard
Daniel Khashabi, Mark Sammons, Ben Zhou, Tom Redman, Christos Christodoulopoulos, Vivek Srikumar, Nicholas Rizzolo, Lev Ratinov, Guanheng Luo, Quang Do, Chen-Tse Tsai, Subhro Roy, Stephen Mayhew, Zhili Feng, John Wieting, Xiaodong Yu, Yangqiu Song, Shashank Gupta, Shyam Upadhyay, Naveen Arivazhagan, Qiang Ning, Shaoshi Ling, Dan Roth
| Challenge: | a corpus-reader module supports popular corpora, feature extraction and annotation modules for semantic and syntactic tasks. |
| Approach: | They propose a library that provides modules to address different challenges . they provide a corpus-reader module that supports popular corpora in the NLP community . |
| Outcome: | The proposed library simplifies the process of design and development of NLP applications by providing modules to address different challenges. |