Papers by Anna Rumshisky
MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs (2025.naacl-long)
Copied to clipboard
Yuhang Zhou, Giannis Karamanolakis, Victor Soto, Anna Rumshisky, Mayank Kulkarni, Furong Huang, Wei Ai, Jianhua Lu
| Challenge: | State-of-the-art methods for merging expert models with different architectures do not address parameter interference and require extensive fine-tuning to restore performance. |
| Approach: | They propose a method for merging experts with different architectures into a unified Mixture-of-Experts model with a goal of enhancing performance in each domain while retaining effectiveness on general tasks. |
| Outcome: | Experiments across multiple domains show that the proposed methods reduce fine-tuning costs and improve performance over state-of-the-art methods. |
RuSentiment: An Enriched Sentiment Analysis Dataset for Social Media in Russian (C18-1)
Copied to clipboard
| Challenge: | RuSentiment is currently the largest in its class for Russian, with 31,185 posts annotated with Fleiss’ kappa of 0.58 (3 annotations per post). |
| Approach: | They propose to use RuSentiment to annotate social media posts in Russian with a kappa of 0.58 and a set of annotation guidelines that are extensible to other languages. |
| Outcome: | The proposed dataset is the largest in its class for Russian, with 31,185 posts annotated with Fleiss’ kappa of 0.58 (3 annotations per post). |
Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained encoder-only and sequence-to-sequence models are computationally expensive. |
| Approach: | They propose a recipe to initialize one model from the other to improve pre-training efficiency. |
| Outcome: | The proposed method matches the performance of a from-scratch model with a multilingual encoder while reducing the total compute cost by 27%. |
What’s in Your Embedding, And How It Predicts Task Performance (C18-1)
Copied to clipboard
| Challenge: | Attempts to find a single technique for general-purpose intrinsic evaluation of word embeddings have so far not been successful. |
| Approach: | They propose a method that quantifies interpretable characteristics of word vector neighborhoods and shows how they correlate with performance on 14 extrinsic and intrinsic task datasets. |
| Outcome: | The proposed approach enables multi-faceted evaluation, parameter search, and generally – a more principled, hypothesis-driven approach to development of distributional semantic representations. |
On Localizing and Deleting Toxic Memories in Large Language Models (2025.findings-naacl)
Copied to clipboard
Anubrata Das, Manoj Kumar, Ninareh Mehrabi, Anil Ramakrishna, Anna Rumshisky, Kai-Wei Chang, Aram Galstyan, Morteza Ziyadi, Rahul Gupta
| Challenge: | Existing methods to reduce toxic generation in large language models are not fully understood. |
| Approach: | They propose to understand the mechanisms that drive toxic generation in large language models by using memory localization to reduce toxic generation. |
| Outcome: | The proposed method reduces toxic generation from 62.86% to 28.61%, but it also improves generation quality. |
Emergent Abilities in Reduced-Scale Generative Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models can solve new tasks without task-specific fine-tuning. |
| Approach: | They propose to use pre-training data to pre-train 36 language models with billions of parameters to investigate whether emergent properties are tied to model size or can be demonstrated by smaller models. |
| Outcome: | The proposed model performs comparable to models trained on unrestricted language. |
Down and Across: Introducing Crossword-Solving as a New NLP Benchmark (2022.acl-long)
Copied to clipboard
| Challenge: | Recent advances in NLP have led to a growing demand for challenging tasks. |
| Approach: | They propose to solve crossword puzzles as a natural language understanding task . they release a corpus of crossword clues from the daily crossword spanning 25 years . |
| Outcome: | The proposed task is based on a corpus of crossword puzzles from the new york times daily crossword spanning 25 years . the dataset contains over half a million unique clue-answer pairs . |
Life after BERT: What do Other Muppets Understand about Language? (2022.acl-long)
Copied to clipboard
| Challenge: | Existing pre-trained transformer analysis studies focus on one or two model families at a time, overlooking the variability of the architecture and pre-training objectives. |
| Approach: | They utilize oLMpics bench- mark and psycholinguistic probing datasets for a diverse set of 29 models including T5, BART, and ALBERT. |
| Outcome: | The proposed model fails to resolve compositional questions in a zero-shot fashion, suggesting that pre-training objectives are not predictive of a model’s linguistic capabilities. |
Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale. (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on high-compute settings, leaving the question of when these abilities begin to emerge largely unanswered. |
| Approach: | They investigate whether effects of pre-training can be observed when problem size is reduced, modeling a smaller, reduced-vocabulary language. |
| Outcome: | The proposed model performance is correlated with pre-training perplexity and performance. |
Revealing the Dark Secrets of BERT (D19-1)
Copied to clipboard
| Challenge: | Existing models of BERT-based learning systems are lacking specific mechanisms that contribute to its success. |
| Approach: | They propose to use GLUE tasks to analyze the interpretation of self-attention, which is one of the underlying components of BERT. |
| Outcome: | The proposed model outperforms the regular model on GLUE tasks by disabling attention in certain heads. |
Adversarial Decomposition of Text Representation (N19-1)
Copied to clipboard
| Challenge: | a new method for adversarial decomposition of text representations is proposed . it is capable of fine-grained controlled change of different aspects of the input sentence . |
| Approach: | They propose a method for adversarial decomposition of text representation . they use vectors responsible for a specific aspect of the input sentence . |
| Outcome: | The proposed method outperforms the embeddings of a regular autoencoder on paraphrase detection tasks. |
Prompt Perturbation Consistency Learning for Robust Language Models (2024.findings-eacl)
Copied to clipboard
Yao Qiang, Subhrangshu Nandi, Ninareh Mehrabi, Greg Ver Steeg, Anoop Kumar, Anna Rumshisky, Aram Galstyan
| Challenge: | Large language models have demonstrated impressive performance on a number of natural language processing tasks, such as question answering and text summarization. |
| Approach: | They propose a method to reduce the performance drop of large language models by regularizing the divergence between losses from clean and perturbed samples. |
| Outcome: | The proposed approach recovers on average 59% and 69% of the performance drop for IC and SF tasks while using ten times fewer augmented data samples. |
A guide to the dataset explosion in QA, NLI, and commonsense reasoning (2020.coling-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to provide an up-to-date guide to the recent datasets . the target audience is the NLP practitioners who are lost in dozens of the recent data sets. |
| Approach: | This tutorial provides an up-to-date guide to the recent datasets . it surveys old and new methodological issues with dataset construction . |
| Outcome: | This tutorial aims to provide an up-to-date guide to the recent datasets . it surveys the old and new methodological issues with dataset construction . |
Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Diverse language model responses are crucial for creative generation, open-ended tasks, and self-improvement training. |
| Approach: | They propose a length-controlled data selection strategy that improves diversity while maintaining length parity. |
| Outcome: | The proposed method improves diversity while maintaining length parity on LLaMA-3.1-8B and Olmo-2 family. |
Automatic Labeling of Problem-Solving Dialogues for Computational Microgenetic Learning Analytics (L18-1)
Copied to clipboard
| Challenge: | This paper presents a recurrent neural network model to automate the analysis of students' computational thinking in problem-solving dialogue. |
| Approach: | They propose a recurrent neural network model to automate the analysis of students' computational thinking in problem-solving dialogue. |
| Outcome: | The proposed model outperforms the baseline model and outperformed the nave model by a large margin. |
Triad-based Neural Network for Coreference Resolution (C18-1)
Copied to clipboard
| Challenge: | Entity coreference resolution aims to identify mentions that refer to the same entity. |
| Approach: | They propose a triad-based neural network system that generates affinity scores between entity mentions for coreference resolution. |
| Outcome: | The proposed system generates affinity scores between mentions for coreference resolution. |
Similarity-Based Reconstruction Loss for Meaning Representation (D18-1)
Copied to clipboard
| Challenge: | a new paper proposes and evaluates a set of loss functions that can be used to train models for representation learning . cross-entropy loss penalizes models when they fail to generate the exact word from ground truth data . |
| Approach: | They propose and evaluate loss functions that can be used to train any neural model for representation learning. |
| Outcome: | The proposed loss functions amplify semantic diversity while preserving original meaning . they show performance improvement on paraphrase detection and language inference tasks . |
Calls to Action on Social Media: Detection, Social Impact, and Censorship Potential (D19-50)
Copied to clipboard
| Challenge: | Calls to action are effective means of mobilization in social networks, but their potential for censorship and predicting offline protest events has not yet been evaluated. |
| Approach: | They examine the possibility of their automatic detection on historical data from the 2011-2013 protests in Bolotnaya, Russia. |
| Outcome: | The political calls to action can be annotated and detected with relatively high accuracy and have a moderate positive correlation with actual rally attendance. |
Context-Aware Neural Model for Temporal Information Extraction (P18-1)
Copied to clipboard
| Challenge: | Existing temporal information extraction systems rely on statistical learning with feature-engineered task-specific models. |
| Approach: | They propose a context-aware neural network model for temporal information extraction using a global context layer. |
| Outcome: | The proposed model outperforms existing models in terms of performance and performance . it is the first model to use NTM-like architecture to process the information from global context in discourse-scale natural text processing. |
When BERT Plays the Lottery, All Tickets Are Winning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Large Transformer-based models are reduced to a smaller number of self-attention heads and layers. |
| Approach: | They propose to prune BERT self-attention heads and layers to find subnetworks with comparable performance . they also extend this technique to multi-layer perceptrons to find out if they are unstable . |
| Outcome: | The proposed models are able to achieve 90% of full model performance with structured pruning and similar-sized subnetworks sampled from the rest of the model perform worse. |
Federated Learning with Noisy User Feedback (2022.naacl-main)
Copied to clipboard
Rahul Sharma, Anil Ramakrishna, Ansel MacLaughlin, Anna Rumshisky, Jimit Majmudar, Clement Chung, Salman Avestimehr, Rahul Gupta
| Challenge: | Artificial Intelligence (AI) and Machine Learning (ML) systems are becoming more popular and are causing concerns over user privacy. |
| Approach: | They propose a method for training ML models using positive and negative user feedback and a framework to extract labels on edge to make FL viable. |
| Outcome: | The proposed method improves significantly over a self-training baseline, achieving performance closer to models trained with full supervision. |
BERT Busters: Outlier Dimensions that Disrupt Transformers (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that pre-trained Transformers are remarkably robust to pruning. |
| Approach: | They show that pre-trained Transformer encoders are surprisingly fragile to pruning . they show that disabling them significantly degrades both the MLM loss and the downstream task performance. |
| Outcome: | The results show that the removal of features in pre-trained transformers significantly degrades both the MLM loss and the downstream task performance. |
Controlled Data Generation via Insertion Operations for NLU (2022.naacl-industry)
Copied to clipboard
| Challenge: | a new approach to annotate live traffic is emerging to be cost-effective and efficient . manual data annotation is expensive and not preferred for meeting customer privacy expectations . |
| Approach: | They propose a targeted synthetic data generation technique by inserting tokens into a given semantic signature. |
| Outcome: | The proposed approach achieves the same accuracy as training with all available data on a voice assistant dataset. |
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes (N19-1)
Copied to clipboard
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Anna Rumshisky, Adam Kalai
| Challenge: | Existing methods for mitigating bias in machine learning systems rely on access to protected attributes such as race, gender, or age. |
| Approach: | They propose a method for discouraging correlation between predicted probability of an individual’s true occupation and a word embedding of their name. |
| Outcome: | The proposed method reduces race and gender biases, with almost no reduction in the classifier’s overall true positive rate. |
NarrativeTime: Dense Temporal Annotation on a Timeline (2024.lrec-main)
Copied to clipboard
| Challenge: | e.g. TimeBank contains 1-5% of all possible tlinks, and this information is underspecified in the text. |
| Approach: | They propose a timeline-based framework that achieves full coverage of all possible TLINKs. |
| Outcome: | The proposed framework achieves full coverage of all possible TLINKs in a text. |
Larger Probes Tell a Different Story: Extending Psycholinguistic Datasets Via In-Context Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language model probing is often used to test specific capabilities of models, but results are limited when benchmarks are small and lack statistical power. |
| Approach: | They extend existing NEG-136 and ROLE-88 benchmarks to 750 sentence pairs and create an extended negation dataset using template-based generation. |
| Outcome: | The proposed datasets show that models perform better on the extended datasets than on the smaller benchmarks. |
A Primer in BERTology: What We Know About How BERT Works (2020.tacl-1)
Copied to clipboard
| Challenge: | a new study examines the current state of knowledge about the BERT model . the model is a stack of transformer encoder layers that are based on multiple self-attention ''heads'' |
| Approach: | They present a survey of over 150 studies of the popular Transformer-based model BERT . they discuss the current state of knowledge about how BERT works and how it is represented . |
| Outcome: | The proposed model is based on the Transformer-based model with state-of-the-art results . the proposed model has little cognitive motivation and is too small to perform ablation studies . |
Chasing the Tail with Domain Generalization: A Case Study on Frequency-Enriched Datasets (2022.aacl-main)
Copied to clipboard
| Challenge: | In academic research, natural language understanding tasks are typically defined by creating annotated datasets in which each utterance is encountered once. |
| Approach: | They propose a method that explicitly uses utterance frequency in training data to learn models that are more robust to unknown distributions. |
| Outcome: | The proposed approach shows up to 7.02% relative improvement over baselines on the tail data. |
Deconstructing In-Context Learning: Understanding Prompts via Corruption (2024.lrec-main)
Copied to clipboard
| Challenge: | Prior work examined how modifying different elements of the prompt can affect model performance, but this limited number of elements made replication challenging. |
| Approach: | They decompose the entire prompt into four components: task description, demonstration inputs, labels, and inline instructions provided for each demonstration. |
| Outcome: | The proposed model is robust to minor prompt modifications, but its underlying pre-trained backbone is brittle . previous studies focused on models with fewer than 15 billion parameters or exclusively examined black-box models like GPT-3 or PaLM, making replication challenging. |