Papers by Nazneen Rajani
Char2Subword: Extending the Subword Embedding Space Using Robust Character Compositionality (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Byte-pair encoding (BPE) is a ubiquitous algorithm in the tokenization process of language models but is only based on pre-training data statistics. |
| Approach: | They propose a character-based subword module that learns the subword embedding table in pre-trained language models like BERT. |
| Outcome: | The proposed method significantly improves the performance on the social media linguistic code-switching evaluation (LinCE) benchmark. |
Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements (2022.emnlp-demos)
Copied to clipboard
Leandro Von Werra, Lewis Tunstall, Abhishek Thakur, Sasha Luccioni, Tristan Thrush, Aleksandra Piktus, Felix Marty, Nazneen Rajani, Victor Mustar, Helen Ngo
| Challenge: | Evaluation is a key part of machine learning, yet there is neo-tooling to support it . auxiliary techniques such as testing for significance, measuring statistical power, and auxiliary methods are not available in ML. |
| Approach: | They propose a set of tools to facilitate the evaluation of models and datasets in machine learning . they propose 'evaluation on the Hub' platform that enables large-scale evaluation of over 75,000 models . |
| Outcome: | The proposed tools can be used to evaluate models and datasets on the Hugging Face Hub. |
What’s New? Summarizing Contributions in Scientific Literature (2023.eacl-main)
Copied to clipboard
| Challenge: | a growing number of academic articles are shared daily, making it difficult to keep up with the latest findings. |
| Approach: | They propose a task of disentangled paper summarization which generates separate summaries for papers and contexts to make it easier to identify key findings shared in articles. |
| Outcome: | The proposed task is more useful than traditional scientific paper summarization. |
SEAL: Interactive Tool for Systematic Error Analysis and Labeling (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Existing models that fail on tail data or rare groups are difficult to identify due to lack of explicit labels. |
| Approach: | They propose a systematic error analysis and labeling tool that uses a two-step approach to identify high-error slices of data and then give human-understandable semantics to those underperforming slices. |
| Outcome: | The proposed tool identifies high-error slices of data and gives human-understandable semantics to those underperforming slices. |
Knowledge-Enriched Natural Language Generation (2021.emnlp-tutorials)
Copied to clipboard
| Challenge: | Knowledge-enriched text generation poses unique challenges in modeling and learning . a roadmap will outline the state-of-the-art methods to tackle these challenges . |
| Approach: | They propose a roadmap to tackle the challenges of knowledge-enriched text generation . they will dive deep into various technical components to illustrate how to represent knowledge . |
| Outcome: | This tutorial outlines the state-of-the-art methods to tackle the problem . it aims to show how to represent knowledge, feed knowledge into a generation model, evaluate results . |
BOOKSUM: A Collection of Datasets for Long-form Narrative Summarization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text summarization datasets include short-form source documents that lack long-range causal and temporal dependencies and contain strong layout and stylistic biases. |
| Approach: | They propose a dataset for long-form narrative summarization that uses human written summaries on three levels of difficulty. |
| Outcome: | The proposed dataset covers documents from the literature domain, such as novels, plays and stories, and includes highly abstractive, human written summaries on three levels of difficulty. |
SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization (2021.acl-demo)
Copied to clipboard
| Challenge: | despite advances in abstractive text summarization, the true performance and failure modes of modern neural models are not yet fully understood due to the black-box nature of neural models and unmanageable scale of recent datasets for manual analysis. |
| Approach: | They propose an open-source tool for visualizing abstractive summaries that enables fine-grained analysis of models, data, and evaluation metrics associated with text summarization. |
| Outcome: | The proposed tool can identify the shortcomings and failure modes of state-of-the-art summarization models. |
Conformal Predictor for Improving Zero-Shot Text Classification Efficiency (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have been shown effective for zero-shot (0shot) text classification. |
| Approach: | They propose to limit the number of likely labels using a fast base classifier-based conformal predictor calibrated on samples labeled by the 0shot model. |
| Outcome: | The proposed models reduce the average inference time for NLI- and NSP-based models by 25.6% and 22.2% without dropping performance below the predefined error rate of 1%. |
FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging (2021.emnlp-main)
Copied to clipboard
| Challenge: | despite popularity of influence functions, their computational cost does not scale well with model and training data size. |
| Approach: | They propose a fast parallel variant that approximates the “influences” of training data-points for test predictions. |
| Outcome: | The proposed method achieves about 80X speedup while being highly correlated with the original influence values. |
HydraSum: Disentangling Style Features in Text Summarization with Multi-Decoder Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization systems implicitly encode “decisions” about summary properties, but these are not enforced. |
| Approach: | They propose a new summarization architecture that extends existing models to a mixture-of-experts version with multiple decoders. |
| Outcome: | The proposed architecture outperforms baseline models in obtaining stylistically-diverse summaries by sampling from individual decoders or their mixtures. |
Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on explainable NLP has shown that few-shot prompting can enable large pre-trained language models (LLMs) to generate grammatical and factual explanations for data labels. |
| Approach: | They propose to use few-shot prompting to generate grammatical and factual explanations for data labels by varying the hardness of the test samples and in-context samples to investigate the link between explainability and sample hardness. |
| Outcome: | The proposed model can generate grammatical and factual explanations for data labels with few-shot prompting. |
CaPE: Contrastive Parameter Ensembling for Reducing Hallucination in Abstractive Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing work suggests that the degree of hallucination depends on factual errors in training data. |
| Approach: | They propose a method to use training data to reduce hallucination by ensembling parameter variations in training data. |
| Outcome: | The proposed method improves on XSUM and CNN/DM datasets on human evaluations and factual metrics. |
Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Small shifts in user behavior can cause sharp drops in agent performance . prior work has shown that LLMs lack robustness to real-world noise and small input perturbations. |
| Approach: | They propose a model-agnostic method for systematically stress testing AI agents that learns directions in activation space corresponding to steerable user traits. |
| Outcome: | The proposed method can be used to stress test AI agents in airline, retail, telecom, and telehealth domains. |
CTRLsum: Towards Generic Controllable Text Summarization (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing summarization systems produce generic summaries that are disconnected from users’ preferences and expectations. |
| Approach: | They propose a generic framework to control generated summaries through a set of keywords. |
| Outcome: | The proposed framework is comparable or better than strong pretrained systems on three domains of summarization datasets and five control tasks. |
Stage-wise Fine-tuning for Graph-to-Text Generation (2021.acl-srw)
Copied to clipboard
| Challenge: | Graph-to-text generation has benefited from pre-trained language models (PLMs) but they fail to fully utilize the structure information of the input graph. |
| Approach: | They propose a structured graph-to-text model with a two-step fine-tuning mechanism which first fine-tracks model on Wikipedia before adapting to graph- to-text generation. |
| Outcome: | The proposed model improves the performance of the English WebNLG 2017 dataset by using tree-level embeddings to capture the inter-dependency structures of the input graph. |