Papers by Ramakanth Pasunuru
Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation (P18-1)
Copied to clipboard
| Challenge: | Recent advances on abstractive summarization have allowed substantial improvements in the quality of the model, but there is still scope for improvement. |
| Approach: | They propose novel multi-task architectures with high-level layer-specific sharing across multiple encoder and decoder layers of the three tasks and soft-sharing mechanisms. |
| Outcome: | The proposed model improves on the CNN/DailyMail and Gigaword datasets and on the DUC-2002 transfer setup. |
Efficient Tool Use with Chain-of-Abstraction Reasoning (2025.coling-main)
Copied to clipboard
Silin Gao, Jane Dwivedi-Yu, Ping Yu, Xiaoqing Ellen Tan, Ramakanth Pasunuru, Olga Golovneva, Koustuv Sinha, Asli Celikyilmaz, Antoine Bosselut, Tianlu Wang
| Challenge: | Recent large language models have made progress at interpreting and executing instructions. |
| Approach: | They propose a method to decouple general reasoning from specialized knowledge . they propose to use abstract reasoning chains and domain tools to reify each chain . |
| Outcome: | The proposed method outperforms baseline methods on QA and mathematical reasoning domains. |
Continual Few-Shot Learning for Text Classification (2021.emnlp-main)
Copied to clipboard
| Challenge: | a large number of end-to-end systems are needed for many tasks in natural language processing. |
| Approach: | They propose a continual few-shot learning task where a system is asked to correct mistakes with a few training examples. |
| Outcome: | The proposed task compares two NLI and one sentiment analysis datasets with baselines from diverse paradigms. |
AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning (N19-1)
Copied to clipboard
| Challenge: | Multi-task learning is an inductive transfer mechanism that leverages information from related tasks to improve the primary model's generalization performance. |
| Approach: | They propose a multitask learning pipeline that finds relevant auxiliary tasks and learns their mixing ratio. |
| Outcome: | The proposed model can find relevant auxiliary tasks and learn their mixing ratio . the proposed model achieves significant performance boosts on several primary tasks . |
Complementary Explanations for Effective In-Context Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have remarkable capabilities in learning from expla- nations in prompts, but there has been limited understanding of exactly how these explana- tions function or why they are effective. |
| Approach: | They propose a maximal marginal relevance-based exemplar selection approach to construct exemplar sets that are both relevant and comple- mentary. |
| Outcome: | The proposed model improves in- context learning performance across three tasks on multiple LLMs. |
Efficient Large Scale Language Modeling with Mixtures of Experts (2022.emnlp-main)
Copied to clipboard
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, Veselin Stoyanov
| Challenge: | Mixture of Experts layers (MoEs) enable efficient scaling of language models . large autoregressive language models such as GPT-3 can be adapted to a wide range of tasks . |
| Approach: | They propose to use Mixture of Experts layers to enable efficient scaling of language models . they find that MoEs are substantially more compute efficient than dense models compared to MoE models - but only when they are more modestly trained . |
| Outcome: | The proposed model outperforms dense models in a wide range of tasks and domains. |
Few-shot Learning with Multilingual Generative Language Models (2022.emnlp-main)
Copied to clipboard
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li
| Challenge: | Large-scale generative language models such as GPT-3 are competitive few-shot learners. |
| Approach: | They train multilingual generative language models on a corpus covering a diverse set of languages and study their few- and zero-shot learning capabilities. |
| Outcome: | The proposed model outperforms GPT-3 on 171 out of 182 directions with 32 training examples and surpasses the official supervised baseline in 45 directions. |
Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation (N19-1)
Copied to clipboard
Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, Ido Dagan
| Challenge: | Manual evaluation methods are perceived as insufficient due to the high cost of the Pyramid method and the required expertise. |
| Approach: | They propose a crowdsourced method that compares system summaries to references and uses crowdsourced scripts to analyze the results. |
| Outcome: | The proposed method shows higher correlation relative to the original Pyramid method. |
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022.naacl-main)
Copied to clipboard
Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, Zornitsa Kozareva
| Challenge: | Existing approaches to improve in-context few-shot learning are pretraining and downstream fewshot evaluation. |
| Approach: | They propose to use self-supervision as an intermediate training stage between pretraining and downstream fewshot usage to train models to perform in-context few shot learning. |
| Outcome: | The proposed model outperforms baseline models on two benchmarks. |
Extending Multi-Document Summarization Evaluation to the Interactive Setting (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to interactive summarization are incomparable and divergent . a key gap in the development and adoption of interactive summaries is the lack of evaluation methodologies and benchmarks for meaningful comparison of systems. |
| Approach: | They propose an end-to-end evaluation framework for interactive summarization based on expansion-based interaction . framework includes procedure of collecting real user sessions, evaluation measures relying on summarizing standards, but adapted to reflect interaction. |
| Outcome: | The proposed evaluation framework is based on evaluations of baseline implementations and is available publicly as a benchmark. |
Efficiently Summarizing Text and Graph Encodings of Multi-Document Clusters (2021.naacl-main)
Copied to clipboard
| Challenge: | Abstractive multi-document summarization (MDS) is a task that has seen advances with the introduction of large-scale datasets and powerful Transformer-based models. |
| Approach: | They propose an efficient graph-enhanced approach to multi-document summarization with an encoder-decoder Transformer model. |
| Outcome: | The proposed model scales to large input documents and improves on a multi-document dataset. |
Game-Based Video-Context Dialogue (D18-1)
Copied to clipboard
| Challenge: | Current dialogue systems focus more on textual and speech context knowledge and are usually based on two speakers. |
| Approach: | They propose to use live soccer game videos and Twitch.tv chats to develop visual-grounded dialogue models. |
| Outcome: | The proposed model can generate relevant temporal and spatial event language from live video and chat history while also being relevant to chat history. |
The ART of LLM Refinement: Ask, Refine, and Trust (2024.naacl-long)
Copied to clipboard
Kumar Shridhar, Koustuv Sinha, Andrew Cohen, Tianlu Wang, Ping Yu, Ramakanth Pasunuru, Mrinmaya Sachan, Jason Weston, Asli Celikyilmaz
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable generative abilities, but can they judge the quality of their own generations and self-improve? |
| Approach: | They propose a reasoning with a refinement strategy called *ART: Ask, Refine, and Trust* that asks necessary questions to decide when an LLM should refine its output and uses it to affirm or deny trust. |
| Outcome: | The proposed reasoning with a refinement strategy achieves a performance gain of +5 points over baselines on two multistep reasoning tasks. |
Multi-Reward Reinforced Summarization with Saliency and Entailment (N18-2)
Copied to clipboard
| Challenge: | Abstractive text summarization is the task of compressing and rewriting a long document into a short summary while maintaining saliency, directed logical entailment, and non-redundancy. |
| Approach: | They propose a novel reward function for ROUGESal and Entail to improve abstractive summarization . they use a coverage-based reward function to combine ROUGE and En Tail . |
| Outcome: | The proposed method achieves state-of-the-art results on CNN/Daily Mail dataset and strong improvements in a test-only transfer setup on DUC-2002. |
FENAS: Flexible and Expressive Neural Architecture Search (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent approaches to architecture search have shown good improvements in terms of performance with reasonable training speed. |
| Approach: | They propose an algorithm with more activation functions, input edges, and atomic operations to search for architectures that are optimal for given task. |
| Outcome: | The proposed algorithm reproduces well-known LSTM and GRU architectures and initializes with them for finding architectures more efficiently. |
ACUEval: Fine-grained Hallucination Evaluation and Correction for Abstractive Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent-proposed evaluation metrics for large language models have a preference-bias . however, such metrics often lack interpretability and only offer a single score . |
| Approach: | They propose a metric that leverages the power of large language models to perform two sub-tasks: decomposing summaries into atomic content units and validating them against the source document. |
| Outcome: | The proposed metric improves faithfulness scores on three summarization evaluation benchmarks by 3% compared to the next-best metric. |
Byte Latent Transformer: Patches Scale Better Than Tokens (2025.acl-long)
Copied to clipboard
Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, Srini Iyer
| Challenge: | Existing large language models (LLMs) are trained on bytes, except for tokenization, which groups bytes into a static set of tokens. |
| Approach: | They propose a new byte-level LLM architecture that encodes bytes into dynamically sized patches, which serve as the primary units of computation. |
| Outcome: | The proposed architecture matches tokenization-based models with improvements in inference efficiency and robustness. |
Proposition-Level Clustering for Multi-Document Summarization (2022.naacl-main)
Copied to clipboard
Ori Ernst, Avi Caciularu, Ori Shapira, Ramakanth Pasunuru, Mohit Bansal, Jacob Goldberger, Ido Dagan
| Challenge: | Existing methods focused on clustering sentences to indicate information saliency and avoid redundancy. |
| Approach: | They propose to group together sub-sentential propositions to generate a representative sentence for each cluster via text fusion. |
| Outcome: | The proposed method improves over the previous state-of-the-art method in the DUC 2004 and TAC 2011 datasets, both in automatic ROUGE scores and human preference. |
Continual and Multi-Task Architecture Search (P19-1)
Copied to clipboard
| Challenge: | Recent studies have shown that architecture search can improve performance on language modeling and image classification tasks with reasonable training speed. |
| Approach: | They propose a continual architecture search approach that continually evolves the model parameters during sequential training of several tasks without losing performance on previously learned tasks. |
| Outcome: | The proposed approach improves language modeling and image classification with reasonable training speed and a weight-sharing strategy. |
Crystal: Introspective Reasoners Reinforced with Self-Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge-augmented reasoning methods fail to capture the *introspective* nature of knowledge required in commonsense reasoning. |
| Approach: | They propose a method to develop an introspective commonsense reasoner that introspects for knowledge statements related to the given question and makes an informed prediction. |
| Outcome: | The proposed method outperforms standard supervised finetuning and chain-of-thought distilled methods and enhances the transparency of the commonsense reasoning process. |
Interactive Query-Assisted Summarization via Deep Reinforcement Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing systems that can perform interactive summarization cannot ingest the full document set or operate at sufficient speed for interactivity. |
| Approach: | They propose two deep reinforcement learning models for interactive summarization task . they use interactive session state and history to refrain from redundancy . |
| Outcome: | The proposed model improves informativeness while preserving positive user experience. |
MURMUR: Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | MURMUR generates highly faithful and correct reasoning paths that lead to 26% more logically consistent summaries on LogicNLG compared to direct prompting. |
| Approach: | They propose a neuro-symbolic modular approach to text generation from semi-structured data with multi-step reasoning that generates reasoning paths using neural and symbolic modules with specific linguistic and logical skills. |
| Outcome: | The proposed method improves on two data-to-text generation tasks, while achieving comparable performance to fine-tuned GPT-2 on out-of-domain data. |
DORB: Dynamically Optimizing Multiple Rewards with Bandits (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in end-to-end neural networks-based approaches have shown wide success in sequence generation tasks. |
| Approach: | They propose to optimize multiple metric rewards simultaneously using a multi-armed bandit approach . they empirically show the effectiveness of their approaches via various automatic metrics and human evaluation . |
| Outcome: | The proposed approach improves on question generation and data-to-text generation using a bandit approach. |
iFacetSum: Coreference-based Interactive Faceted Summarization for Multi-Document Exploration (2021.emnlp-demo)
Copied to clipboard
Eran Hirsch, Alon Eirew, Ori Shapira, Avi Caciularu, Arie Cattan, Ori Ernst, Ramakanth Pasunuru, Hadar Ronen, Mohit Bansal, Ido Dagan
| Challenge: | iFS provides a faceted navigation scheme that provides abstractive summaries for the user’s selections. |
| Approach: | They propose a web application that integrates interactive summarization and faceted search to provide a faceted navigation scheme that yields abstractive summaries for the user's selections. |
| Outcome: | The proposed system provides a comprehensive overview as well as particular details regard-ing subtopics of interest. |
Dynamic Multi-Level Multi-Task Learning for Sentence Simplification (C18-1)
Copied to clipboard
| Challenge: | Sentence simplification is the task of improving readability and understandability of an input text. |
| Approach: | They propose a strong pointer-copy mechanism based sequence-to-sequence sentence simplification model and a novel ‘multi-level’ soft sharing approach where each auxiliary task shares different (higher versus lower) level layers of the model. |
| Outcome: | The proposed model outperforms competing simplification systems in SARI and FKGL automatic metrics, and human evaluation. |
Training Trajectories of Language Models Across Scales (2023.acl-long)
Copied to clipboard
Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. |
| Approach: | They analyze the training checkpoints of different-sized OPT models on next-token prediction, sequence-level generation and downstream tasks. |
| Outcome: | The results show that language models of different sizes learn more during training . small models halt at hallucinations, larger ones learn to assign lower probabilities . |