Papers by Mostofa Patwary
Large Scale Multi-Actor Generative Dialog Modeling (2020.acl-main)
Copied to clipboard
| Challenge: | Non-goal oriented dialog agents typically exhibit inconsistent personality across conversations or the average personality of all users. |
| Approach: | They propose a model that conditionally models past conversations to probabilistically model multi-turn conversations in the actor’s persona. |
| Outcome: | The proposed model improves perplexity on 1.7M held out Reddit conversations by 0.47 on scaling from 117M to 8.3B parameters. |
MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models (2020.emnlp-main)
Copied to clipboard
Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, Bryan Catanzaro
| Challenge: | Existing pre-trained large language models have shown unparalleled generative capabilities, but they are not controllable. |
| Approach: | They propose a framework that uses large-scale language models and adds control to text generation by incorporating an external knowledge base. |
| Outcome: | The proposed model generates more fluent, consistent, and coherent stories with less repetition and higher diversity compared to previous work on the ROC story dataset. |
Context Generation Improves Open Domain Question Answering (2023.findings-eacl)
Copied to clipboard
Dan Su, Mostofa Patwary, Shrimai Prabhumoye, Peng Xu, Ryan Prenger, Mohammad Shoeybi, Pascale Fung, Anima Anandkumar, Bryan Catanzaro
| Challenge: | Existing closed-book question answering methods do not fully exploit the parameterized knowledge. |
| Approach: | They propose a closed-book QA framework which uses a coarse-to-fine approach to extract the relevant knowledge and answer a question. |
| Outcome: | The proposed method outperforms open-book QA methods on three QA benchmarks. |
LLM-Evolve: Evaluation for LLM’s Evolving Capability on Benchmarks (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models evaluate LLMs on i.i.d. tasks, overlooking their ability to learn iteratively from past experiences. |
| Approach: | They propose a framework which extends established benchmarks to sequential problem-solving settings and provides feedback after each round to build a demonstration memory that the models can query in future tasks. |
| Outcome: | The proposed framework can improve performance of LLMs by learning from past interactions and improve models' performance over time. |
Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning (2026.eacl-long)
Copied to clipboard
Syeda Nahida Akter, Shrimai Prabhumoye, Matvei Novikov, Seungju Han, Ying Lin, Evelina Bakhturina, Eric Nyberg, Yejin Choi, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
| Challenge: | Prior work has successfully applied Reinforcement Learning (RL) to mathematical reasoning, but generalization to broader domains remains challenging due to limited data and lack of verifiable rewards for unstructured domains. |
| Approach: | They propose a framework that integrates multi-domain corpora into RL training to improve generalization across diverse reasoning tasks. |
| Outcome: | The proposed framework improves generalization across diverse reasoning tasks. |
Training Question Answering Models From Synthetic Data (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on question and answer generation aims to improve question answering models given limited amount of labeled data. |
| Approach: | They synthesize questions and answers from a synthetic text corpus generated by an 8.3 billion parameter GPT-2 model and achieve 88.4 Exact Match (EM) and 93.9 F1 score on the SQuAD1.1 dev set. |
| Outcome: | The proposed model achieves higher accuracy than the SQUAD1.1 training set questions using synthetic questions and answers than the training set question. |
Data, Data Everywhere: A Guide for Pretraining Dataset Construction (2024.emnlp-main)
Copied to clipboard
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Bo Liu, Aastha Jhunjhunwala, Zhilin Wang, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
| Challenge: | Recent language models have impressive capabilities on a number of evaluation areas. |
| Approach: | They conduct systematic analysis of pretraining set construction to identify which methods yield the greatest gains in model accuracy. |
| Outcome: | The proposed method can be used to refine and improve a pretraining set. |
Adding Instructions during Pretraining: Effective way of Controlling Toxicity in Language Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Pretrained large language models generate harmful language encompassing hate speech, abusive language, social biases, and threats. |
| Approach: | They propose two strategies that augment pretraining data to reduce model toxicity . MEDA adds raw toxicity score as meta-data and INST adds instructions indicating toxicity to pretraining samples. |
| Outcome: | The proposed strategies reduce toxicity probability up to 61% while preserving accuracy on five benchmark NLP tasks and improving AUC scores on bias detection tasks by 1.3%. |
BioMegatron: Larger Biomedical Domain Language Model (2020.emnlp-main)
Copied to clipboard
Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, Raghav Mani
| Challenge: | Existing studies on domain language models do not study the factors affecting performance on domain languages. |
| Approach: | They empirically evaluate factors that can affect performance on domain language applications . sub-word vocabulary set, model size, pre-training corpus, and domain transfer are important . |
| Outcome: | The results show language models trained on biomedical text perform better on biomedicine benchmarks than those trained on general domain text corpora. |
Evaluating Parameter Efficient Learning for Generation (2022.emnlp-main)
Copied to clipboard
Peng Xu, Mostofa Patwary, Shrimai Prabhumoye, Virginia Adams, Ryan Prenger, Wei Ping, Nayeon Lee, Mohammad Shoeybi, Bryan Catanzaro
| Challenge: | Parameter efficient learning methods (PERMs) are gaining attention for their ability to adapt to a downstream task. |
| Approach: | They propose to use parameter efficient learning methods to improve model adaptation . they compare in-domain evaluations and generalizations to unseen domains and new datasets . |
| Outcome: | The proposed method outperforms finetuning and PERMs in in-domain evaluations. |
Multi-Stage Prompting for Knowledgeable Dialogue Generation (2022.findings-acl)
Copied to clipboard
Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, Bryan Catanzaro
| Challenge: | Existing knowledge-grounded dialogue systems typically use finetuned versions of a pretrained language model and large-scale knowledge bases. |
| Approach: | They propose a multi-stage prompting approach to generate knowledgeable responses from a single pretrained LM. |
| Outcome: | The proposed model outperforms the state-of-the-art retrieval-based model in terms of knowledge relevance and correctness by 5.8% and 5%, respectively. |
Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset (2025.acl-long)
Copied to clipboard
Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
| Challenge: | Recent Common Crawl datasets remove 90% of data, limiting their suitability for long token horizon training. |
| Approach: | They propose to combine classifier ensembling, synthetic data rephrasing and heuristic filters to achieve better trade-offs between accuracy and data quantity. |
| Outcome: | The proposed model-based filtering improves MMLU by 5.6 over DCLM for 15T tokens . the full 6.3T token dataset matches DCLM on MMLO, but contains four times more unique real tokens than DCLM . |
End-to-End Training of Neural Retrievers for Open-Domain Question Answering (2021.acl-long)
Copied to clipboard
Devendra Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton, Bryan Catanzaro
| Challenge: | Recent work on training neural retrievers for open-domain question answering (OpenQA) has employed both supervised and unsupervised methods. |
| Approach: | They propose an approach of unsupervised pre-training with the Inverse Cloze Task and masked salient spans followed by supervised finetuning using question-context pairs. |
| Outcome: | The proposed approach outperforms models like REALM and RAG in retrieval accuracy and answer extraction. |