Papers by Dipanjan Das
SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation (2023.emnlp-main)
Copied to clipboard
Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, Ankur Parikh
| Challenge: | evaluating the quality of generated text is a difficult problem for large language models. |
| Approach: | They propose a dataset for multilingual, multifaceted summarization evaluation. |
| Outcome: | The proposed dataset can be used to train multilingual summarization systems . it shows that the dataset performs well on the out-of-domain meta-evaluation benchmarks TRUE and mFACE . |
WikiAtomicEdits: A Multilingual Corpus of Wikipedia Edits for Modeling Language and Discourse (D18-1)
Copied to clipboard
| Challenge: | a corpus of 43 million atomic edits is available for Wikipedia edit history . edits are instances in which a human editor has inserted a single contiguous phrase into, or deleted a contigous phrase from, an existing sentence. |
| Approach: | They use Wikipedia edit history to mine atomic edits across 8 languages . they find edits contain instances in which a human editor has inserted a single phrase into, or deleted a contiguous phrase from, an existing sentence. |
| Outcome: | The data show that edits differ from the language observed in standard corpora and that models trained on edits encode different aspects of semantics and discourse than models trained in raw text. |
Query Refinement Prompts for Closed-Book Long-Form QA (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can answer questions and produce long-form texts, but the latter is difficult to evaluate since they are subjective in nature. |
| Approach: | They propose query refinement prompts that encourage LLMs to express multifacetedness and generate long-form answers covering multiple facets of the question. |
| Outcome: | The proposed model outperforms fully finetuned models in the closed-book setting and retrieve-then-generate open-book models. |
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (P19-1)
Copied to clipboard
| Challenge: | Existing text generation metrics rely on reference texts, such as BLEU and ROUGE, but they are too expensive to apply repeatedly. |
| Approach: | They propose a metric which aligns n-grams from the generated texts to the semi-structured data before computing their precision and recall. |
| Outcome: | The proposed metric correlates with human judgments better than existing text generation metrics while being easier to use. |
Text Generation with Exemplar-based Adaptive Decoding (N19-1)
Copied to clipboard
| Challenge: | Empirical results show that the proposed model achieves strong performance and outperforms comparable baselines. |
| Approach: | They propose a conditioned text generation model that uses a template-based approach to generate content from input text. |
| Outcome: | The proposed model outperforms baselines on abstractive text summarization and data-to-text generation. |
Dolomites: Domain-Specific Long-Form Methodical Tasks (2025.tacl-1)
Copied to clipboard
Chaitanya Malaviya, Priyanka Agrawal, Kuzman Ganchev, Pranesh Srinivasan, Fantine Huot, Jonathan Berant, Mark Yatskar, Dipanjan Das, Mirella Lapata, Chris Alberti
| Challenge: | Experts in various fields perform methodical writing tasks to plan, organize, and report their work. |
| Approach: | They propose a benchmark with specifications for 519 methodical writing tasks . they use expert revisions of up to 10 model-generated examples to evaluate contemporary language models. |
| Outcome: | The proposed benchmark includes specifications for 519 methodical writing tasks . it includes examples with input and output examples, and is available at https://dolomites-benchmark.github.io/ . |
Decontextualization: Making Sentences Stand-Alone (2021.tacl-1)
Copied to clipboard
| Challenge: | Taking excerpts of text can be problematic, as key pieces may not be explicit in a local window. |
| Approach: | They define a problem of sentence decontextualization by rewriting a sentence to be interpretable out of context while preserving its meaning. |
| Outcome: | The proposed method can be used in question answering and document understanding tasks. |
ToTTo: A Controlled Table-To-Text Generation Dataset (2020.emnlp-main)
Copied to clipboard
Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, Dipanjan Das
| Challenge: | Existing methods for data-to-text generation often hallucinate phrases not supported by the Wikipedia table. |
| Approach: | They propose a controlled task where annotators directly revise existing Wikipedia sentences to generate a one-sentence description. |
| Outcome: | The proposed task produces a one-sentence description from a Wikipedia table and highlighted cells. |
Syntactic Data Augmentation Increases Robustness to Inference Heuristics (2020.acl-main)
Copied to clipboard
| Challenge: | Pretrained neural models lack sensitivity to word order on controlled challenge sets . augmentation methods that improve accuracy on standard training sets may be a problem . |
| Approach: | They propose to augment standard training sets with syntactically informative examples by applying syntastic transformations to sentences from the MNLI corpus. |
| Outcome: | The proposed method improved BERT’s accuracy on controlled examples that diagnose sensitivity to word order from 0.28 to 0.73 without affecting performance on the MNLI test set. |
BERT Rediscovers the Classical NLP Pipeline (P19-1)
Copied to clipboard
| Challenge: | Pre-trained text encoders have advanced the state of the art on many NLP tasks . Qualitative analysis reveals that the model can and often does adjust this pipeline dynamically . |
| Approach: | They aim to quantify where linguistic information is captured within a network model . they aim to use pre-trained text encoders to displace static word embeddings . |
| Outcome: | The proposed model can adjust the pipeline dynamically, revealing lower-level decisions on the basis of disambiguation from higher-level representations. |
Conditional Generation with a Question-Answering Blueprint (2023.tacl-1)
Copied to clipboard
Shashi Narayan, Joshua Maynez, Reinald Kim Amplayo, Kuzman Ganchev, Annie Louis, Fantine Huot, Anders Sandholm, Dipanjan Das, Mirella Lapata
| Challenge: | Neural generation models often struggle to identify which content units are salient. |
| Approach: | They propose a new conceptualization of text plans as a sequence of question-answer pairs . they propose QA blueprints as QA proxy for content selection and planning . |
| Outcome: | The proposed model improves existing datasets with QA blueprints as proxy for content selection and planning. |
Learning To Split and Rephrase From Wikipedia Edit History (D18-1)
Copied to clipboard
| Challenge: | Performing split and rephrase tasks is one of the main operations in text simplification, alongside paraphrasing and dropping less salient content. |
| Approach: | They propose to use Wikipedia's edit history to extract a rich new dataset for the task. |
| Outcome: | The proposed model scores 32 BLEU points above the previous best on the WebSplit benchmark. |
Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features (2021.acl-long)
Copied to clipboard
| Challenge: | Existing systems that strive to be informative teachers are difficult to build . knowledge grounded dialogue systems are difficult because of limited training objectives . |
| Approach: | They propose to train a generative neural dialogue model that is controlled to stay faithful to evidence . they propose to use additional inputs to generate more objective responses . |
| Outcome: | The proposed model produces responses that are perceived by humans to be objective and faithful to evidence. |
BLEURT: Learning Robust Metrics for Text Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Text generation has made significant advances, but evaluation metrics have lagged behind. |
| Approach: | They propose a learning evaluation metric for English based on BERT . BLEURT can model human judgment with a few thousand possibly biased training examples . |
| Outcome: | The proposed model can model human judgment with a few thousand potentially biased training examples. |
A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generation (2022.acl-long)
Copied to clipboard
Shashi Narayan, Gonçalo Simões, Yao Zhao, Joshua Maynez, Dipanjan Das, Michael Collins, Mirella Lapata
| Challenge: | Composition Sampling is a simple but effective method to generate diverse outputs for conditional generation of higher quality compared to previous stochastic decoding strategies. |
| Approach: | They propose a method to generate diverse outputs for conditional generation . they use a plan-based neural generation model that is trained to create a composition of the output and then generate by conditioning on it and the input. |
| Outcome: | The proposed method avoids text degeneration by first sampling a composition in the form of an entity chain and then using beam search to generate the best possible text grounded to this entity chain. |
Identifying Well-formed Natural Language Questions (D18-1)
Copied to clipboard
| Challenge: | Understanding search queries is a hard problem as it involves dealing with “word salad” text. |
| Approach: | They propose to identify a well-formed natural language question from a dataset of 25,100 questions. |
| Outcome: | The proposed classifier improves the performance of neural sequence-to-sequence models for generating questions for reading comprehension. |
Text-Blueprint: An Interactive Platform for Plan-based Conditional Generation (2023.eacl-demo)
Copied to clipboard
Fantine Huot, Joshua Maynez, Shashi Narayan, Reinald Kim Amplayo, Kuzman Ganchev, Annie Priyadarshini Louis, Anders Sandholm, Dipanjan Das, Mirella Lapata
| Challenge: | Recent work shows that conditional generation models can be useful to control the text generation process, leading to irrelevant, repetitive, and hallucinated content. |
| Approach: | They propose a web browser-based demonstration for query-focused summarization that uses a sequence of question-answer pairs as a blueprint plan for guiding text generation. |
| Outcome: | The proposed model can be used to generate query-focused summarization text using question-answer pairs as a blueprint plan. |