Papers by Chulaka Gunasekara
X-FACTOR: A Cross-metric Evaluation of Factual Correctness in Abstractive Summarization (2022.emnlp-main)
Copied to clipboard
Subhajit Chaudhury, Sarathkrishna Swaminathan, Chulaka Gunasekara, Maxwell Crouse, Srinivas Ravishankar, Daiki Kimura, Keerthiram Murugesan, Ramón Fernandez Astudillo, Tahira Naseem, Pavan Kapanipathi, Alexander Gray
| Challenge: | Abstractive summarization models produce factually inconsistent summaries that are not supported by the original article. |
| Approach: | They propose a fact-aware filtering mechanism that improves the factuality of abstractive summarization models. |
| Outcome: | The proposed method improves the quality of training data and the factuality of generated summaries. |
doc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset (2020.emnlp-main)
Copied to clipboard
| Challenge: | doc2dial dataset is a goal-oriented document-grounded dialogue model . it is based on how the authors compose documents for guiding end users . |
| Approach: | They propose a dataset of goal-oriented dialogues grounded in documents . they use annotated conversations with an average of 14 turns to generate conversational utterances . |
| Outcome: | The proposed dataset includes over 4500 annotated conversations with an average of 14 turns grounded in over 450 documents from four domains. |
Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks (2024.emnlp-industry)
Copied to clipboard
Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, G P Shrivatsa Bhargav, Maxwell Crouse, Chulaka Gunasekara, Shajith Ikbal, Sachindra Joshi, Hima Karanam, Vineet Kumar, Asim Munawar, Sumit Neelam, Dinesh Raghu, Udit Sharma, Adriana Soria, Dheeraj Sreedhar, Praveen Venkateswaran, Merve Unuvar, David Cox, Salim Roukos, Luis Lastras, Pavan Kapanipathi
| Challenge: | Existing research explores the use of Large Language Models (LLMs) as the backbone of agentic systems. |
| Approach: | They propose a model trained using a multi-task training approach on seven fundamental tasks encompassed in function calling that has better generalizability on multiple tasks across seven evaluation benchmarks. |
| Outcome: | The proposed model outperforms more than 15 other models on out-of-domain datasets and ranks among the top on the Berkeley Function Calling Leaderboard (BFCL). |
Agent Assist through Conversation Analysis (2020.emnlp-demos)
Copied to clipboard
Kshitij Fadnis, Nathaniel Mills, Jatin Ganhotra, Haggai Roitman, Gaurav Pandey, Doron Cohen, Yosi Mass, Shai Erera, Chulaka Gunasekara, Danish Contractor, Siva Patel, Q. Vera Liao, Sachindra Joshi, Luis Lastras, David Konopnicki
| Challenge: | Using conversational approach to information retrieval for agent assistance, customer support agents are a critical part of an organization's customer support team. |
| Approach: | They propose a conversational approach to information retrieval for agent assistance that monitors an evolving conversation and recommends both responses and URLs of documents. |
| Outcome: | The proposed system monitors an evolving conversation and recommends both responses and URLs of documents the agent can use in replies to their client. |
MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types (2023.findings-acl)
Copied to clipboard
Keerthiram Murugesan, Sarathkrishna Swaminathan, Soham Dan, Subhajit Chaudhury, Chulaka Gunasekara, Maxwell Crouse, Diwakar Mahajan, Ibrahim Abdelaziz, Achille Fokoue, Pavan Kapanipathi, Salim Roukos, Alexander Gray
| Challenge: | Existing evaluation metrics for machine text are inadequate to capture quality of text . a recent study has focused on task-specific evaluation metrics or on properties of machine-generated text based on mismatch errors . |
| Approach: | They propose a new evaluation scheme based on fine-grained mismatch errors . they propose 13 mismatch error types to guide the model for better prediction of human judgments . |
| Outcome: | The proposed evaluation scheme is based on mismatch errors in 7 NLP tasks . the mismatch error types guide the model for better prediction of human judgments . |
TWEETSUMM - A Dialog Summarization Dataset for Customer Service (2021.findings-emnlp)
Copied to clipboard
Guy Feigenblat, Chulaka Gunasekara, Benjamin Sznajder, Sachindra Joshi, David Konopnicki, Ranit Aharonov
| Challenge: | a dataset focused on customer care dialog summarization is the first to focus on real-world customer care conversations . it contains extractive and abstractive summaries, and extractive summarizing methods are also introduced . |
| Approach: | They present a customer care dialog summarization dataset with 6500 human annotated summaries . they introduce an unsupervised method for extracting dialog summary data . |
| Outcome: | The proposed method is based on real-world customer support dialogs and includes extractive and abstractive summaries. |
Using Question Answering Rewards to Improve Abstractive Summarization (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Neural abstractive summarization models have seen improvements in recent years, but they still suffer from multiple drawbacks. |
| Approach: | They propose a general framework to train abstractive summarization models to alleviate these issues by question-answering based rewards. |
| Outcome: | The proposed framework is preferred over general abstractive summarization models. |
Conversational Document Prediction to Assist Customer Care Agents (2020.emnlp-main)
Copied to clipboard
Jatin Ganhotra, Haggai Roitman, Doron Cohen, Nathaniel Mills, Chulaka Gunasekara, Yosi Mass, Sachindra Joshi, Luis Lastras, David Konopnicki
| Challenge: | Using a conversational search system, the agent/system can ask clarification questions and interactively modify the search results as the conversation progresses. |
| Approach: | They propose to use a public dataset to analyze the task of predicting the documents that customer care agents can use to facilitate users’ needs. |
| Outcome: | The proposed model is more efficient than existing models and is more cost-effective than existing ones. |
The Benefits of Bad Advice: Autocontrastive Decoding across Model Layers (2023.acl-long)
Copied to clipboard
Ariel Gera, Roni Friedman, Ofir Arviv, Chulaka Gunasekara, Benjamin Sznajder, Noam Slonim, Eyal Shnarch
| Challenge: | Existing approaches to apply language models to tasks that require intermediate representations are less informative. |
| Approach: | They propose a novel approach that utilizes the contrast between layers to improve text generation outputs. |
| Outcome: | The proposed approach mitigates degenerative behaviors of the model in open-ended generation, significantly improving the quality of generated texts. |
Does Structure Matter? Encoding Documents for Machine Reading Comprehension (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing Transformer-based models for machine reading comprehension treat documents as flat sequences. |
| Approach: | They propose a Transformer-based method that reads a document as tree slices and jointly trains and consults the modules at inference time. |
| Outcome: | The proposed method outperforms several baseline approaches on two datasets from varied domains. |
Implicit Discourse Relation Classification: We Need to Talk about Evaluation (2020.acl-main)
Copied to clipboard
| Challenge: | Lack of consistency in preprocessing and evaluation poses challenges to fair comparison of results in literature. |
| Approach: | They propose an improved evaluation protocol for implicit relation classification on PDTB 2.0 . they report strong baseline results from pretrained sentence encoders . |
| Outcome: | The proposed evaluation protocol improves the existing framework and provides strong baseline results. |
A Large-Scale Corpus for Conversation Disentanglement (P19-1)
Copied to clipboard
Jonathan K. Kummerfeld, Sai R. Gouravajhala, Joseph J. Peper, Vignesh Athreya, Chulaka Gunasekara, Jatin Ganhotra, Siva Sankalp Patel, Lazaros C Polymenakos, Walter Lasecki
| Challenge: | a dataset of 77,563 messages manually annotated with reply-structure graphs disentangles conversations and defines internal conversation structure. |
| Approach: | They use a dataset of 77,563 messages manually annotated with reply-structure graphs to disentangle conversations and define internal conversation structure. |
| Outcome: | The new dataset is 16 times larger than all previous datasets combined and includes adjudication of annotation disagreements and context. |
Semi-Structured Object Sequence Encoders (2023.findings-emnlp)
Copied to clipboard
Rudra Murthy, Riyaz Bhat, Chulaka Gunasekara, Siva Patel, Hui Wan, Tejas Dhamecha, Danish Contractor, Marina Danilevsky
| Challenge: | Semi-structured object sequences are often represented as a sequence of key-value pairs over time . authors propose a two-part approach that takes each key independently and encodes a representation of its values over time. |
| Approach: | They propose a two-part approach that first considers each key independently and encodes a representation of its values over time. |
| Outcome: | The proposed approach outperforms existing methods on multiple prediction tasks using real-world data. |
Summary Grounded Conversation Generation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets for conversation summarization are small due to the lack of large-scale datasets. |
| Approach: | They propose three approaches to generate summary grounded conversations, and evaluate the generated conversations using automatic measures and human judgements. |
| Outcome: | The proposed models can generate entire conversations with only a summary of a conversation as the input. |
Explaining Neural Network Predictions on Sentence Pairs via Learning Word-Group Masks (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to explain neural network models are computationally inefficient for text inputs. |
| Approach: | They propose a method to implicitly detect word correlations by grouping correlated words from input text pairs together and measuring their contribution to corresponding NLP tasks. |
| Outcome: | The proposed method is evaluated with two different model architectures across four datasets. |