Papers by Maxwell Crouse
X-FACTOR: A Cross-metric Evaluation of Factual Correctness in Abstractive Summarization (2022.emnlp-main)
Copied to clipboard
Subhajit Chaudhury, Sarathkrishna Swaminathan, Chulaka Gunasekara, Maxwell Crouse, Srinivas Ravishankar, Daiki Kimura, Keerthiram Murugesan, Ramón Fernandez Astudillo, Tahira Naseem, Pavan Kapanipathi, Alexander Gray
| Challenge: | Abstractive summarization models produce factually inconsistent summaries that are not supported by the original article. |
| Approach: | They propose a fact-aware filtering mechanism that improves the factuality of abstractive summarization models. |
| Outcome: | The proposed method improves the quality of training data and the factuality of generated summaries. |
NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls (2025.emnlp-main)
Copied to clipboard
Kinjal Basu, Ibrahim Abdelaziz, Kiran Kate, Mayank Agarwal, Maxwell Crouse, Yara Rizk, Kelsey Bradford, Asim Munawar, Sadhana Kumaravel, Saurabh Goyal, Xin Wang, Luis A. Lastras, Pavan Kapanipathi
| Challenge: | Existing benchmarks and datasets for tool calling have lagged behind . nested sequencing is a common problem in LLMs, but it is not enough to evaluate them. |
| Approach: | They propose a benchmark to evaluate LLMs on nested sequences of API calls, i.e. sequences where the output of one API call is passed as input to a subsequent call. |
| Outcome: | The proposed model achieves a full sequence match accuracy of 28% and a win-rate of 60% on nested sequences of API calls. |
Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks (2024.emnlp-industry)
Copied to clipboard
Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, G P Shrivatsa Bhargav, Maxwell Crouse, Chulaka Gunasekara, Shajith Ikbal, Sachindra Joshi, Hima Karanam, Vineet Kumar, Asim Munawar, Sumit Neelam, Dinesh Raghu, Udit Sharma, Adriana Soria, Dheeraj Sreedhar, Praveen Venkateswaran, Merve Unuvar, David Cox, Salim Roukos, Luis Lastras, Pavan Kapanipathi
| Challenge: | Existing research explores the use of Large Language Models (LLMs) as the backbone of agentic systems. |
| Approach: | They propose a model trained using a multi-task training approach on seven fundamental tasks encompassed in function calling that has better generalizability on multiple tasks across seven evaluation benchmarks. |
| Outcome: | The proposed model outperforms more than 15 other models on out-of-domain datasets and ranks among the top on the Berkeley Function Calling Leaderboard (BFCL). |
MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types (2023.findings-acl)
Copied to clipboard
Keerthiram Murugesan, Sarathkrishna Swaminathan, Soham Dan, Subhajit Chaudhury, Chulaka Gunasekara, Maxwell Crouse, Diwakar Mahajan, Ibrahim Abdelaziz, Achille Fokoue, Pavan Kapanipathi, Salim Roukos, Alexander Gray
| Challenge: | Existing evaluation metrics for machine text are inadequate to capture quality of text . a recent study has focused on task-specific evaluation metrics or on properties of machine-generated text based on mismatch errors . |
| Approach: | They propose a new evaluation scheme based on fine-grained mismatch errors . they propose 13 mismatch error types to guide the model for better prediction of human judgments . |
| Outcome: | The proposed evaluation scheme is based on mismatch errors in 7 NLP tasks . the mismatch error types guide the model for better prediction of human judgments . |
Self-Supervised Rule Learning to Link Text Segments to Relational Elements of Structured Knowledge (2023.findings-emnlp)
Copied to clipboard
Shajith Ikbal, Udit Sharma, Hima Karanam, Sumit Neelam, Ronny Luss, Dheeraj Sreedhar, Pavan Kapanipathi, Naweed Khan, Kyle Erwin, Ndivhuwo Makondo, Ibrahim Abdelaziz, Achille Fokoue, Alexander Gray, Maxwell Crouse, Subhajit Chaudhury, Chitra Subramanian
| Challenge: | Various approaches have been tried to map predicate components of a natural language (NL) text segment onto their corresponding predicates within a knowledge base (KB). |
| Approach: | They propose a neuro-symbolic approach to self-learn rules that serve as interpretable knowledge to perform relation linking in knowledge base question answering systems. |
| Outcome: | The proposed approach achieves an average performance gain of 17% on CLUTRR and relation linking in a KBQA system. |
API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs (2024.acl-long)
Copied to clipboard
Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Vernon Austel, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, Luis Lastras
| Challenge: | Existing methods to train and test large language models that involve calls to tools and APIs are lacking. |
| Approach: | They propose a large corpora for training and systematic testing of tool-augmented LLMs. |
| Outcome: | The proposed datasets mimic real-world scenarios involving API-tasks and slot filling. |
Laziness Is a Virtue When It Comes to Compositionality in Neural Semantic Parsing (2023.acl-long)
Copied to clipboard
Maxwell Crouse, Pavan Kapanipathi, Subhajit Chaudhury, Tahira Naseem, Ramon Fernandez Astudillo, Achille Fokoue, Tim Klinger
| Challenge: | Compositional generalization is a key feature of human intelligence and has been identified as a major point of weakness in neural methods for semantic parsing. |
| Approach: | They propose a neural parsing generation method that constructs logical forms from the bottom up, beginning from the logical form’s leaves. |
| Outcome: | The proposed method outperforms general-purpose parsers on a CFQ dataset and two other Text-to-SQL datasets while also being competitive with parser that have been tailored to each task. |