Papers by Soumya Sanyal
RobustLR: A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that perform deductive reasoning on inputs containing rules and statements in the English natural language do not perform consistently on the RobustLR test set. |
| Approach: | They propose a diagnostic benchmark that evaluates the robustness of language models to minimal logical edits in inputs and different logical equivalence conditions. |
| Outcome: | The proposed models do not perform consistently on the RobustLR test set. |
Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verification (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evidence that humans make numerous inferences to understand discourse and text is not fully understood. |
| Approach: | They propose to use textual inference datasets with multi-sentence premises to solve the entailment verification problem. |
| Outcome: | The proposed model outperforms GPT-3.5 and rivals GPL-4 in EV tasks. |
FaiRR: Faithful and Robust Deductive Reasoning over Natural Language (2022.acl-long)
Copied to clipboard
| Challenge: | Currently, black-box models generate both the proof graph and intermediate inferences within the same model and thus may be unfaithful. |
| Approach: | They propose a transformer-based model that can perform deductive reasoning on a logical rulebase containing rules and statements written in natural language. |
| Outcome: | The proposed model is robust to language perturbations and faster at inference than previous models on existing reasoning datasets. |
Self-contradictory reasoning evaluation and detection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown impressive reasoning ability, but many downstream reasoning tasks focus on performance-wise evaluation. |
| Approach: | They define and assess the Self-Contra rate across three datasets and delve into finer-grained categories of Self-contra reasoning. |
| Outcome: | The proposed model can detect self-contra reasoning with a 52.2% F1 score, much lower than for humans. |
Discretized Integrated Gradients for Explaining Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Integrated Gradients (IG) is widely adopted due to its desirable explanation axioms and the ease of gradient computation. |
| Approach: | They propose an attribution-based explanation algorithm that uses averaging the model's output gradient interpolated along a straight-line path in the input data space. |
| Outcome: | The proposed method is compared with IG on multiple sentiment classification datasets. |
APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning (2023.acl-long)
Copied to clipboard
Soumya Sanyal, Yichong Xu, Shuohang Wang, Ziyi Yang, Reid Pryzant, Wenhao Yu, Chenguang Zhu, Xiang Ren
| Challenge: | Existing methods to improve logical reasoning skills require complex data processing. |
| Approach: | They propose an adaptive pretraining approach to improve logical reasoning over text . they use a subset of Wikipedia sentences for pretraining and a sentence-level classification loss . |
| Outcome: | The proposed model outperforms baselines on LogiQA and ReClor. |
A Re-evaluation of Knowledge Graph Completion Methods (2020.acl-main)
Copied to clipboard
| Challenge: | Knowledge Graph Completion (KGC) aims at automatically predicting missing links for large-scale knowledge graphs. |
| Approach: | They propose a protocol to evaluate KGC methods that is robust to handle bias in the model, which can substantially affect the final results. |
| Outcome: | The proposed evaluation protocol is robust to handle bias in the model, which can substantially affect the final results. |
Mixing Inference-time Experts for Enhancing LLM Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for improving reasoning quality in large language models are limited to using a single expert. |
| Approach: | They propose a framework that finetunes and merges expert logits from one LLM . they use commonsense and entailment reasoning experts to improve chain-of-thought reasoning . |
| Outcome: | The proposed framework outperforms baselines on three question-answering datasets. |