Papers by Sergio Servantez
Chain of Logic: Rule-Based Reasoning with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Logic models are prone to hallucinations and are not able to perform basic tasks like drafting and drafting documents. |
| Approach: | They propose a new prompting method which elicits rule-based reasoning through decomposition and recomposition. |
| Outcome: | The proposed method outperforms other prompting methods including chain of thought and self-ask on eight rule-based reasoning tasks. |
OpenExempt: A Diagnostic Benchmark for Legal Reasoning and a Framework for Creating Custom Benchmarks on Demand (2026.findings-acl)
Copied to clipboard
| Challenge: | Reasoning benchmarks are expensive to build and ill suited for isolating specific failure modes. |
| Approach: | They propose a framework and benchmark for diagnostic evaluation of legal reasoning that uses symbolic representations of U.S. Bankruptcy Code statutes to generate large space of reasoning tasks and their machine-computable solutions on demand. |
| Outcome: | The proposed framework and benchmark provides diagnostic insights into the competencies and failure modes of language models. |