Exploring the Effectiveness of Prompt Engineering for Legal Reasoning Tasks (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that Chain-of-Thought (CoT) prompts improve tasks such as arithmetic and common-sense reasoning. |
| Approach: | They evaluate CoT prompts and various prompting strategies for legal reasoning tasks . they find that the best results are achieved with prompts derived from specific legal reasoning techniques . |
| Outcome: | The proposed approaches improve the COLIEE entailment task on the Japanese bar exam . the proposed approaches surpass the best system from 2022 with an accuracy of 0.789 . |
Similar Papers
Prompt Space Optimizing Few-shot Reasoning Success with Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Prompt engineering is an essential technique for enhancing the abilities of large language models (LLMs) by providing explicit and specific instructions. |
| Approach: | They propose a new approach that uses text embeddings to obtain basis vectors by matrix decomposition and constructs a space for representing all prompts. |
| Outcome: | The proposed approach significantly outperforms state-of-the-art prompt paradigms on ten public reasoning benchmarks. |
“Well, Keep Thinking”: Enhancing LLM Reasoning with Adaptive Injection Decoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit strong reasoning abilities, often attributed to few-shot or zero-shot Chain-of-Thought (CoT) prompting. |
| Approach: | They propose a decoding strategy that nudges LLMs to continue reasoning, thereby preventing immature reasoning processes. |
| Outcome: | The proposed method significantly improves LLM reasoning capabilities on diverse reasoning benchmarks. |
Reasoning for Translation: Comparative Analysis of Chain-of-Thought and Tree-of-Thought Prompting for LLM Translation (2025.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been used for specialized tasks but their application to machine translation has received little attention. |
| Approach: | They evaluate reasoning-based prompting strategies across multiple language pairs and domains and measure their effect on translation quality. |
| Outcome: | The proposed prompting strategies outperform traditional prompting methods across language pairs and domains and achieve improvements of up to 6.4 BLs. |
Why Prompt Design Matters and Works: A Complexity Analysis of Prompt Search Space in LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) prompting has emerged as a practical workaround, but most CoT-based methods rely on a single generic prompt like “think step by step” with no task-specific adaptation. |
| Approach: | They propose a theoretical framework that explains why some prompts succeed while others fail by using a generic generic prompt like "think step by step" they show that prompts function as selectors, extracting specific task-relevant information from the model's full hidden state during CoT reasoning. |
| Outcome: | The proposed framework explains why some prompts succeed while others fail. |
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have recently been shown to deliver impressive performance in various NLP tasks. |
| Approach: | They propose a plan-and-solve (PS) prompting that includes a few manual steps to generate reasoning steps and improves the quality of generated reasoning steps. |
| Outcome: | The proposed strategy outperforms Zero-shot-CoT on ten reasoning problems and has comparable performance to 8-shot CoT prompting on the math reasoning problem. |
What Makes Chain-of-Thought Prompting Effective? A Counterfactual Study (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using a few-shot prompt, we examine the effects of symbols and patterns on in-context learning in large language models. |
| Approach: | They employ a counterfactual prompting approach by manipulating examples and testing the consequences on model behavior. |
| Outcome: | The proposed approach allows us to understand the relative contributions of symbols and patterns on in-context learning. |
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters (2023.acl-long)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) prompting can dramatically improve the multi-step reasoning abilities of large language models (LLMs). |
| Approach: | They propose to use Chain-of-Thought (CoT) prompting to encourage the LLM to generate intermediate rationales for solving a problem by providing a series of reasoning steps in the demonstrations. |
| Outcome: | The proposed model can generate coherent lines of reasoning even with invalid demonstrations while still generating coherent lines during inference. |
Better Zero-Shot Reasoning with Role-Play Prompting (2024.naacl-long)
Copied to clipboard
Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, Xiaohang Dong
| Challenge: | Recent years have witnessed a paradigm shift in natural language processing, driven by large language models such as GPT-3, PaLM, and Llama. |
| Approach: | They propose a strategy for role-play prompting and assess its performance under the zero-shot setting. |
| Outcome: | The proposed method outperforms the standard zero-shot prompting approach across 12 reasoning benchmarks. |
A Systematic Survey of Automatic Prompt Optimization Techniques (2025.emnlp-main)
Copied to clipboard
Kiran Ramnath, Kang Zhou, Sheng Guan, Soumya Smruti Mishra, Xuan Qi, Zhengyuan Shen, Shuai Wang, Sangmin Woo, Sullam Jeoung, Yawei Wang, Haozhu Wang, Han Ding, Yuzhe Lu, Zhichao Xu, Yun Zhou, Balasubramaniam Srinivasan, Qiaojing Yan, Yueyan Chen, Haibo Ding, Panpan Xu, Lin Lee Cheong
| Challenge: | Recent advances in prompt engineering have created impediments for end users to adopt . however, prompt engineering remains an impedance due to rapid advances in models, tasks, and associated best practices. |
| Approach: | They propose to define APO as a 5-part unifying framework and categorize all relevant works based on their salient features. |
| Outcome: | The proposed framework aims to improve the performance of large language models on various tasks. |
Active Prompting with Chain-of-Thought for Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to annotate large language models rely on a fixed set of human-annotated exemplars, which are not always the most effective for different tasks. |
| Approach: | They propose a method to adapt large language models to different tasks with task-specific example prompts (annotated with human-designed CoT reasoning) they introduce several metrics to characterize uncertainty so as to select the most uncertain questions for annotation. |
| Outcome: | The proposed method significantly improves performance on eight complex reasoning tasks. |