JFLD: A Japanese Benchmark for Deductive Reasoning Based on Formal Logic (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have proficiently solved a broad range of tasks with their rich knowledge but struggle with logical reasoning. |
| Approach: | They propose a deductive reasoning benchmark for Japanese that assesses logical reasoning abilities isolated from knowledge and various reasoning rules. |
| Outcome: | The proposed benchmarks assess whether LLMs can generate logical steps to (dis)prove a given hypothesis based on a set of facts. |
Similar Papers
JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) focus on general domains, with fewer advancements in Japanese biomedical LLMs. |
| Approach: | They propose a benchmark for Japanese large language models with eight LLMs across four categories and 20 Japanese biomedical datasets for comparison. |
| Outcome: | The proposed benchmark includes eight LLMs across four categories and 20 Japanese biomedical datasets across five tasks. |
JGLUE: Japanese General Language Understanding Evaluation (2022.lrec-1)
Copied to clipboard
| Challenge: | There is no benchmark for Japanese to evaluate and analyze NLU ability from different perspectives. |
| Approach: | They build a Japanese NLU benchmark from scratch without translation to measure general NLU ability in Japanese. |
| Outcome: | a Japanese NLU benchmark is built from scratch without translation to measure general NLU ability in Japanese. |
LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models (2024.acl-long)
Copied to clipboard
Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral
| Challenge: | Existing work investigating the logical reasoning ability of large language models has focused only on a couple of inference rules of propositional and first-order logics. |
| Approach: | They propose to use a natural language question-answering dataset to evaluate the logical reasoning ability of large language models. |
| Outcome: | The proposed model performs poorly on a range of natural language questions using chain-of-thought prompting. |
Constructing a Japanese Verdict Prediction Dataset for Fact-Checking of LLM-Generated Texts (2026.acl-srw)
Copied to clipboard
Miwa Masano, Hirokazu Kiyomaru, Atsushi Keyaki, Kaito Horio, Rei Minamoto, Ribeka Keyaki, Kouta Nakayama, Hideyuki Tachibana, Daisuke Kawahara
| Challenge: | Text generated by Large Language Models (LLMs) may contain plausible but incorrect information known as hallucinations. |
| Approach: | They extend the label set for verdict prediction to capture claim-evidence relationships humans would commonly interpret as supported or refuted. |
| Outcome: | The proposed system improves F1 by 4 percentage points compared to baseline. |
JBLiMP: Japanese Benchmark of Linguistic Minimal Pairs (2023.findings-eacl)
Copied to clipboard
| Challenge: | In this paper, we compare syntactic knowledge of language models across different languages. |
| Approach: | They introduce a dataset for targeted syntactic evaluations of language models in Japanese. |
| Outcome: | The proposed dataset compares the syntactic knowledge of language models across languages. |
DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing logic reasoning benchmarks are limited in language diversity and their distributions are deviated from ideal distributions, which may lead to biased evaluation results. |
| Approach: | They propose a new logic benchmark DivLogicEval that uses natural sentences to evaluate logical reasoning . |
| Outcome: | The proposed evaluation metric mitigates bias and randomness inherent in LLMs. |
Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on diagnosing LMs' reasoning abilities in natural language understanding tasks. |
| Approach: | They propose a diagnostic method for first-order logic reasoning with a proposed benchmark, LogicNLI. |
| Outcome: | The proposed method disentangles the target FOL reasoning from commonsense inference and can be used to diagnose LMs from four perspectives: accuracy, robustness, generalization, and interpretability. |
Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models (2024.emnlp-main)
Copied to clipboard
Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varshney, Chitta Baral
| Challenge: | Existing logical reasoning evaluation benchmarks focus on simplistic single-step or multi-step reasoning with limited set of inference rules. |
| Approach: | They propose to use a multi-step logical reasoning evaluation dataset to measure their ability for human-like multi- step logical thinking. |
| Outcome: | The proposed dataset covers three logic types including propositional, first-order, and non-monotonic logic with various inference rules and depths. |
Reason from Fallacy: Enhancing Large Language Models’ Logical Reasoning through Logical Fallacy Understanding (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated good performance in many reasoning tasks, but struggle with some more complex reasoning tasks including logical reasoning. |
| Approach: | They propose five concrete tasks from three cognitive dimensions of WHAT, WHY, and HOW to evaluate LLMs’ capability of logical fallacy understanding. |
| Outcome: | The proposed dataset can be used to evaluate LLMs’ LFU capability and to fine-tune LLM models to obtain significantly enhanced performance on logical reasoning. |
Evaluating Large Language Models with Enterprise Benchmarks (2025.naacl-industry)
Copied to clipboard
Bing Zhang, Mikio Takeuchi, Ryo Kawahara, Shubhi Asthana, Maruf Hossain, Guang-Jie Ren, Kate Soule, Yifan Mai, Yada Zhu
| Challenge: | Existing benchmarks lack domain-specific datasets for evaluating large language models . existing benchmarks often lack domain specific datasets, which can be difficult to convert to standardized metrics or regulatory issues. |
| Approach: | They propose to use 25 publicly available domain-specific English benchmarks from diverse domains . they propose to combine a wide range of natural language processing tasks for holistic evaluation . |
| Outcome: | The proposed framework includes 25 publicly available domain-specific English benchmarks from diverse enterprise domains like financial services, legal, climate, cyber security, and 2 public Japanese finance benchmarks. |