FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) are prone to misuse and misinformation, posing serious compliance risks. |
| Approach: | They propose a bilingual red-teaming benchmark to test an LLM’s refusal of requests that violate financial compliance. |
| Outcome: | The proposed benchmark is based on real-world financial crime cases and ethical violations and includes 14 subcategories covering financial crimes and ethical breaches. |
Similar Papers
FinHarmBench: Financial Jailbreak Benchmark and Unsupervised Safety Fine-Tuning via Refusal Steering Distillation (2026.acl-industry)
Copied to clipboard
Yubin Choi, Yujin Yang, Subin Kim, Seokil Ham, Seungju Cho, Jungmin Son, Youngjun Kwak, Changick Kim
| Challenge: | Existing safety benchmarks focus on general harms and lack the granularity needed to capture domain-specific financial threats. |
| Approach: | They propose a benchmark to evaluate financially harmful and confusable benign prompts. |
| Outcome: | The proposed framework improves refusal behavior without annotating refusal responses. |
CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models (2025.naacl-long)
Copied to clipboard
Ying Nie, Binwei Yan, Tianyu Guo, Hao Liu, Haoyu Wang, Wei He, Binfan Zheng, Weihao Wang, Qiang Li, Weijian Sun, Yunhe Wang, Dacheng Tao
| Challenge: | Large language models (LLMs) have achieved remarkable performance on various NLP tasks, yet their potential in more challenging task like finance, has not been fully explored. |
| Approach: | They propose a benchmark to assess the financial knowledge of large language models (LLMs) in China. |
| Outcome: | The proposed benchmark is the most comprehensive evaluation benchmark to date for LLMs in finance. |
Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial Documents (2026.findings-acl)
Copied to clipboard
Ying He, Zhouhong Gu, Zhecheng Hu, Yubo Zhou, Hao Shen, Jiaqing Liang, Zhaoqian Dai, Ma Shuguang, Fei Yu, Yanghua Xiao, Zhixu Li
| Challenge: | Existing LLMs struggle to identify errors in financial documents, a study shows . 18% of financial practitioners make errors daily, one-third make errors several times weekly, and 59% make errors multiple times monthly. |
| Approach: | They introduce FinED-Bench, a publicly available Benchmark for financial error detection . it covers nine real-world financial scenarios and includes over 900 documents in 2025 . supervised fine-tuning can significantly improve the performance of weaker LLMs, they show . |
| Outcome: | The proposed benchmark covers nine real-world financial scenarios and includes over 900 documents reported in 2025 that are unseen by existing language models. |
Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducements (2025.findings-acl)
Copied to clipboard
Shu Yang, Shenzhe Zhu, Zeyu Wu, Keyu Wang, Junchi Yao, Junchao Wu, Lijie Hu, Mengdi Li, Derek F. Wong, Di Wang
| Challenge: | Existing fraud detection benchmarks focus on single-turn classification tasks, failing to capture dynamic nature of real-world fraud attempts. |
| Approach: | They propose a bilingual benchmark to assess LLMs' ability to resist fraud and phishing attacks across five key fraud categories: Fraudulent Services, Impersonation, Phishing Scams, Fake Job Postings, and Online Relationships. |
| Outcome: | The proposed model improves in role-play settings and in e-commerce and recommendation systems. |
Is Safety Standard Same for Everyone? User-Specific Safety Evaluation of Large Language Models (2025.findings-emnlp)
Copied to clipboard
Yeonjun In, Wonjoong Kim, Kanghoon Yoon, Sungchul Kim, Mehrab Tanjim, Sangwu Park, Kibum Kim, Chanyoung Park
| Challenge: | Extensive benchmarks evaluate LLM safety relying heavily on general standards . no benchmark datasets exist to evaluate the user-specific safety of LLMs . |
| Approach: | a new benchmark is designed to assess user-specific aspect of LLM safety . authors propose a simple remedy based on chain-of-thought to improve user-specified safety. |
| Outcome: | a new benchmark assesses the user-specific aspect of LLM safety . the proposed solution improves user-specified safety by chain-of-thought . |
INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent (2025.acl-long)
Copied to clipboard
Haohang Li, Yupeng Cao, Yangyang Yu, Shashidhar Reddy Javaji, Zhiyang Deng, Yueru He, Yuechen Jiang, Zining Zhu, K.p. Subbalakshmi, Jimin Huang, Lingfei Qian, Xueqing Peng, Jordan W. Suchow, Qianqian Xie
| Challenge: | Recent advances have underscored the potential of large language model (LLM)-based agents in financial decision-making. |
| Approach: | They propose to evaluate LLM agents using 13 different LLMs as backbone models across various market environments and tasks. |
| Outcome: | The proposed framework assesses the reasoning and decision-making capabilities of 13 different LLMs across various market environments and tasks. |
FinMaster: A Holistic Benchmark for Full-Pipeline Financial Management with Large Language Models (2026.findings-acl)
Copied to clipboard
Junzhe Jiang, Chang Yang, Aixin Cui, Sihan Jin, Yujing Zhang, Yilin Xiao, Ruiyu Wang, Bo Li, Xiao Huang, Danny Dongning Sun, Xinrun Wang
| Challenge: | Existing benchmarks lack domain-specific data, realistic workflow-level task design, and standardized workflow- level evaluation. |
| Approach: | a new benchmark evaluates large language models on financial management workflows . the global financial services market is projected to grow to $37 trillion by 2027 . |
| Outcome: | a new benchmark for large language models on financial management workflows reveals critical capability gaps . accuracy drops from 90% on basic tasks to 40% on complex scenarios requiring multi-step reasoning . the global financial services market reached $25.8 trillion in 2022 and is projected to grow to $37 trillion by 2027 . |
Learning to Conceal Risk: Controllable Multi-turn Red Teaming for LLMs in the Financial Domain (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly deployed in high-stakes domains such as finance where unsafe behavior can lead to serious regulatory risks. |
| Approach: | They propose a black-box multi-turn risk-concealed redteaming framework that progressively conceals surface-level risk while exploiting regulatory-violating behaviors. |
| Outcome: | Experiments on nine widely used LLMs show that the proposed framework achieves 93.19% average attack success rate (ASR) and improves the average ASR to 95.00%. |
Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection (2026.findings-acl)
Copied to clipboard
Zhiwei Liu, Yupeng Cao, Yuechen Jiang, Mohsinul Kabir, Polydoros Giannouris, Chen Xu, Ziyang Xu, Tianlei Zhu, Md. Tariquzzaman, Triantafillos Papadopoulos, Yan Wang, Lingfei Qian, Xueqing Peng, Zhuohan Xie, Ye Yuan, Saeed Almheiri, Abdulrazzaq Alnajjar, Ming-Bin Chen, Harry Stuart, Paul Thompson, Prayag Tiwari, Alejandro Lopez-Lira, Xue Liu, Jimin Huang, Sophia Ananiadou
| Challenge: | Existing research on LLM biases has focused on direct questioning or general-purpose settings . pronounced behavioral biase despite their growing deployment in financial analysis, forecasting, and decision support. |
| Approach: | They propose a benchmark to evaluate behavioral biases of large language models in MFMD . they use a multilingual financial misinformation dataset to integrate these with misinformation claims . |
| Outcome: | The proposed benchmark evaluates behavioral biases of large language models across economic scenarios. |
Fin-Bias: Comprehensive Evaluation for LLM Decision-Making under human bias in Finance Domain (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models (LLMs) are limited to small sample and fail to demonstrate LLM susceptibility to context with potential human bias. |
| Approach: | They propose a benchmark for evaluating LLM investment decision-making when faced with uncertainty and possible human-biased opinions. |
| Outcome: | The proposed model can herd the explicit bias in context and even exceed human performance in predicting future stock return. |