Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models (2025.findings-emnlp)
Copied to clipboard
Xiaojun Wu, Junxi Liu, Huan-Yi Su, Zhouchi Lin, Yiyan Qi, Chengjin Xu, Jiajun Su, Jiajie Zhong, Fuwei Wang, Saizhuo Wang, Fengrui Hua, Jia Li, Jian Guo
| Challenge: | Existing financial benchmarks suffer from limited language and task coverage, low-quality datasets, and inadequate adaptability for LLM evaluation. |
| Approach: | They propose a bilingual benchmark for financial LLMs that assesses models’ language understanding and generation capabilities. |
| Outcome: | The proposed bilingual benchmark assesses models’ language understanding and generation capabilities. |
Similar Papers
FinMaster: A Holistic Benchmark for Full-Pipeline Financial Management with Large Language Models (2026.findings-acl)
Copied to clipboard
Junzhe Jiang, Chang Yang, Aixin Cui, Sihan Jin, Yujing Zhang, Yilin Xiao, Ruiyu Wang, Bo Li, Xiao Huang, Danny Dongning Sun, Xinrun Wang
| Challenge: | Existing benchmarks lack domain-specific data, realistic workflow-level task design, and standardized workflow- level evaluation. |
| Approach: | a new benchmark evaluates large language models on financial management workflows . the global financial services market is projected to grow to $37 trillion by 2027 . |
| Outcome: | a new benchmark for large language models on financial management workflows reveals critical capability gaps . accuracy drops from 90% on basic tasks to 40% on complex scenarios requiring multi-step reasoning . the global financial services market reached $25.8 trillion in 2022 and is projected to grow to $37 trillion by 2027 . |
Benchmarking Large Language Models on CFLUE - A Chinese Financial Language Understanding Evaluation Dataset (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have revolutionized natural language processing (NLP) there is an urgent need for new benchmarks to keep pace with the development of LLMs. |
| Approach: | They propose a benchmark to assess the capability of large language models (LLMs) they use a dataset to provide both knowledge assessment and application assessment . |
| Outcome: | The proposed benchmark provides datasets tailored for knowledge assessment and application assessment. |
MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application (2026.acl-long)
Copied to clipboard
Xueqing Peng, Lingfei Qian, Yan Wang, Ruoyu Xiang, Yueru He, Yang Ren, Mingyang Jiang, Vincent Jim Zhang, Yuqing Guo, Jeff Zhao, Huan He, Yi Han, Yun Feng, Yuechen Jiang, Yupeng Cao, Haohang Li, Yangyang Yu, Xiaoyu Wang, Penglei Gao, Shengyuan Lin, Keyi Wang, Shanshan Yang, Yilun Zhao, Zhiwei Liu, Peng Lu, Jerry Huang, Suyuchen Wang, Triantafillos Papadopoulos, Polydoros Giannouris, Efstathia Soufleri, Nuo Chen, Zhiyang Deng, Heming Fu, Yijia Zhao, Mingquan Lin, Meikang Qiu, Kaleb E Smith, Arman Cohan, Xiao-Yang Liu, Jimin Huang, Guojun Xiong, Alejandro Lopez-Lira, Xi Chen, Junichi Tsujii, Jian-Yun Nie, Sophia Ananiadou, Qianqian Xie
| Challenge: | Existing evaluations of LLMs in finance are text-only, monolingual, and largely saturated by current models. |
| Approach: | They propose a multilingual and multimodal benchmark for evaluating LLMs in real financial contexts. |
| Outcome: | The first expert-annotated multilingual and multimodal benchmark is released . it evaluates 21 leading LLMs and shows they perform better in multilingual settings . |
CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models (2025.naacl-long)
Copied to clipboard
Ying Nie, Binwei Yan, Tianyu Guo, Hao Liu, Haoyu Wang, Wei He, Binfan Zheng, Weihao Wang, Qiang Li, Weijian Sun, Yunhe Wang, Dacheng Tao
| Challenge: | Large language models (LLMs) have achieved remarkable performance on various NLP tasks, yet their potential in more challenging task like finance, has not been fully explored. |
| Approach: | They propose a benchmark to assess the financial knowledge of large language models (LLMs) in China. |
| Outcome: | The proposed benchmark is the most comprehensive evaluation benchmark to date for LLMs in finance. |
Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized general natural language preprocessing tasks, but their performance in financial domains is not evaluated comprehensively. |
| Approach: | They propose a framework to evaluate financial language models on financial tasks . they compare performance of auto-encoding language models and ChatGPT . |
| Outcome: | The proposed framework compares the performance of auto-encoding language models and the LLM ChatGPT on financial tasks. |
FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation (2025.acl-long)
Copied to clipboard
Junyu Luo, Zhizhuo Kou, Liming Yang, Xiao Luo, Jinsheng Huang, Zhiping Xiao, Jingshu Peng, Chengzhong Liu, Jiaming Ji, Xuanzhe Liu, Sirui Han, Ming Zhang, Yike Guo
| Challenge: | Multimodal Large Language Models (MLLMs) have experienced rapid development in recent years, but there is a notable lack of effective and specialized multimodal evaluation datasets in the financial domain. |
| Approach: | They introduce FinMME, a multimodal large language model with 11,000 financial research samples and 20 annotators. |
| Outcome: | The proposed model performs better than state-of-the-art models, highlighting its challenging nature. |
FINKRX: Establishing Best Practices for Korean Financial NLP (2025.acl-industry)
Copied to clipboard
| Challenge: | Existing tools to evaluate large language models in the financial domain are limited by the inherently closed nature of the financial industry. |
| Approach: | They present the first open leaderboard for evaluating Korean large language models focused on finance. |
| Outcome: | The proposed model is FINKRX, a fully open and transparent LLM built using these best practices. |
FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models (2025.naacl-long)
Copied to clipboard
Xin Guo, Haotian Xia, Zhaowei Liu, Hanyang Cao, Zhi Yang, Zhiqiang Liu, Sizhe Wang, Jinyi Niu, Chuqi Wang, Yanhui Wang, Xiaolong Liang, Xiaoming Huang, Bing Zhu, Zhongyu Wei, Yun Chen, Weining Shen, Liwen Zhang
| Challenge: | Large language models have demonstrated outstanding performance in various natural language processing tasks, but their security capabilities in the financial domain have not been explored. |
| Approach: | They propose to use a benchmark to evaluate large language models' financial domain knowledge and practical abilities. |
| Outcome: | The proposed benchmark evaluates large language models' financial domain knowledge and practical abilities. |
Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance (2025.emnlp-main)
Copied to clipboard
Xueqing Peng, Triantafillos Papadopoulos, Efstathia Soufleri, Polydoros Giannouris, Ruoyu Xiang, Yan Wang, Lingfei Qian, Jimin Huang, Qianqian Xie, Sophia Ananiadou
| Challenge: | Greek is the dominant language of the world's merchant navy and is a key language for international trade. |
| Approach: | They propose to develop a Greek financial evaluation benchmark and a financial LLM fine-tuned on Greek-specific financial data to bridge this gap. |
| Outcome: | The proposed benchmarks surpass GPT-4 by 8.33%, GPT- 4o by 26.83%, and Deepseek-V3 by 67.74%. |
UCFE: A User-Centric Financial Expertise Benchmark for Large Language Models (2025.findings-naacl)
Copied to clipboard
Yuzhe Yang, Yifei Zhang, Yan Hu, Yilin Guo, Ruoli Gan, Yueru He, Mingcong Lei, Xiao Zhang, Haining Wang, Qianqian Xie, Jimin Huang, Honghai Yu, Benyou Wang
| Challenge: | Recent advances in large language models (LLMs) have expanded their potential applications in finance. |
| Approach: | They propose a framework to evaluate the ability of large language models to handle financial tasks using human expert evaluations and task-specific interactions. |
| Outcome: | The proposed framework evaluates the ability of large language models to handle complex financial tasks and combines human expert evaluations with dynamic, task-specific interactions to simulate the complexities of evolving financial scenarios. |