Papers by Ming Shen
Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shaping (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing LLMs’ short-context reasoning but falters in long-contemporal scenarios requiring precise grounding and multi-hop reasoning. |
| Approach: | They propose a framework that constructs high-difficulty, multi-hop long-context QA pairs with inherent reasoning chains to overcome this bottleneck. |
| Outcome: | The proposed framework outperforms RLVR baselines and matches frontier LLMs while using far fewer parameters. |
Aligning VLM Assistants with Personalized Situated Cognition (2025.acl-long)
Copied to clipboard
Yongqi Li, Shen Zhou, Xiaohu Li, Xin Miao, Jintao Wen, Mayi Xu, Jianhao Chen, Birong Pan, Hankun Kang, Yuanyuan Zhu, Ming Zhong, Tieyun Qian
| Challenge: | Existing studies on vision-language models aligned with general human objectives have not been successful because people with diversified backgrounds have different cognition even in the same situation. |
| Approach: | They propose to characterize individuals based on the sociological concept of Role-Set and then evaluate their actions to see whether personalized alignment is achieved. |
| Outcome: | The proposed framework constructs a cognition-aware and action-based reward model for personalized alignment. |
Experience-driven Multi-turn Reinforcement Learning for GUI Agents (2026.acl-long)
Copied to clipboard
Zhengxi Lu, Jiabo Ye, Fei Tang, Yongliang Shen, Haiyang Xu, Ziwei Zheng, Weiming Lu, Ming Yan, Fei Huang, Jun Xiao, Yueting Zhuang
| Challenge: | GUI agents have demonstrated remarkable progress in automating complex user interface interactions . training such agents for long-horizon tasks remains challenging due to limited rewards and prohibitive costs. |
| Approach: | They propose a method that leverages expert trajectories as environment experiences for on-policy multi-turn training. |
| Outcome: | The proposed method achieves significant gains over the base model with 1K public trajectories as RL experiences . it achieves competitive performance against strong baselines such as UI-TARS-7B and GPT-4o . |
SocialBench: Sociality Evaluation of Role-Playing Conversational Agents (2024.findings-acl)
Copied to clipboard
Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Gao Xing, Weizhou Shen, Xiaojun Quan, Chenliang Li, Ji Zhang, Fei Huang
| Challenge: | Existing studies on role-playing agents have focused on enhancing their conversational capability, role-specific knowledge and style, but there has been a gap in assessing their social intelligence. |
| Approach: | They propose a benchmark to evaluate the sociality of role-playing agents using LLMs. |
| Outcome: | The proposed benchmark is constructed from various sources and covers a wide range of 500 characters and over 6,000 question prompts and 30,800 multi-turn role-playing utterances. |
Governance in Motion: Co-evolution of Constitutions and AI models for Scalable Safety (2025.emnlp-main)
Copied to clipboard
Chenhao Huang, Ziyu Shen, Yicong Ren, Huiyuan Zheng, Jiazheng Zhang, Mingxu Chai, Ming Zhang, Shihan Dou, Fan Mo, Jie Shi, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing approaches to align large language models with human preferences lack flexibility . static alignment preferences lack the ability to correct misaligned behaviors as they emerge . |
| Approach: | They propose a framework that enables dynamic and continuous alignment of large language models with human preferences. |
| Outcome: | The proposed framework improves safety and accuracy of a 7B model with human annotations. |
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models perform well on commonsense-reasoning benchmark datasets, but building machines with commonsence to compose plausible sentences remains challenging. |
| Approach: | They propose a constrained text generation task for generative commonsense reasoning that generates a coherent sentence using common concepts. |
| Outcome: | The proposed task generates a coherent sentence describing an everyday scenario using common concepts over 35k concept-sets. |
MAPS: Motivation-Aware Personalized Search via LLM-Driven Consultation Alignment (2025.acl-long)
Copied to clipboard
| Challenge: | Existing personalized product search methods assume that users’ query fully captures their real motivation, but in practice, user's queries do not always articulate the requirements. |
| Approach: | They propose a Motivation-Aware Personalized Search method that embeds queries and consultations into a unified semantic space via LLMs and utilizes a Mixture of Attention Experts (MoAE) to prioritize critical semantics. |
| Outcome: | Extensive experiments on real and synthetic data show that the proposed method outperforms existing methods in retrieval and ranking tasks. |
Reaction Miner: An Integrated System for Chemical Reaction Extraction from Textual Data (2023.emnlp-demo)
Copied to clipboard
Ming Zhong, Siru Ouyang, Yizhu Jiao, Priyanka Kargupta, Leo Luo, Yanzhen Shen, Bobby Zhou, Xianrui Zhong, Xuan Liu, Hongxiang Li, Jinfeng Xiao, Minhao Jiang, Vivian Hu, Xuan Wang, Heng Ji, Martin Burke, Huimin Zhao, Jiawei Han
| Challenge: | Reaction Miner is a system designed to extract chemical reactions from raw scientific PDFs. |
| Approach: | They propose a system that extracts chemical reactions directly from raw scientific PDFs. |
| Outcome: | The proposed system can extract chemical reactions from raw scientific PDFs. |
TransferTOD: A Generalizable Chinese Multi-Domain Task-Oriented Dialogue System with Transfer Capabilities (2024.emnlp-main)
Copied to clipboard
Ming Zhang, Caishuang Huang, Yilong Wu, Shichun Liu, Huiyuan Zheng, Yurui Dong, Yujiong Shen, Shihan Dou, Jun Zhao, Junjie Ye, Qi Zhang, Tao Gui, Xuanjing Huang
| Challenge: | Current datasets cater to user-led systems and are limited to predefined specific scenarios and slots. |
| Approach: | They propose to use a Chinese dialogue dataset to train a model that authentically simulates human-computer dialogues in 30 popular life service scenarios. |
| Outcome: | The proposed model achieves a joint accuracy of 75.09% in out-of-domain evaluations . it also achieves notable abilities in slot filling and questioning . |
Writing-RL: Advancing Long-form Writing via Adaptive Curriculum Reinforcement Learning (2026.acl-long)
Copied to clipboard
Xuanyu Lei, Chenliang Li, Yuning Wu, Kaiming Liu, Weizhou Shen, Peng Li, Ming Yan, Fei Huang, Ya-Qin Zhang, Yang Liu
| Challenge: | Recent advances in Large Language Models (LLMs) have enabled strong performance in long-form writing, but current training paradigms remain limited. |
| Approach: | They propose an Adaptive Curriculum Reinforcement Learning framework to advance long-form writing capabilities beyond SFT. |
| Outcome: | Experiments on 7B-scale writer models show that Writing-RL improves long-form writing performance over strong SFT baselines. |
Mutual-Taught for Co-adapting Policy and Reward Models (2025.acl-long)
Copied to clipboard
Tianyuan Shi, Canbin Huang, Fanqi Wan, Longguang Zhong, Ziyi Yang, Weizhou Shen, Xiaojun Quan, Ming Yan
| Challenge: | Experimental results show that this iterative approach leads to consistent improvements in both the policy model and reward model. |
| Approach: | They propose a method that iteratively improves both the policy model and reward model without requiring additional human annotation. |
| Outcome: | The proposed method improves both the policy model and reward model without human annotation. |
PFDial: A Structured Dialogue Instruction Fine-tuning Method Based on UML Flowcharts (2025.findings-acl)
Copied to clipboard
Ming Zhang, Yuhui Wang, Yujiong Shen, Tingyi Yang, Changhao Jiang, Yilong Wu, Shihan Dou, Qinhao Chen, Zhiheng Xi, Zhihao Zhang, Yi Dong, Zhen Wang, Zhihui Fei, Mingyang Wan, Tao Liang, Guojun Ma, Qi Zhang, Tao Gui, Xuanjing Huang
| Challenge: | Large Language Models (LLMs) have shown remarkable progress in dialogue and reasoning, but they struggle to solve strictly constrained dialogue tasks. |
| Approach: | They construct a dataset that contains 12,705 high-quality Chinese dialogue instructions from 440 flowcharts containing 5,055 process nodes. |
| Outcome: | The proposed model outperforms GPT-4o models on backward transitions and outperformed GPT-42 models on the same dataset. |
Small LLMs Are Weak Tool Learners: A Multi-LLM Agent (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized natural language processing with impressive capabilities, but they lack domain specificity, real-time information and face challenges in solving specialized problems. |
| Approach: | They propose a multi-LLM approach that decomposes the aforementioned capabilities into a planner, caller, and summarizer. |
| Outcome: | The proposed model outperforms existing models by demonstrating its effectiveness and advantages in tool learning. |
TriggerNER: Learning with Entity Triggers as Explanations for Named Entity Recognition (2020.acl-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a fundamental information extraction task that focuses on extracting entities from a given text and classifying them using pre-defined categories. |
| Approach: | They propose to use “entity triggers” to facilitate label-efficient learning of NER models. |
| Outcome: | The proposed model is significantly more cost-effective than the traditional neural NER frameworks. |
PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models (LMs) are a powerful transfer learning approach for knowledge graph (KG) completion. |
| Approach: | They propose a parameter-lite transfer learning approach for pretrained language models for knowledge graph (KG) completion. |
| Outcome: | The proposed model outperforms the state-of-the-art models on a knowledge graph completion benchmark by tuning 1% of the parameters. |
Visual Zero-Shot E-Commerce Product Attribute Value Extraction (2025.naacl-industry)
Copied to clipboard
| Challenge: | Existing zero-shot product attribute value extraction approaches require sellers to manually provide product descriptions. |
| Approach: | They propose a cross-modal zero-shot attribute value generation framework based on CLIP that uses product images as inputs for zero- shot inference. |
| Outcome: | The proposed framework significantly outperforms other vision-language models for zero-shot attribute value extraction. |
TRIGO: Benchmarking Formal Mathematical Proof Reduction for Generative Language Models (2023.emnlp-main)
Copied to clipboard
Jing Xiong, Jianhao Shen, Ye Yuan, Haiming Wang, Yichun Yin, Zhengying Liu, Lin Li, Zhijiang Guo, Qingxing Cao, Yinya Huang, Chuanyang Zheng, Xiaodan Liang, Ming Zhang, Qun Liu
| Challenge: | Automated theorem proving (ATP) benchmarks focus on symbolic inference but rarely involve understanding complex number combination reasoning. |
| Approach: | They propose a benchmark that requires a model to reduce a trigonometric expression with step-by-step proof and evaluates a generative LM’s reasoning ability on formulas and ability to manipulate, group, and factor number terms. |
| Outcome: | The proposed benchmark evaluates a generative LM’s reasoning ability on formulas and ability to manipulate, group, and factor number terms. |
LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models (2026.acl-long)
Copied to clipboard
Ming Zhang, Yujiong Shen, Jingyi Deng, Yuhui Wang, Huayu Sha, Kexin Tan, Qiyuan Peng, Yue Zhang, Junzhe Wang, Shichun Liu, Yueyuan Huang, Jingqi Tong, Changhao Jiang, Yilong Wu, Zhihao Zhang, Mingqi Wu, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing evaluation of Large Language Models on static benchmarks is vulnerable to data contamination and leaderboard overfitting. |
| Approach: | LLMEval-Fair framework provides a framework for dynamic evaluation of Large Language Models . evaluators use a proprietary bank of 220k graduate-level questions to analyze model data . |
| Outcome: | LLMEval-Fair provides robust and credible evaluation framework for Large Language Models . it provides a strong empirical validation for the dynamic evaluation paradigm . |
Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Speculative decoding is a novel method to expedite inference in autoregressive (large) language models. |
| Approach: | They propose to use a smaller model as a draft model to speculate a block of tokens, which the target model then evaluates for acceptance. |
| Outcome: | The proposed method can be used to accelerate inference in autoregressive (large) language models by using smaller models as draft models to speculate tokens for multiple inference steps. |
A Survey on Training-free Alignment of Large Language Models (2025.findings-emnlp)
Copied to clipboard
Birong Pan, Yongqi Li, Weiyu Zhang, Wenpeng Lu, Mayi Xu, Shen Zhou, Yuanyuan Zhu, Ming Zhong, Tieyun Qian
| Challenge: | a survey of large language models (LLMs) aims to ensure outputs adhere to human values, ethical standards, and legal norms. |
| Approach: | They present the first systematic review of TF alignment methods . they categorize them by stages of pre-decoding, in-decoder and post-decoration . |
| Outcome: | The proposed methods are based on training-free (TF) alignment techniques . they are able to be used in open-source and closed-source environments without retraining . |
Efficient Transformer Parameter Reuse via Zero-Token Mechanism (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to scaling up parameter counts are impractical for users with limited computational resources. |
| Approach: | They propose a decoupled parameter cycling strategy that employs a head-tail decoupling strategy to decouple the first (head) and last (tail) layers from the parameter cycling process. |
| Outcome: | The proposed approach achieves superior performance under strict parameter constraints and significantly reduces computational overhead via early exits. |
Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction (2021.acl-short)
Copied to clipboard
| Challenge: | Masked Noun-Phrase Prediction (MNPP) is a pre-training strategy to tackle pronoun resolution in a fully unsupervised setting. |
| Approach: | They propose a pre-training strategy to tackle pronoun resolution in an unsupervised setting by fine-tuning a large pre-trained model on a human-labeled dataset and then transferring to a smaller dataset such as Winograd Schema Challenge (WSC). |
| Outcome: | The proposed method outperforms all previous unsupervised methods on all datasets by large margins. |
ModelScope-Agent: Building Your Customizable Agent System with Open-source Large Language Models (2023.emnlp-demo)
Copied to clipboard
Chenliang Li, He Chen, Ming Yan, Weizhou Shen, Haiyang Xu, Zhikai Wu, Zhicheng Zhang, Wenmeng Zhou, Yingda Chen, Chen Cheng, Hongzhu Shi, Ji Zhang, Fei Huang, Jingren Zhou
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities to comprehend human intentions, engage in reasoning, and design planning-like behavior. |
| Approach: | They propose a framework that equips large language models with tool-use capabilities . they propose LLaMA and Chat-GLM as controllers, and a model-based agent framework . |
| Outcome: | The proposed framework equips open-source LLMs with tool-use capabilities . it provides a user-friendly system library with a customizable engine design . |
Generate & Rank: A Multi-task Framework for Math Word Problems (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies formalize MWP as a generation task but mathematical expressions are prone to minor mistakes. |
| Approach: | They propose a ranking task for math word problem (MWP) that learns from its own mistakes and distinguishes between correct and incorrect expressions. |
| Outcome: | The proposed model outperforms baselines on the classical Math23k dataset and is 7% higher than the state-of-the-art. |
InternalInspector I2: Robust Confidence Estimation in LLMs through Internal States (2024.findings-emnlp)
Copied to clipboard
Mohammad Beigi, Ying Shen, Runing Yang, Zihao Lin, Qifan Wang, Ankith Mohan, Jianfeng He, Ming Jin, Chang-Tien Lu, Lifu Huang
| Challenge: | Large Language Models (LLMs) often struggle with generating reliable outputs, often producing high-confidence inaccuracies known as hallucinations. |
| Approach: | They propose a framework that leverages contrastive learning on internal states including attention states, feed-forward states, and activation states of all layers to enhance confidence estimation in LLMs. |
| Outcome: | The framework outperforms existing methods in the hallucination detection benchmark HaluEval and the previous methods at the same time. |
BFS-Prover: Scalable Best-First Tree Search for LLM-based Automatic Theorem Proving (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to theorem proving in large language models rely on value functions and/or Monte Carlo Tree Search (MCTS), but the potential of simpler methods like Best-First Tree Search remains underexplored. |
| Approach: | They propose a scalable expert iteration framework that implements strategic data filtering at each expert iteration round, excluding problems solvable via beam search node expansion to focus on harder cases. |
| Outcome: | The proposed framework achieves a state-of-the-art score of 72.95 on the MiniF2F test set and challenges the perceived necessity of complex tree search methods. |
MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding (2026.acl-long)
Copied to clipboard
Fuwen Luo, Shengfeng Lou, Chi Chen, Ziyue Wang, Chenliang Li, Weizhou Shen, Jiyue Guo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Yang Liu
| Challenge: | Existing methods for MLLMs struggle with fine-grained temporal reasoning . despite advances in video understanding, current methods struggle with time-sensitive tasks . |
| Approach: | They propose a time-stamp-aware multi-segment grounding method that enhances temporal understanding by introducing timestamps. |
| Outcome: | The proposed method outperforms existing methods on time-sensitive tasks and generalizes well across diverse temporal understanding scenarios. |
ToW: Thoughts of Words Improve Reasoning in Large Language Models (2025.naacl-long)
Copied to clipboard
Zhikun Xu, Ming Shen, Jacob Dineen, Zhaonan Li, Xiao Ye, Shijie Lu, Aswin Rrv, Chitta Baral, Ben Zhou
| Challenge: | Unlike other data augmentation methods, thoughts of words (TOW) views next-word prediction as a core reasoning task and injects fine-grained thoughts into pre-training texts. |
| Approach: | They propose a training-time data-augmentation method called thoughts of words (TOW) that injects fine-grained thoughts directly into a next-word prediction task and teaches the model to understand how the observed next word is related to previous contexts. |
| Outcome: | The proposed method reduces model hallucination by 10% and improves reasoning performance by 7% to 9% on average. |
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation (2025.findings-emnlp)
Copied to clipboard
Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Current medical benchmarks have limitations in question design, data sources and evaluation methods. |
| Approach: | They propose a new benchmark covering five core medical areas . it includes 2,996 questions created from real-world electronic health records . |
| Outcome: | The proposed model covers five core medical areas and includes 2,996 questions created from real-world electronic health records and expert-designed clinical scenarios. |
Simple Yet Effective Synthetic Dataset Construction for Unsupervised Opinion Summarization (2023.findings-eacl)
Copied to clipboard
| Challenge: | generating aspect-specific and general opinion summaries is challenging due to the lack of annotated data. |
| Approach: | They propose two unsupervised approaches to generate aspect-specific and general opinion summaries by training on synthetic datasets constructed with aspect-related review contents. |
| Outcome: | The proposed method outperforms existing methods on space and Oposum+ and on other metrics. |
Recovering Gold from Black Sand: Multilingual Dense Passage Retrieval with Hard and False Negative Samples (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to recover and utilize negative samples have not been effective on all languages. |
| Approach: | They propose a multilingual dense passage retrieval framework to recover and utilize negative samples. |
| Outcome: | The proposed framework outperforms strong sparse, dense and hybrid baselines on a high-quality dataset covering eleven typologically diverse languages. |
Discover and Prove: An Open-source Agentic Framework for Hard Mode Automated Theorem Proving in Lean 4 (2026.acl-long)
Copied to clipboard
Chengwu Liu, Yichun Yin, Ye Yuan, Jiaxuan Xie, Botao Li, Siqi Li, Jianhao Shen, Yan Xu, Lifeng Shang, Ming Zhang
| Challenge: | Existing approaches to solving mathematical problems fall into two broad categories: informal methods and formal methods. |
| Approach: | They propose to use LLM natural-language reasoning to discover answers . they introduce Discover And Prove framework that rewrites Hard Mode statements into Easy Mode ones for existing ATP provers. |
| Outcome: | The proposed framework can be used to prove hard mode statements on ATP benchmarks. |
Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories (2025.emnlp-main)
Copied to clipboard
Mohammad Beigi, Ying Shen, Parshin Shojaee, Qifan Wang, Zichao Wang, Chandan K. Reddy, Ming Jin, Lifu Huang
| Challenge: | Existing studies show that large language models inadvertently foster sycophancy . scophancies are a tendency of models to blindly conform to user preferences without critical reasoning or self-reflection. |
| Approach: | They propose a method to reduce sycophancy by combining uncertainty-aware Monte Carlo tree search and progress-based reinforcement learning. |
| Outcome: | The proposed model outperforms baseline models in effectively reducing sycophancy while maintaining performance on out-of-distribution inputs. |
QA‐LIGN: Aligning LLMs through Constitutionally Decomposed QA (2025.findings-emnlp)
Copied to clipboard
Jacob Dineen, Aswin Rrv, Qin Liu, Zhikun Xu, Xiao Ye, Ming Shen, Zhaonan Li, Shijie Lu, Chitta Baral, Muhao Chen, Ben Zhou
| Challenge: | QA-LIGN decomposes monolithic rewards into interpretable principle-specific evaluations . scalar rewards obscure which objectives drive the training signal . |
| Approach: | a new method decomposes monolithic rewards into interpretable principle-specific evaluations . QA-LIGN reduces attack success rates by up to 68.7% while maintaining a 0.67% false refusal rate . |
| Outcome: | QA-LIGN reduces attack success rates by up to 68.7% while maintaining a 0.67% false refusal rate . the results outperform DPO and GRPO with state-of-the-art reward models given equivalent training . |
When Personalization Misleads: Understanding and Mitigating Hallucinations in Personalized LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Personalization can inadvertently distort factual reasoning when faced with factual queries. |
| Approach: | They propose a lightweight inference-time approach that mitigates personalization-induced factual distortions while preserving personalized behavior. |
| Outcome: | Experiments across multiple LLM backbones and personalization methods show that FPPS significantly improves factual accuracy while maintaining personalized performance. |
Similarity = Value? Consultation Value-Assessment and Alignment for Personalized Search (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods rely on semantic similarity to align historical consultations with current queries due to the absence of ‘value’ labels, but this lacks exploration of needs in user consultations. |
| Approach: | They propose a consultation value assessment framework that evaluates historical consultations from three novel perspectives: (1) Scenario Scope Value, (2) Posterior Action Value, and (3) Time Decay Value. |
| Outcome: | The proposed model outperforms baselines on public and commercial datasets on both retrieval and ranking tasks. |
Measuring Social Norms of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing datasets that evaluate a general understanding of social science are inadequate to understand social norms. |
| Approach: | They propose a multi-agent framework to improve large language models’ ability to understand social norms by comparing them to elementary students. |
| Outcome: | The proposed framework improves large language models to be on par with humans. |
Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training (2026.acl-long)
Copied to clipboard
Changhao Jiang, Ming Zhang, Yifei Cao, Junjie Ye, Xiaoran Fan, Shihan Dou, Zhiheng Xi, Jiajun Sun, Yi Dong, Yujiong Shen, Jingqi Tong, Baoyu Fan, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing methods to predict performance of large language models are lacking . authors propose a size-dependent mutual information predictor for closed-book question answering accuracy . |
| Approach: | They propose a size-dependent mutual information predictor that integrates knowledge frequency, knowledge specificity, and model size to forecast closed-book question answering accuracy. |
| Outcome: | The proposed method outperforms baseline models and achieves R2 > 0.7 in predicting QA accuracy without additional training. |