Bayes Test of Precision, Recall, and F1 Measure for Comparison of Two Natural Language Processing Models (P19-1)
Copied to clipboard
| Challenge: | Existing t-tests for cross-validation (CV) are inappropriate for model comparison . existing t tests for cross validation (CV), such as 52 CV t test and F ttest, are inadequate . |
| Approach: | They propose to use a block-regularized 32 CV to compare two NLP models . they calibrate the posterior distributions of P, R, and F1 and derive an accurate interval estimation of P and R . |
| Outcome: | The proposed model could regularize the difference in certain frequency distributions over linguistic units and yield stable estimators of P, R, and F1. |
Similar Papers
Is the Best Better? Bayesian Statistical Model Comparison for Natural Language Processing (2020.emnlp-main)
Copied to clipboard
| Challenge: | a recent study raises concerns about the use of standard splits to compare models . we compare the performance of six English part-of-speech taggers to those of other models based on standard split analysis . |
| Approach: | They propose a Bayesian statistical model comparison technique using k-fold cross-validation . they rank six English part-of-speech taggers across two data sets and three evaluation metrics . |
| Outcome: | The proposed method ranks English part-of-speech taggers on two data sets and three evaluation metrics. |
Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) tend to be unreliable on fact-based answers. |
| Approach: | They propose a framework for comparing LLMs' confidence over fact-based answers with hidden-state probes that are more reliable than hidden-status probes. |
| Outcome: | The proposed methods show that hidden-state probes provide the most reliable confidence estimates despite requiring access to weights and supervision data. |
Semantic Accuracy in Natural Language Generation: A Thesis Proposal (2023.acl-srw)
Copied to clipboard
| Challenge: | Using large pre-trained language models, it is essential to research their reliability . if a human does not know the answer to a question, the socially acceptable behavior is to say 'I do not know' failing to fulfill this expectation can lead to distrust, or spread of misinformation. |
| Approach: | They propose a method for evaluating semantic accuracy and a benchmark for NLG metrics. |
| Outcome: | The proposed method evaluates semantic accuracy and provides a benchmark for NLG metrics. |
Precise Model Benchmarking with Only a Few Observations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Accurate evaluation of large language models is crucial for identifying their strengths and weaknesses. |
| Approach: | They propose an empirical Bayes estimator that balances direct and regression estimates for each subgroup separately, improving the precision of subgroup-level estimates of model performance. |
| Outcome: | The proposed model reduces the mean squared error by up to 50% on multiple datasets. |
Please, Don’t Forget the Difference and the Confidence Interval when Seeking for the State-of-the-Art Status (2022.lrec-1)
Copied to clipboard
| Challenge: | comparing NLP systems by performance has become an essential question . comparing systems by performing performance criterion is criticized for allowing chance to determine superiority . |
| Approach: | They propose to use bootstrap confidence intervals instead of state-of-the-art status and statistical significance testing to compare NLP system performance. |
| Outcome: | The bootstrap confidence intervals are used to compare NLP system performance . the bootstrap test is more accurate than state-of-the-art status and statistical significance testing . |
Towards Robust Comparisons of NLP Models: A Case Study (2025.coling-main)
Copied to clipboard
| Challenge: | Existing statistical tests to compare the test scores of different NLP models have been proposed to account for nuisance factors such as noise, randomness, or hyperparameter values. |
| Approach: | They propose a regression analysis which isolates the effect of nuisance factors from the effects of the models’ capabilities. |
| Outcome: | The proposed model is able to show that the difference between BioLinkBERT and MSR BiomedBERT is 7 times smaller than previously reported. |
Semantic Evaluation of Multilingual Data-to-Text Generation via NLI Fine-Tuning: Precision, Recall and F1 scores (2025.findings-acl)
Copied to clipboard
| Challenge: | KG-to-Text models are prone to errors like Additions and Omissions, and few languages are taken into account since both train and test data are not readily available. |
| Approach: | They propose a multilingual evaluation framework that is reference-less . it allows estimating how much a KG-to-Text Model under- (omission) or over- (addition) generates. |
| Outcome: | The proposed evaluation framework outperforms prior reference-less metrics in correlation with human judgments and provides scores for precision and recall. |
SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models (2025.naacl-industry)
Copied to clipboard
| Challenge: | Typical evaluations of Large Language Models (LLMs) report a single accuracy metric per dataset, often derived from an optimized setup. |
| Approach: | They propose a framework for non-adversarial evaluation of large language models that evaluates models by repeatedly testing them on the same benchmarks in various setups. |
| Outcome: | The proposed framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. |
Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual large-scale pretrained language models store factual knowledge, but large variations are observed across languages. |
| Approach: | They propose a ranking-based consistency metric to evaluate cross-lingual consistency of factual knowledge in multilingual PLMs. |
| Outcome: | The proposed metric evaluates cross-lingual consistency of factual knowledge across languages independently from accuracy. |
Uncertainty in Language Models: Assessment through Rank-Calibration (2024.emnlp-main)
Copied to clipboard
Xinmeng Huang, Shuo Li, Mengxin Yu, Matteo Sesia, Hamed Hassani, Insup Lee, Osbert Bastani, Edgar Dobriban
| Challenge: | Language Models (LMs) have shown promising performance in natural language generation . however, it is crucial to correctly quantify their level of uncertainty in responding to inputs. |
| Approach: | They propose a framework to quantify uncertainty and confidence for Large Language Models . they use a Rank-calibration framework to measure uncertainty and confident responses . |
| Outcome: | The proposed framework assesses uncertainty and confidence measures for LMs. |