| Challenge: | Natural language processing (NLP) is a vast field, with a wide variety of tasks, languages, and domains. |
| Approach: | They build regression models to predict evaluation score of an NLP experiment . they find that their models can produce meaningful predictions over unseen languages . |
| Outcome: | The proposed model outperforms baseline models and human experts on 9 different tasks. |
Similar Papers
Towards More Fine-grained and Reliable NLP Performance Prediction (2021.eacl-main)
Copied to clipboard
| Challenge: | Performance prediction is a task of estimating a system’s performance without performing experiments. |
| Approach: | They propose to understand reliability of performance prediction models from two angles: confidence intervals and calibration. |
| Outcome: | The proposed methods demonstrate the feasibility of fine-grained performance prediction and the necessity to perform reliability analysis for performance prediction methods in the future. |
On the Evaluation of Neural Selective Prediction Methods for Natural Language Processing (2023.acl-long)
Copied to clipboard
| Challenge: | Existing techniques for selective classification are lacking in the literature. |
| Approach: | They propose a methodological blueprint and a metric for calibrating confidence functions for selective prediction. |
| Outcome: | The proposed method improves on the GLUE benchmark and the proposed refinement metric provides a calibrated evaluation of confidence functions for selective prediction. |
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
Performance Prediction via Bayesian Matrix Factorisation for Multilingual Natural Language Processing Tasks (2023.eacl-main)
Copied to clipboard
| Challenge: | Performance prediction for natural language processing (NLP) is based on a framework of Bayesian matrix factorisation . it avoids hyperparameter tuning and provides uncertainty estimates over predictions. |
| Approach: | They propose to use Bayesian matrix factorisation to predict the performance of language pairs depicted by grey cells. |
| Outcome: | The proposed framework outperforms the state-of-the-art in several NLP benchmarks, including machine translation and cross-lingual entity linking. |
Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview (2020.acl-main)
Copied to clipboard
| Challenge: | a growing number of studies address the effect of bias on predictions, but no unifying framework exists . a general phenomenon of biased predictive models in NLP is not recent, authors say . |
| Approach: | They propose a unifying framework for identifying and reducing bias in natural language processing . they propose to differentiate two consequences of bias and four potential origins of bias . |
| Outcome: | The proposed framework provides an overview of predictive bias in natural language processing . it differentiates two consequences of bias and four potential origins of bias: label bias, selection bias, model overamplification, and semantic bias. |
Efficient Methods for Natural Language Processing: A Survey (2023.tacl-1)
Copied to clipboard
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
| Challenge: | Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data, but using only scale to improve performance means resource consumption also grows. |
| Approach: | They propose to use data, time, storage, or energy to improve model performance. |
| Outcome: | The proposed methods and findings provide guidance for conducting NLP under limited resources and point towards promising research directions for developing more efficient methods. |
Semantic Accuracy in Natural Language Generation: A Thesis Proposal (2023.acl-srw)
Copied to clipboard
| Challenge: | Using large pre-trained language models, it is essential to research their reliability . if a human does not know the answer to a question, the socially acceptable behavior is to say 'I do not know' failing to fulfill this expectation can lead to distrust, or spread of misinformation. |
| Approach: | They propose a method for evaluating semantic accuracy and a benchmark for NLG metrics. |
| Outcome: | The proposed method evaluates semantic accuracy and provides a benchmark for NLG metrics. |
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)
Copied to clipboard
| Challenge: | introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. |
| Approach: | They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them. |
| Outcome: | The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods. |
Predicting Language Models’ Success at Zero-Shot Probabilistic Prediction (2025.findings-emnlp)
Copied to clipboard
Kevin Ren, Santiago Cortes-Gomez, Carlos Miguel Patiño, Ananya Joshi, Ruiqi Lyu, Jingjing Tang, Alistair Turcan, Khurram Yamin, Steven Wu, Bryan Wilder
| Challenge: | Recent work has investigated the capabilities of large language models (LLMs) as zero-shot models for generating individual-level characteristics. |
| Approach: | They conduct a large-scale empirical study of large language models’ zero-shot predictive capabilities across a wide range of tabular prediction tasks. |
| Outcome: | The results show that LLMs perform well on the base prediction task, and when they perform well, they are more likely to provide high-quality predictions. |
Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | In the recent past, a popular way of evaluating natural language understanding was to consider a model’s ability to perform natural language inference (NLI) tasks. |
| Approach: | They focus on five different NLI benchmarks across six models of different scales and examine how their accuracies develop during training. |
| Outcome: | The softmax distributions of models align with human label distributions in cases where statements are ambiguous or vague. |