Papers with BIG-bench
Confidence and Stability of Global and Pairwise Scores in NLP Evaluation (2025.acl-srw)
Copied to clipboard
| Challenge: | Modern natural language processing benchmarks are often represented as pairwise comparison leaderboards, such as LMSYS Arena. |
| Approach: | They investigate the strengths and weaknesses of global scores and pairwise comparisons to aid decision-making in selecting appropriate model evaluation strategies. |
| Outcome: | The proposed method underestimates strong models with rare errors or low confidence, while relying on global scores can be more effective. |
Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant progress in utilizing tools, but their closed-source nature and high inference costs pose limitations on their adaptability. |
| Approach: | They propose a framework that performs task-solving by first creating a toolkit and then integrating the planning and calling of tools through a chain-of-solve approach. |
| Outcome: | The proposed model performs task-solving by harnessing Toolink's creativity and CoS ability on ChatGPT and finetunes the LLaMA-7B model. |
ThinkSum: Probabilistic reasoning over sets using large language models (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have a substantial capacity for high-level analogical reasoning, but they fail in scenarios that require reasoning over multiple objects or facts and making sequences of logical deductions. |
| Approach: | They propose a two-stage probabilistic inference paradigm, ThinkSum, which reasons over sets of objects or facts in a structured manner. |
| Outcome: | The proposed paradigm improves on the BIG-bench suite of evaluation tasks. |
Model Performance-Guided Evaluation Data Selection for Effective Prompt Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing prompt engineering methods rely on randomly selected evaluation subsets, leading to suboptimal prompts. |
| Approach: | They propose an iterative evaluation data selection approach for effective prompt optimization using real time model performance. |
| Outcome: | The proposed approach improves effectiveness by 1.6% to 3.1% and stability by 50% to 55.5% on two datasets BIG-bench and LIAR and two models GPT-3.5 and GPT-4o-mini. |
FinGPT: Large Generative Models for a Small Language (2023.emnlp-main)
Copied to clipboard
Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, Sampo Pyysalo
| Challenge: | Neural language models excel in many tasks in NLP but are limited to smaller languages. |
| Approach: | They propose two approaches to pretrain large language models for Finnish . they train seven monolingual models from scratch and use Finnish as pretraining data . |
| Outcome: | The proposed model is based on a dataset of Finnish web crawls, news, social media and eBooks. |
How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that large language models can be used to predict performance on new configurations. |
| Approach: | They investigate the predictability of large language model capabilities by using BIG-bench . they find a subset of BIG-Bench tasks as informative as BIG-bnch Hard . |
| Outcome: | The proposed model achieves an R2 score greater than 95% on BIG-bench . the model is 3 smaller than BIG-Bench Hard, and the model performs better on the full set. |
Efficiently Enhancing Zero-Shot Performance of Instruction Following Model via Retrieval of Soft Prompt (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that adding a instruction tuning stage to training large language models can improve zero-shot task generalization. |
| Approach: | They propose a method that retrieves promptspecific source prompt embeddings from training instances . they train soft prompt embeds for each prompt through prompt tuning and store the samples . |
| Outcome: | The proposed method outperforms hard prompts on unseen tasks by 2.39% points and outperformed 10 out of 11 datasets. |