Papers with ensembling
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension (D19-58)
Copied to clipboard
| Challenge: | MRQA datasets have been used to benchmark progress in general-purpose language understanding. |
| Approach: | They propose to combine 18 question answering datasets into one shared task to evaluate their generalization capabilities. |
| Outcome: | The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than baseline based on BERT. |
A Neural Pipeline Approach for the PharmaCoNER Shared Task using Contextual Exhaustive Models (D19-57)
Copied to clipboard
| Challenge: | NER and concept indexing perform named entity recognition and concept identifiers (CUIs) in a knowledge base. |
| Approach: | They propose a neural pipeline approach that performs named entity recognition (NER) and concept indexing (CI) they use bi-LSTM to capture the semantic information of a sequence and classify them into entities or no entities . |
| Outcome: | The proposed approach performs named entity recognition (NER) and concept indexing (CI) which links them to concept unique identifiers (CUIs) in a knowledge base. |
Facebook AI’s WAT19 Myanmar-English Translation Task Submission (D19-52)
Copied to clipboard
Peng-Jen Chen, Jiajun Shen, Matthew Le, Vishrav Chaudhary, Ahmed El-Kishky, Guillaume Wenzek, Myle Ott, Marc’Aurelio Ranzato
| Challenge: | Using back-translation, we can improve generalization by using noisy channel re-ranking and ensembling. |
| Approach: | They propose to use BPE-based transformer models to leverage monolingual data to improve generalization and use noisy channel re-ranking and ensembling to improve results. |
| Outcome: | The proposed system improves on the baseline system trained exclusively on the provided small parallel dataset, and the human evaluation and BLEU score are higher. |
Massively Multilingual Transfer for NER (P19-1)
Copied to clipboard
| Challenge: | Existing approaches for cross-lingual transfer use a single source language, but there are exceptions. |
| Approach: | They propose two techniques for modulating the transfer, suitable for zero-shot or few-shot learning, respectively. |
| Outcome: | The proposed methods are much more effective than baseline models and rival oracle selection of the single best individual model. |
Semantic Role Labeling of NomBank Partitives (2025.coling-main)
Copied to clipboard
| Challenge: | Semantic role labeling (SRL) is a way to represent semantic concepts via labeled predicate/argument pairs. |
| Approach: | They describe a semantic role labeling task that uses a set of predicate/argument pairs to represent semantic concepts. |
| Outcome: | The highest scoring system achieves an F1 of 91.74% using “gold” parses from the Penn Treebank and 91.12% when using the Berkeley Neural parser. |
Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling (2023.acl-long)
Copied to clipboard
| Challenge: | ensembling BERT models often improves accuracy but at the cost of significantly more computation and memory footprint. |
| Approach: | They propose a new ensembling method for CLS-based prediction tasks that is almost as efficient as a single BERT model. |
| Outcome: | The proposed method outperforms existing BERT models on GLUE and SuperGLUE with 100 training samples. |
What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing and new approaches to zero-shot knowledge composition are lacking in NLP. |
| Approach: | They propose a framework for zero-shot module composition that unifies existing and some novel variations for selecting, weighting, and combining parameter modules under a single unified notion. |
| Outcome: | The proposed framework enables a systematic unification of concepts and enables the first comprehensive benchmarking study of various zero-shot knowledge composition strategies. |
Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities in a wide range of tasks and contexts. |
| Approach: | They propose to use a token-level ensembling method to exploit the probability information at each generation step and to avoid early incorrect tokens. |
| Outcome: | The proposed method breaks the existing community performance ceiling and improves on several benchmarks. |
Learning Morphosyntactic Analyzers from the Bible via Iterative Annotation Projection across 26 Languages (P19-1)
Copied to clipboard
| Challenge: | Currently, computational tools for low-resource languages are limited by a lack of supervised training data. |
| Approach: | They propose to use English taggers and parsers to project morphological information onto translations of the Bible in 26 different test languages. |
| Outcome: | The proposed method reduces lemmatization and morphological analysis over a strong initial system. |
Measuring and Mitigating Local Instability in Deep Neural Networks (2023.findings-acl)
Copied to clipboard
| Challenge: | Uncertain details like random initialization can change the outputs of a trained system with potentially disastrous consequences. |
| Approach: | They propose a model stability problem by studying how the predictions of a deep neural network change as a consequence of stochasticity in the training process. |
| Outcome: | The proposed method outperforms data-agnostic methods and is 90% cheaper than the gold standard. |
Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to generate multiple independent CoTs, combining them through ensembling or other post-hoc strategies, have been shown to be effective in boosting performance. |
| Approach: | They propose a method where LLMs are fine-tuned to generate a sequence of Diverse Chains of Thought (DCoT) within a single inference step. |
| Outcome: | The proposed model can generate multiple chains of thought within a single inference step without external feedback. |
Stacking with Auxiliary Features for Visual Question Answering (N18-1)
Copied to clipboard
| Challenge: | Visual Question Answering (VQA) is a challenging task that requires systems to reason about natural language and vision. |
| Approach: | They propose four categories of auxiliary features for ensembling for VQA . three out of the four categories can be inferred from an image-question pair . fourth category uses model-specific explanations . |
| Outcome: | The proposed techniques improve performance for visual question answering (VQA) given an image and a natural language question, the task is to provide an accurate natural language answer. |
Corpora Generation for Grammatical Error Correction (N19-1)
Copied to clipboard
| Challenge: | Grammatical Error Correction (GEC) is a computational task that requires large amounts of data to solve. |
| Approach: | They propose two approaches to generate large parallel datasets for GEC using publicly available Wikipedia edit histories using minimal filtration heuristics and round-trip translation through bridge languages. |
| Outcome: | The proposed methods yield similar sized parallel corpora with around 4B tokens and are far ahead of the state-of-the-art on the CoNLL ‘14 benchmark and the JFLEG task. |
Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on using synthetic data for grammatical error correction . lack of annotated training data hinders progress in the field . |
| Approach: | They propose auxiliary tasks that exploit alignment between original and corrected sentences . they propose a sequence-to-sequence problem and perform multi-task training . |
| Outcome: | The proposed auxiliary tasks outperform the best models with a BART-based model on 11B parameters. |
Neural Transduction for Multilingual Lexical Translation (2020.coling-main)
Copied to clipboard
| Challenge: | a method for completing multilingual translation dictionaries is proposed . a 27% relative improvement in whole-word accuracy is achieved when multilingual data is unavailable . |
| Approach: | They propose a method for completing multilingual translation dictionaries using multilingual inputs and multilingual decoding objective. |
| Outcome: | The proposed method can synthesize new word forms in multilingual translation dictionaries . it can perform in settings where correct translations have not been observed in text . |
Detecting Urgency Status of Crisis Tweets: A Transfer Learning Approach for Low Resource Languages (2020.coling-main)
Copied to clipboard
| Challenge: | We train monolingual and cross-lingual classifiers on the extracted features of tweets . we use a few state-of-the-art contextual embeddings to extract features of the tweets. |
| Approach: | They propose to use tweets to train a dataset of English and two low-resource languages to train zero-shot transfer models. |
| Outcome: | The proposed model performs well in English and in low-resource languages . the proposed model is based on state-of-the-art embeddings and semi-supervised methods . |
ISAAQ - Mastering Textbook Questions with Pre-trained Transformers and Bottom-Up and Top-Down Attention (2020.emnlp-main)
Copied to clipboard
| Challenge: | Textbook Question Answering is a complex task that requires reasoning with multimodal information from text and diagrams. |
| Approach: | They propose to use transformer language models and bottom-up and top-down attention to tackle the language and visual understanding challenges of text and diagrams. |
| Outcome: | The proposed system achieves unprecedented accuracies on all TQA question types . the system also obtains state-of-the-art results in other demanding datasets . |
Fighting Randomness with Randomness: Mitigating Optimisation Instability of Fine-Tuning using Delayed Ensemble and Noisy Interpolation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Despite the success of fine-tuning, it still displays model performance instability, especially with limited data. |
| Approach: | They propose a new mitigation strategy that leverages the strengths of ensembling, noise regularisation and model interpolation while retaining computational efficiency. |
| Outcome: | The proposed mitigation strategy outperforms the best performing mitigation strategy (Ensemble) while using only a fraction of its cost. |
Fisher Mask Nodes for Language Model Merging (2024.lrec-main)
Copied to clipboard
| Challenge: | Pre-trained models are ubiquitous in natural language processing, but individual fine-tuned models require significant overhead in multi-task scenarios. |
| Approach: | They propose a method for fine-tuning pre-trained models for Transformers using Fisher information. |
| Outcome: | The proposed method outperforms Fisher-weighted averaging in a fraction of the computational cost. |
Two-Stage Fine-Tuning for Improved Bias and Variance for Large Pretrained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work challenges the bias-variance trade-off . large pretrained models can have large variance and overfit domain-specific data . |
| Approach: | They propose a bias-variance trade-off that implies learning methods need to balance complexity with data size to minimize under-fitting and over-fit. |
| Outcome: | The proposed method achieves strong results on SuperGLUE and clinical information extraction tasks. |
Ensembling Prompting Strategies for Zero-Shot Hierarchical Text Classification with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Hierarchical text classification is a challenging task in natural language processing. |
| Approach: | They propose a method which integrates the results of diverse prompting strategies to promote LLMs’ reliability. |
| Outcome: | The proposed method boosts the performance of single prompting strategies and achieves SOTA results on three benchmark datasets. |
The Devil Is in the Word Alignment Details: On Translation-Based Cross-Lingual Transfer for Token Classification Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Translation-based strategies for cross-lingual transfer XLT include label projection . word aligners (WAs) are commonly used for label projection, but low-level design decisions for using them have not been investigated . |
| Approach: | They revisit word aligners (WAs) for label projection and propose a new projection strategy that outperforms WAs. |
| Outcome: | The proposed projection strategy outperforms marker-based methods in token classification tasks. |
AdaFuse: Adaptive Ensemble Decoding for Large Language Models (2026.acl-long)
Copied to clipboard
Chengming Cui, Tianxin Wei, Ziyi Chen, Ruizhong Qiu, Zhichen Zeng, Zhining Liu, Xuying Ning, Duo Zhou, Jingrui He
| Challenge: | Existing ensemble approaches to large language models lack flexibility for mid-generation adaptation. |
| Approach: | They propose an adaptive ensemble decoding framework that dynamically selects semantically appropriate fusion units during generation. |
| Outcome: | The proposed framework outperforms existing ensemble frameworks on open-domain QA, arithmetic reasoning, and machine translation tasks. |