Are Layout-Infused Language Models Robust to Layout Distribution Shifts? A Case Study with Scientific Documents (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work has shown that infusing layout features into language models improves processing of visually-rich documents such as scientific papers. |
| Approach: | They propose a method to evaluate layout-infused language models that incorporate layout features into their models to emulate layout distribution shifts. |
| Outcome: | The proposed model performs better under layout distribution shifts than in-distribution conditions. |
Similar Papers
VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent work has improved extraction accuracy by incorporating elementary layout information, for example, each token’s 2D position on the page, into language model pretraining. |
| Approach: | They propose a method that explicitly models VIsual LAyout (VILA) groups, that is, text lines or text blocks, to further improve extraction accuracy. |
| Outcome: | The proposed methods show that inserting special tokens denoting layout group boundaries can lead to a 1.9% Macro F1 improvement in token classification. |
Distribution Shifts Are Bottlenecks: Extensive Evaluation for Grounding Language Models to Knowledge Bases (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing benchmarks fail to reflect robustness challenges and fairly evaluate models. |
| Approach: | They propose to ground language models to knowledge bases to investigate distribution shifts in language and linguistic aspects of distribution shift. |
| Outcome: | The proposed method fails to evaluate language models in large and small datasets . the proposed model fails to cope with unseen schemas and language variations . |
Text Classification Under Class Distribution Shift: A Survey (2026.eacl-long)
Copied to clipboard
| Challenge: | ML models assume that training and test data are sampled from the same distribution, but in daily practice, this assumption is often broken. |
| Approach: | They survey articles studying open-set text classification to understand the distribution shifts and mitigation approaches for each problem setup. |
| Outcome: | The proposed methods can solve problems caused by the shifting class distribution in open-set text classification and related tasks. |
Problem Solved? Information Extraction Design Space for Layout-Rich Documents using LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new open-source layout-aware IE test suite is available for download at https://github.com/gayecolakoglu/layIE-LLM. |
| Approach: | They propose an open-source layout-aware IE test suite that provides a layout-based IE pipeline. |
| Outcome: | The proposed method achieves 13.3–37.5 F1 points more than a baseline configuration using the same LLM. |
From Distributional to Overton Pluralism: Investigating Large Language Model Alignment (2025.naacl-long)
Copied to clipboard
| Challenge: | a large language model's (LLM) output distribution is changed by an alignment process . a recent study shows that aligned models surface information that cannot be recovered from base models without fine-tuning. |
| Approach: | They analyze two aspects of the alignment process that change output distributions . they find alignment suppresses irrelevant and unhelpful content . |
| Outcome: | The proposed model can be imitated without fine-tuning by using in-context examples and lower-resolution semantic hints about response content. |
DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Structured pruning reduces model size but often causes uneven degradation across domains, leading to biased performance. |
| Approach: | They propose a method that dynamically adjusts the data distribution during training to restore balanced performance across heterogeneous and multi-tasking data. |
| Outcome: | Experiments in monolingual and multilingual settings show that the proposed method surpasses similarly sized models in pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning. |
LayoutLLM: Large Language Model Instruction Tuning for Visually Rich Document Understanding (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to enhance document comprehension require fine-tuning for each task and dataset, and are expensive to train and operate. |
| Approach: | They propose a more flexible document analysis method that integrates visual-rich document understanding with large-scale language models (LLMs) by leveraging existing research in document image understanding and LLMs’ superior language understanding capabilities, the proposed model performs an understanding of document images in a single model. |
| Outcome: | The proposed model improves on the baseline model in document image understanding tasks. |
Generative Data Augmentation using LLMs improves Distributional Robustness in Question Answering (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing domain adaptation methods do not account for unseen natural distribution shifts. |
| Approach: | They perform experiments on 4 different datasets under varying amounts of distribution shift . they analyze how "in-the-wild" generation can help achieve domain generalization . |
| Outcome: | The proposed approach augments reading comprehension datasets with generated data to improve robustness towards natural distribution shifts. |
Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance (2026.acl-long)
Copied to clipboard
| Challenge: | a large-scale evaluation of deployed LLMs under natural prompt distribution shift is needed . natural prompt behavior shifts can cause performance degradation in dynamic, real-world settings . |
| Approach: | They propose a data-centric framework for measuring natural prompt distribution shift . they train models on 4.68M training prompts and evaluate on 57.6k prompts . |
| Outcome: | The proposed framework evaluates natural prompt distribution shift in LLMs over time and between user groups. |
Language Models Resist Alignment: Evidence From Data Compression (2025.acl-long)
Copied to clipboard
Jiaming Ji, Kaile Wang, Tianyi Alex Qiu, Boyuan Chen, Jiayi Zhou, Changye Li, Hantao Lou, Josef Dai, Yunhuai Liu, Yaodong Yang
| Challenge: | Large language models (LLMs) may exhibit undesirable behaviors due to the inevitable biases and harmful content present in training. |
| Approach: | They propose to investigate the elasticity of large language models by examining their performance. |
| Outcome: | The proposed model performance declines rapidly before reverting to the pre-training distribution, the authors show . the proposed model weight and code are available at pku-lm-res ist-alignment.github.io. |