Papers with NIAH
TAIL: A Toolkit for Automatic and Realistic Long-Context Large Language Model Evaluation (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Existing evaluation methods for long-context large language models are overly simplistic and require extensive human annotations. |
| Approach: | They propose an automatic toolkit to create realistic evaluation benchmarks . they use a document-grounded benchmark to generate question-answer pairs . |
| Outcome: | The proposed toolkit provides a way to create realistic evaluation benchmarks and visualize performance metrics of evaluated models. |
LongGenBench: Long-context Generation Benchmark (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current long-context benchmarks focus on retrieval-based tests, requiring Large Language Models to locate specific information within extensive input contexts. |
| Approach: | They propose a long-context generation benchmark that allows for flexible configurations of customized generation context lengths. |
| Outcome: | The proposed benchmark improves performance on NIAH and other retrieval-based tests. |
Bridging the Memorization-Utilization Gap: Near-Lossless Context Compression via Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in context compression have failed to effectively utilize compressed representations for downstream tasks. |
| Approach: | They propose a holistic training paradigm that uses outcome-based RL to enable implicit expansion. |
| Outcome: | The proposed model outperforms previous models on NIAH, LongBench and multi-hop reasoning. |
Fico: Evaluating Vision-Language Models under Visual Fidelity and Compression at Scale (2026.findings-acl)
Copied to clipboard
| Challenge: | Visual text compression is emerging paradigm for rendering text as images for processing by vision-language models. |
| Approach: | They propose a benchmark to assess VLM robustness under dense visual inputs. |
| Outcome: | Evaluating 13 general-purpose VLMs and 3 OCR-specialized models reveals performance drops sharply under increased density or reduced resolution; cross-task transfer between OCR, NIAH, and VQA is limited; and VQ is comparatively robust because low-level details are lost before high-level semantics. |