“What is the value of templates?” Rethinking Document Information Extraction Datasets for LLMs (2024.findings-emnlp)
Copied to clipboard
Ran Zmigrod, Pranav Shetty, Mathieu Sibue, Zhiqiang Ma, Armineh Nourbakhsh, Xiaomo Liu, Manuela Veloso
| Challenge: | Existing work on prompt-response datasets for visually rich document understanding (VRDU) is labor-intensive. |
| Approach: | They propose a set of questions that are transformed from a key information extraction template to a prompt-response format using a plethora of bespoke templates. |
| Outcome: | The proposed datasets are compared with baseline models on K2Q with zero-shot prompting. |
Similar Papers
PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are useful for low-resource scenarios and time-restricted applications. |
| Approach: | They propose a large-scale evaluation tool for large language models that uses prompts . they evaluate 720 prompt templates for open-source LLM-based metrics on MT and summarization datasets a 6.6M evaluations. |
| Outcome: | The proposed model evaluates 720 prompt templates on machine translation and summarization datasets. |
Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segments (2025.acl-long)
Copied to clipboard
| Challenge: | Specialized non-LLM NLP-based solutions lack reasoning and are not able to infer values not explicitly present in documents. |
| Approach: | They propose a novel LLM-based approach that organizes VRDs into localized semantic textual segments called semantic blocks. |
| Outcome: | The proposed approach outperforms the state-of-the-art on public VRD benchmarks by 1-3% in F1 scores and is resilient to document formats previously not encountered. |
Prompt2Model: Generating Deployable Models from Natural Language Instructions (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a step backward from traditional special-purpose NLP models . they require extensive computational resources for deployment and can be gated behind APIs . |
| Approach: | They propose a general-purpose method that takes a natural language task description and uses it to train a special-purpose model. |
| Outcome: | The proposed method outperforms a strong LLM by 20% while being 700 times smaller. |
The Death and Life of Great Prompts: Analyzing the Evolution of LLM Prompts from the Structural Perspective (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has shown that high-quality prompts are essential for LLMs to produce accurate and relevant responses. |
| Approach: | They analyze 10,538 in-the-wild prompts collected from various platforms and develop a framework that decomposes the prompts into eight key components. |
| Outcome: | The proposed framework decomposes 10,538 in-the-wild prompts into eight components. |
A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends (2026.findings-acl)
Copied to clipboard
Yihao Ding, Siwen Luo, Yue Dai, Yanbei Jiang, Zechuan Li, Qiang Sun, Geoffrey Martin, Wei Liu, Yifan Peng
| Challenge: | Visually Rich Document Understanding (VRDU) frameworks are a key area of research . early approaches to VRDU relied on manually crafted rules and domain-specific heuristics . conventional deep learning approaches do not integrate the diverse modalities in documents . |
| Approach: | They review recent advances in MLLM-based Visually Rich Document Understanding (VRDU) their findings highlight emerging trends and promising research directions . |
| Outcome: | The proposed frameworks are scalable, reliable, and adaptable, the authors argue . their findings highlight emerging trends and promising research directions . |
PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) excel in zero-shot document ranking tasks. |
| Approach: | They propose a prompt-based re-ranking method that requires no further training but is only feasible for reranking a handful of candidates due to computational costs. |
| Outcome: | The proposed method can retrieve documents from the entire corpus without training and with a large amount of paired text data. |
Retrieval-Augmented Modular Prompt Tuning for Low-Resource Data-to-Text Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Data-to-text generation methods are often limited by data sparsity and lack of training data. |
| Approach: | They propose a retrieval-augmented modular prompt tuning method that generates texts with few hallucinations from structured data inputs. |
| Outcome: | The proposed method generates texts with few hallucinations and achieves state-of-the-art performance on a dataset for drone handover message generation. |
Empirical Study of Zero-shot Keyphrase Extraction with Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | a prompting-based approach can effectively supersede traditional KE methods, a study shows . our code is available at https://github.com/kangnlp/zero-shot-keyphrase-extraction-with-LLMs. |
| Approach: | They propose four prompting strategies for zero-shot keyphrase extraction using Large Language Models. |
| Outcome: | The proposed prompting strategies outperform state-of-the-art prompting methods on KE benchmark datasets. |
Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation (2023.findings-acl)
Copied to clipboard
Xingdi Yuan, Tong Wang, Yen-Hsiang Wang, Emery Fine, Rania Abdelghani, Hélène Sauzéon, Pierre-Yves Oudeyer
| Challenge: | Large Language Models (LLMs) have demonstrated impressive prowess in natural language generation. |
| Approach: | They propose a method to select high-quality questions from LLM-generated candidates using round-trip and prompt-based scoring. |
| Outcome: | The proposed approach can select high-quality questions from a set of LLM-generated candidates without modification of the underlying model nor rely on human annotations. |
UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents (2026.acl-long)
Copied to clipboard
Yifan Ji, Zhipeng Xu, Zhenghao Liu, Zulong Chen, Qian Zhang, Zhibo Yang, Junyang Lin, Yu Gu, Ge Yu, Maosong Sun
| Challenge: | Recent Large Multimodal Models (LMMs) have shown promising potential for performing end-to-end KIE directly from document images. |
| Approach: | They propose a benchmark to evaluate the performance of Large Multimodal Models (LMMs) using a constrained-category KIE track and an open-categorical KIE Track. |
| Outcome: | Experiments on 15 state-of-the-art LMMs show performance degradation under diverse schema definitions, long-tail key fields, and complex layouts, along with pronounced performance disparities across different document types and scenarios. |