Lost in Formatting: How Output Formats Skew LLM Performance on Information Extraction (2026.eacl-long)
Copied to clipboard
| Challenge: | Information extraction systems, powered by Large Language Models (LLMs), are increasingly deployed in high-stakes domains such as biomedicine. |
| Approach: | They propose to use output formatting as a critical yet largely overlooked hyperparameter in information extraction tasks. |
| Outcome: | The output formatting is a critical but largely overlooked hyperparameter in large language models on information extraction tasks. |
Similar Papers
LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs (2025.naacl-long)
Copied to clipboard
Do Xuan Long, Ngoc-Hai Nguyen, Tiviatis Sim, Hieu Dao, Shafiq Joty, Kenji Kawaguchi, Nancy F. Chen, Min-Yen Kan
| Challenge: | Using format-following capabilities, state-of-the-art large language models (LLMs) can be leveraged to tailor outputs to specific task formats. |
| Approach: | They propose to define a format bias evaluation metric and establish effective strategies to reduce it. |
| Outcome: | The proposed evaluation reduces the variance in ChatGPT’s performance among wrapping formats from 235.33 to 0.71 (%2) |
Quantifying the Impact of Structured Output Format on Large Language Models through Causal Inference (2026.findings-eacl)
Copied to clipboard
| Challenge: | Prior studies have examined the impact of structured output on LLMs’ generation quality, often presenting one-way findings. |
| Approach: | They propose to derive five potential causal structures characterizing the influence of structured output on LLMs’ generation using one assumed and two guaranteed constraints. |
| Outcome: | The proposed pipeline can be extended to other modules and is not limited to structured output but can be used in industrial applications. |
A Thorough Examination of Decoding Methods in the Era of LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Decoding methods are essential for converting language models from next-token predictors into practical task solvers. |
| Approach: | They propose to evaluate decoding methods in general-purpose large language models . they find that decoding method performance is notably task-dependent . |
| Outcome: | The proposed methods perform task-dependently and are influenced by alignment, model size, and quantization. |
Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generation (2025.coling-main)
Copied to clipboard
| Challenge: | Generative large language models generate a high-dimensional probability distribution over all tokens in their vocabulary. |
| Approach: | They conduct extensive sensitivity analyses to determine how hyperparameter choices shape the outputs of generative large language models. |
| Outcome: | The proposed methods influence the distribution of diversity and coherence metrics in human-written text, but the optimal configurations vary across models and tasks. |
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Large Language Model Performance. (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Structured generation is used to extract key output information from large language models (LLMs). |
| Approach: | They examine whether constraints on generation space impact LLMs’ abilities, including reasoning and domain knowledge comprehension. |
| Outcome: | The proposed model is based on a few-shot in-context learning and instruction-following capabilities. |
Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that Large Language Models (LLMs) can be used as translation evaluators. |
| Approach: | They propose to use both coarse-grained and fine-grounded prompts to discern the utility of source versus reference data in machine translation evaluation tasks. |
| Outcome: | The proposed model can be used to evaluate translations in multiple languages. |
Butterfly Effects in Frame Semantic Parsing: impact of data processing on model ranking (C18-1)
Copied to clipboard
| Challenge: | a common contribution to computational linguistics research is a new model for a specific task. |
| Approach: | They propose an open-source standardized processing pipeline for frame semantic parsing . they propose a standard evaluation resource that can be shared and reused for robust comparison . |
| Outcome: | The proposed model can be shared and reused for robust model comparison. |
Demystifying Instruction Mixing for Fine-tuning Large Language Models (2024.acl-srw)
Copied to clipboard
| Challenge: | Instruction tuning is effective for aligning large language models with human instructions, but the procedure to optimizing the mixing of instruction datasets is still unclear. |
| Approach: | They categorize instructions into three primary types: NLP downstream tasks, coding, and general chat. |
| Outcome: | The proposed method improves performance of large language models (LLMs) but it is difficult to combine different instruction datasets to optimize overall performance. |
From Distributional to Overton Pluralism: Investigating Large Language Model Alignment (2025.naacl-long)
Copied to clipboard
| Challenge: | a large language model's (LLM) output distribution is changed by an alignment process . a recent study shows that aligned models surface information that cannot be recovered from base models without fine-tuning. |
| Approach: | They analyze two aspects of the alignment process that change output distributions . they find alignment suppresses irrelevant and unhelpful content . |
| Outcome: | The proposed model can be imitated without fine-tuning by using in-context examples and lower-resolution semantic hints about response content. |
Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies focus on data selection but lack a clear, unified framework . variability in experimental settings complicates systematic comparisons . |
| Approach: | They propose a three-stage scheme to standardize data selection for fine-tuning large language models . they propose unified comparison approach that incorporates ratio-based efficiency and ranking-based feasibility metrics to address inconsistencies across experiments. |
| Outcome: | The proposed scheme outperforms existing methods in a dozen key studies and identifies key challenges. |