Papers by Gyuho Shim
Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models are often judged by their scores on standard benchmarks, yet such scores often overstate real capability since they mask the mix of skills a task actually demands. |
| Approach: | They propose a diagnostic framework that decomposes benchmark performance into ten cognitively grounded abilities and computes an Ability Impact Score (AIS) AIS quantifies how much each ability contributes to a model’s success on a given benchmark. |
| Outcome: | The proposed framework decomposes performance into ten cognitively grounded abilities and computes an Ability Impact Score (AIS) that quantifies how much each ability contributes to a model’s success on a given benchmark. |
REVISE: A Framework for Revising OCRed text in Practical Information Systems with Data Contamination Strategy (2025.acl-industry)
Copied to clipboard
| Challenge: | Existing Document AI frameworks lack the capability to structurally organize and manage document information. |
| Approach: | They propose a framework that corrects OCR errors at the character, word, and structural levels and a synthetic data generation strategy that realistically simulates such errors to train an effective correction model. |
| Outcome: | The proposed framework improves document retrieval and question answering tasks by correcting errors introduced by OCR errors at the character, word, and structural levels. |
HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to document-based Opendomain Question Answering (ODQA) use flat text chunks or page-level images to locate the correct document. |
| Approach: | They propose a hierarchical tree-based multimodal retrieval framework that elevates document hierarchy to a first-class retrieval signal. |
| Outcome: | The proposed framework outperforms page- and chunk-based baselines on ODQA benchmarks and improves retrieval recall by 12.9% and end-to-end QA performance by 6.8%. |