Papers by Kihyuk Sohn
Where is the answer? An empirical study of positional bias for parametric knowledge extraction in language model (2025.naacl-long)
Copied to clipboard
| Challenge: | Language model (LM) stores diverse factual knowledge in their parameters, which is learned during self-supervised training on unlabeled documents. |
| Approach: | They investigate the issue of "perplexity curse" in the continued training of language model (LM) they find that all studied LMs suffer from positional bias in the training document . |
| Outcome: | The proposed model is able to extract information from multiple questions with diverse queries. |
FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction (2023.acl-long)
Copied to clipboard
Chen-Yu Lee, Chun-Liang Li, Hao Zhang, Timothy Dozat, Vincent Perot, Guolong Su, Xiang Zhang, Kihyuk Sohn, Nikolay Glushnev, Renshen Wang, Joshua Ainslie, Shangbang Long, Siyang Qin, Yasuhisa Fujii, Nan Hua, Tomas Pfister
| Challenge: | Existing approaches that extend the mask language modeling to other modalities require careful multi-task tuning, complex reconstruction target designs, or additional pre-training data. |
| Approach: | They propose a centralized multimodal graph contrastive learning strategy to unify self-supervised pre-training for all modalities in one loss. |
| Outcome: | The proposed model achieves state-of-the-art performance on FUNSD, CORD, SROIE and Payment benchmarks with a more compact model size. |