Papers by Junu Kim
Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes (2024.findings-acl)
Copied to clipboard
Sunjun Kweon, Junu Kim, Jiyoun Kim, Sujeong Im, Eunbyeol Cho, Seongsu Bae, Jungwoo Oh, Gyubok Lee, Jong Hak Moon, Seng Chan You, Seungjin Baek, Chang Hoon Han, Yoon Bin Jung, Yohan Jo, Edward Choi
| Challenge: | Clinical notes are an extensive repository of information specific to individual patients. |
| Approach: | They create synthetic large-scale clinical notes using publicly available case reports extracted from biomedical literature and train a clinical large language model, Asclepius. |
| Outcome: | The proposed model outperforms several other models and is supported by detailed evaluations conducted by GPT-4 and medical professionals. |
LayerNorm Induces Recency Bias in Transformer Decoders (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that stacking causal self-attention layers alone induces a positional bias in attention scores toward earlier tokens, but this differs from the bias toward later tokens observed in Transformer decoders, known as recency bias. |
| Approach: | They propose to stack causal self-attention layers and layer norm to induce recency bias in Transformer decoders by analyzing the interaction between causal self and other architectural components. |
| Outcome: | The proposed method provides new theoretical insights into how positional information interacts with architectural components and suggests improvements in positional encoding strategies. |