Papers by Nakyung Lee
Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new framework for self-attention models is proposed to address this problem . it injects *multi-hop* relationships into the attention graph, allowing for better performance . |
| Approach: | They propose a framework that injects *multi-hop* relationships through a belief propagation process. |
| Outcome: | The proposed framework helps prevent entropy collapse in deeper layers and maintains GTD at task-appropriate levels. |
Efficient Process Reward Modeling via Contrastive Mutual Information (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to verify intermediate reasoning steps require human annotators to assign reward scores to each reasoning step, which is labor-intensive and costly. |
| Approach: | They propose a method that leverages the model's internal probability to infer step-level supervision while significantly reducing the computational burden of annotating dataset. |
| Outcome: | The proposed method reduces dataset construction time by 84% and token generation by 98% compared to MC estimation, while achieving higher accuracy on process-level evaluations and mathematical reasoning benchmarks. |