Papers by Kaiyu He
Is Grokking Worthwhile? Functional Analysis and Transferability of Generalization Circuits in Transformers (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models struggle with the "curse of two-hop reasoning" in compositional tasks. |
| Approach: | They propose to form a "Generalization Circuit" during a prolonged "grokking" phase . they argue that grokkking is the process of integrating memorized atomic facts into an easy-acquire reasoning path. |
| Outcome: | The proposed model is superior to non-grokked models, but it requires a large computational cost . the study shows that grokking is not the sudden acquisition of a new reasoning paradigm . |
LMR-BENCH: Evaluating LLM Agent’s Ability on Reproducing Language Modeling Research (2025.emnlp-main)
Copied to clipboard
Shuo Yan, Ruochen Li, Ziming Luo, Zimu Wang, Daoyang Li, Liqiang Jing, Kaiyu He, Peilin Wu, Juntong Ni, George Michalopoulos, Yue Zhang, Ziyang Zhang, Mian Zhang, Zhiyu Chen, Xinya Du
| Challenge: | Large language model (LLM) agents have demonstrated remarkable potential in advancing scientific discovery, but their capability in reproducing code from research papers remains underexplored. |
| Approach: | They propose to evaluate LLM agents' ability to reproduce scientific research papers by analyzing code reproduction tasks from 23 research papers published in top-tier NLP venues. |
| Outcome: | The proposed benchmark systematically evaluates the capability of large language model (LLM) agents on code reproduction from Language Modeling Research. |
IDEA: Enhancing the Rule Learning Ability of Large Language Model Agent through Induction, Deduction, and Abduction (2025.findings-acl)
Copied to clipboard
| Challenge: | RULEARN is a benchmark to assess the rule-learning abilities of large language models (LLMs) in interactive environments. |
| Approach: | They propose a framework that integrates the process of **I**nduction, **De**duction, and **A**bduction. |
| Outcome: | The proposed framework improves on the baseline and human-like rule learning in real-world scenarios. |