Papers by Yinxi Li

Copied to clipboard

Challenge:	Large language models (LLMs) for code rely on subword tokenizers learned from mixed natural language text and programming language code but driven by statistics rather than grammar.
Approach:	They propose a framework that applies semantic-preserving rewrite rules to create code variants differing only in tokenization.
Outcome:	The proposed framework can create code variants differing only in tokenization . the findings highlight the need for grammar-aware tokenization for future code LLMs.