Papers by Zheng Lingchao
LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model (2026.acl-long)
Copied to clipboard
| Challenge: | Existing parallel tokenization methods suffer from inconsistent results due to boundary artifacts that occur after merging. |
| Approach: | They propose a Lossless Parallel Tokenization framework that ensures output identical to standard sequential tokenization. |
| Outcome: | The proposed method achieves significant speedup while guaranteeing lossless tokenization. |