Papers by Abishek Stephen
TokCollate: A Comprehensive Tool for Tokenizer Evaluation and Visualization across Languages (2026.acl-demo)
Copied to clipboard
| Challenge: | Tokenization quality varies significantly across languages, leading to disparities in LLM performance and cost for speakers of less-resourced languages. |
| Approach: | They propose a Python-based evaluation framework and a JavaScript visualization interface that evaluates tokenizers in a variety of languages. |
| Outcome: | TokCollate is an evaluation framework for tokenizers with a JavaScript visualization interface. |
Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing metric for subword tokenization evaluation for morphological plausibility requires unavailable or inconsistent gold segmentation data. |
| Approach: | They propose a morpho-syntactic feature-based metric for subword tokenization evaluation. |
| Outcome: | The proposed metric correlates well with traditional morpheme boundary recall while being more broadly applicable across languages with different morphological systems. |