Papers by Tatsuya Hiraoka
Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing subword regularizations use multiple segmentations during training but only use one segmentation in inference. |
| Approach: | They propose an inference strategy that uses multiple subword segmentations to solve this discrepancy in the training process and inference. |
| Outcome: | The proposed strategy reduces the cost of training and improves the performance of models trained with subword regularization in low-resource machine translation tasks. |
SubRegWeigh: Effective and Efficient Annotation Weighing with Subword Regularization (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods to reduce the adverse effect of annotation errors are time-consuming because they require many trained models to detect errors. |
| Approach: | They propose a method that uses a tokenization technique called subword regularization to simulate multiple error detection models for detecting errors. |
| Outcome: | The proposed method performs weighting weighting four to five times faster than existing methods and improves in document classification and named entity recognition tasks. |
MaxMatch-Dropout: Subword Regularization for WordPiece (2022.coling-1)
Copied to clipboard
| Challenge: | Existing subword regularization methods are specialized to a particular tokenizer type. |
| Approach: | They propose a subword regularization method for WordPiece that uses a maximum matching algorithm for tokenization. |
| Outcome: | The proposed method improves the performance of text classification and machine translation tasks as well as other subword regularization methods. |
Corpus-Dependent Subcharacter Encoding via HMM-Guided Code Assignment (2026.acl-long)
Copied to clipboard
| Challenge: | Latom is a corpus-dependent alternative to byte encoding that learns fixed-length atomic codes from text. |
| Approach: | They propose a corpus-dependent alternative to byte encoding that learns fixed-length atomic codes from text. |
| Outcome: | The proposed framework improves text classification accuracy and reduces decoding errors. |
Optimizing Word Segmentation for Downstream Task (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to optimize tokenizations for downstream tasks are not suitable for traditional NLP. |
| Approach: | They propose a method to explore a tokenization appropriate for a downstream task . they train a model to assign a high probability to such appropriate tokenization based on the downstream task loss . |
| Outcome: | The proposed method improves sentiment analysis and textual entailment tasks . it is also integrated into state-of-the-art contextualized embeddings and reports a positive effect . |
Sycophancy Hides Linearly in the Attention Heads (2026.eacl-long)
Copied to clipboard
Rifo Ahmad Genadi, Munachiso Samuel Nwadike, Nurdaulet Mukhituly, Tatsuya Hiraoka, Hilal AlQuabeh, Kentaro Inui
| Challenge: | Using TruthfulQA as the base dataset, we find that probes trained on TruthfulQ transfer effectively to other factual QA benchmarks. |
| Approach: | They train linear probes across the residual stream, multilayer perceptron, and attention layers to analyze where sycophancy signals emerge. |
| Outcome: | The proposed model can be used to steer truthfulness and toxicity behaviors. |
VLURes: Benchmarking Long-Text Grounding and Cross-Lingual Robustness in Vision Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | ***VLURes** provides a practical testbed for long-text grounding and multilingual robustness in web-realistic agent settings. |
| Approach: | They propose a multilingual benchmark for evaluating vision-language models under long-text grounding. |
| Outcome: | ***VLURes** provides a testbed for long-text grounding and multilingual robustness in web-realistic agent settings. |
Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on surface-level display of performance degradation due to typos. |
| Approach: | They propose a method to identify typo neurons and typo heads that work actively when inputs contain typos. |
| Outcome: | The proposed method identifies typo neurons and typo heads that work actively when inputs contain typos. |
Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles (2025.acl-long)
Copied to clipboard
Munachiso S Nwadike, Zangir Iklassov, Toluwani Aremu, Tatsuya Hiraoka, Benjamin Heinzerling, Velibor Bojkovic, Hilal AlQuabeh, Martin Takáč, Kentaro Inui
| Challenge: | Existing models that use sequential data can bypass the limitations of unidirectional causality. |
| Approach: | They propose a mechanism that enables large language models to bypass unidirectional causality . they propose 'cycle tokens' that enable recall of preceding tokens from succeeding ones . |
| Outcome: | The proposed model bypasses the limitations of unidirectional causality by enabling recall of preceding contexts. |
Spelling-out is not Straightforward: LLMs’ Capability of Tokenization from Token to Characters (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) can spell out tokens character by character with high accuracy, yet struggle with more complex character-level tasks. |
| Approach: | They examine how large language models internally represent character-level information during the spelling-out process. |
| Outcome: | The embedding layer does not fully encode character-level information, especially beyond the first character. |
Word-level Perturbation Considering Word Length and Compositional Subwords (2022.findings-acl)
Copied to clipboard
| Challenge: | Word replacement considering length and compositional word replacement are effective word-level perturbations. |
| Approach: | They propose two simple modifications for word-level perturbation: Word Replacement considering Length and Compositional Word Replacement. |
| Outcome: | The proposed methods improve word-level perturbation and classification performance. |
Repetition Neurons: How Do Language Models Produce Repetitions? (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing studies on text generation with LLMs focus on attention heads, framing repetition as a key mechanism for in-context learning. |
| Approach: | They introduce repetition neurons that are regarded as "skill neurons" responsible for the repetition problem in text generation tasks. |
| Outcome: | The authors identify repetition neurons as "skill neurons" that perceive repetition as a task to copy the previous context repeatedly, similar to in-context learning. |
The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing studies have focused on simple factual recall, but we have not explored how this is used in more complex queries. |
| Approach: | They propose to identify low-dimensional subspaces which encode numerical attributes associated with entities in comparison prompts. |
| Outcome: | The proposed model can answer numeric comparison questions using a low-dimensional subspace of theembedding space. |
Stochastic Tokenization with a Language Model for Neural Text Classification (P19-1)
Copied to clipboard
| Challenge: | Sentences segmented with words or subwords can be difficult to perform text classification tasks. |
| Approach: | They propose a method to learn tokenization and text classification simultaneously to address these problems. |
| Outcome: | The proposed method improves on sentiment analysis in Japanese and Chinese using tokenization and text classification models. |
Joint Optimization of Tokenization and Downstream Model (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have reported that an appropriate tokenization depends on each downstream task. |
| Approach: | They propose a method to find an appropriate tokenization to a downstream task by optimizing a tokenizer and a model. |
| Outcome: | The proposed method improves on text classification and machine translation tasks. |