Papers by Ryosuke Takahashi
Leveraging Three Types of Embeddings from Masked Language Models in Idiom Token Classification (2022.starsem-1)
Copied to clipboard
| Challenge: | Recent research shows that contextualized word embeddings can give promising results for idiom token classification. |
| Approach: | They propose to leverage contextualized word embeddings from masked language models to improve idiom token classification. |
| Outcome: | The proposed method improves idiom token classification for English and Japanese datasets. |
Suppressing Final Layer Hidden State Jumps in Transformer Pretraining (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing models exhibit only slight changes in the angular distance between the input and output hidden state vectors in the middle layers . |
| Approach: | They propose a jump-suppressing regularizer which penalizes large hidden state displacements near the final layer during pre-training. |
| Outcome: | The proposed method significantly reduces hidden state jumps in the final layer and increases model capacity. |