Papers by Shun Kiyono
An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction (D19-1)
Copied to clipboard
| Challenge: | incorporating pseudo data in the training of grammatical error correction models has been a key factor in improving performance of such models. |
| Approach: | They investigate the choice of how pseudo data should be generated or used in a grammatical error correction model and show that the results are state-of-the-art. |
| Outcome: | The proposed method achieves state-of-the-art on the CoNLL-2014 test set and the official test set of the BEA-2019 shared task without making any modifications to the model architecture. |
Large Vocabulary Size Improves Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have investigated the properties of internal layers in large language models, but no studies have defined the vocabulary size. |
| Approach: | They propose a method to use a new vocabulary instead of the pre-defined one in a continual training scenario. |
| Outcome: | The proposed method outperforms the model with the pre-defined vocabulary in a continual training scenario. |
Pseudo Zero Pronoun Resolution Improves Zero Anaphora Resolution (2021.emnlp-main)
Copied to clipboard
| Challenge: | Masked language models have contributed to drastic performance improvements with regard to zero anaphora resolution (ZAR). |
| Approach: | They propose a pretraining task that trains MLMs on anaphoric relations with explicit supervision and a finetuning method that remedies a notorious discrepancy. |
| Outcome: | The proposed method improves zero anaphora resolution in Japanese ZAR . it uses a pretrain task and finetuning task to correct the discrepancy . |
ESPnet-ST: All-in-One Speech Translation Toolkit (2020.acl-demos)
Copied to clipboard
Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, Shinji Watanabe
| Challenge: | ESPnet-ST is a new project for the quick development of speech-to-speech translation systems. |
| Approach: | They propose a framework for rapid development of speech-to-speech translation systems . they provide all-in-one recipes including data pre-processing, feature extraction, training, and decoding pipelines . |
| Outcome: | The proposed model outperforms the current state-of-the-art models on a wide range of benchmark datasets. |
Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for incorporating a masked language model into an EncDec model have potential drawbacks when applied to GEC. |
| Approach: | They propose to incorporate a pre-trained masked language model (MLM) into an encoder-decoder model for grammatical error correction. |
| Outcome: | The proposed method achieves state-of-the-art on BEA-2019 and CoNLL-2014 benchmarks. |
Rethinking Perturbations in Encoder-Decoders for Fast Training (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing studies have proposed various regularization methods to avoid over-fitting. |
| Approach: | They propose to use scheduled sampling and adversarial perturbations to regularize neural models but they are not efficient enough for training time. |
| Outcome: | The proposed methods achieve comparable scores even though they are faster. |
B2T Connection: Serving Stability and Performance in Deep Transformers (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to prevent the vanishing gradient problem in deep neural networks are not effective. |
| Approach: | They propose a method that can equip both higher stability and effective training by a simple modification from Post-LN. |
| Outcome: | The proposed method outperforms Pre-LN and Post-Ln on a wide range of tasks. |
A Self-Refinement Strategy for Noise Reduction in Grammatical Error Correction (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches for grammatical error correction (GEC) rely on supervised learning with manually created datasets. |
| Approach: | They propose to denoise GEC datasets by leveraging prediction consistency of existing models. |
| Outcome: | The proposed method outperforms baseline methods on CoNLL-2014, JFLEG, and BEA-2019 benchmarks. |
An Empirical Study of Contextual Data Augmentation for Japanese Zero Anaphora Resolution (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to augment labeled data are limited by the scarcity of labeles . a method called contextual data augmentation (CDA) can be used to augment labels . |
| Approach: | They propose a data augmentation method that generates labeled training instances using a pretrained language model. |
| Outcome: | The proposed method can improve the quality of augmented training data compared to the conventional method. |
SHAPE: Shifted Absolute Position Embedding for Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing position representations suffer from a lack of generalization to test data with unseen lengths or high computational cost. |
| Approach: | They propose to achieve shift invariance by randomly shifting absolute positions during training by a SHAPE algorithm that is empirically comparable to its counterpart. |
| Outcome: | The proposed method outperforms existing representations on sequence-to-sequence tasks due to extrapolation, i.e., the ability to generalize to sequences that are longer than those observed during training. |
Effective Adversarial Regularization for Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Existing (small) perturbations that induce a critical prediction error in machine learning models are often referred to as adversarial examples. |
| Approach: | They propose to use adversarial perturbations to regularize text classification tasks by adding adversarials to a typical NMT model structure. |
| Outcome: | The proposed method significantly improves performance of NMT models, such as LSTM-based and Transformer-based models. |