Papers by Masahiro Kaneko

Disentangling the Effects of Unlearning in Measuring Parametric Faithfulness of Chain-of-Thought (2026.acl-srw)

Challenge:	Existing studies present tokens, examples, and hints for corrections, but do not directly explain the reasons in natural language.
Approach:	They propose a method called controlled generation with Prompt Insertion that uses Large Language Models to explain the reasons for corrections in natural language.
Outcome:	The proposed method can explain the reasons for corrections in natural language by guiding the LLMs to generate explanations for all correction points.

Copied to clipboard

Ryo Mitsuhashi, Gaku Morio, Ayana Niwa, Masahiro Kaneko, Kentaro Inui, Terufumi Morishita, Yuta Koreeda, Yasuhiro Sogawa

Reasoning

Challenge:	Chain-of-Thought (CoT) has been debated as a model's faithfulness to internal reasoning process.
Approach:	They propose to use unlearning to measure parametric faithfulness of models by adjusting for unintended artifacts of unlearning.
Outcome:	The proposed metric accounts for the unintended artifacts of unlearning and shows that it is non-negligible.

From Heard to Lived Opinions: Simulating Opinion Dynamics with Grounded LLM Agents in Economic Environments (2026.findings-acl)

Copied to clipboard

Ryuji Hashimoto, Masahiro Kaneko, Ryosuke Takata, Takehiro Takayanagi, Kiyoshi Izumi

Sentiment Analysis

Challenge:	Existing studies on opinion dynamics (OD) focus primarily on opinion exchange, with opinion change driven by linguistic interaction.
Approach:	They propose a OD simulation framework that grounds LLM-based agents in an economic environment and allows them to act and receive environmental feedback.
Outcome:	The proposed framework shows that LLM-based agents can act and receive environmental feedback at both individual and population levels while generating larger distributional shifts.

Dictionary-based Debiasing of Pre-trained Word Embeddings (2021.eacl-main)

Copied to clipboard

A Multilingual Social Bias Benchmark Incorporating Thinking Processes (2026.acl-long)

Challenge:	Existing methods for learning word embeddings using dictionaries do not require access to training resources or knowledge regarding the word embeds used.
Approach:	They propose a method for debiasing pre-trained word embeddings using dictionaries . they learn constraints that must be satisfied by unbiased word embeds from dictionary definitions .
Outcome:	The proposed method removes unfair biases encoded in pre-trained word embeddings while preserving useful semantics.

Copied to clipboard

Masahiro Kaneko, Danushka Bollegala, Timothy Baldwin

Reasoning

Challenge:	Large Language Models (LLMs) can learn useful knowledge and harmful stereotypes, making bias evaluation essential.
Approach:	They propose a multilingual social bias benchmark that incorporates human-generated reasoning as part of the thinking process.
Outcome:	The proposed method demonstrates superior performance over LLM-generated methods . human-generated thinking yields higher-quality evaluations than template-based approaches .

Social Bias Evaluation for Large Language Models Requires Prompt Variations (2025.findings-emnlp)

Copied to clipboard

Rem Hida, Masahiro Kaneko, Naoaki Okazaki

In-Contextual Gender Bias Suppression for Large Language Models (2024.findings-eacl)

Challenge:	Recent studies have tried to evaluate and mitigate social biases accurately using limited prompts.
Approach:	They investigate the sensitivity of Large Language Models when changing prompt variations . they found that LLM rankings fluctuate across prompts for both task performance and social bias .
Outcome:	The results show that LLM rankings fluctuate when changing prompt variations .

Copied to clipboard

Daisuke Oba, Masahiro Kaneko, Danushka Bollegala

IMPARA: Impact-Based Metric for GEC Using Parallel Data (2022.coling-1)

Challenge:	Prior work has proposed debiasing methods that require human labelled examples, data augmentation and fine-tuning of LLMs, which are computationally expensive.
Approach:	They propose to suppress gender biases by providing textual preambles from manually designed templates and real-world statistics without accessing model parameters.
Outcome:	The proposed methods suppress gender biases in English LLMs using a CrowsPairs dataset without accessing model parameters.

Copied to clipboard

Koki Maeda, Masahiro Kaneko, Naoaki Okazaki

Sampling-based Pseudo-Likelihood for Membership Inference Attacks (2025.findings-acl)

Challenge:	Existing methods for automatic evaluation of grammatical error correction require multiple reference sentences or manual scores.
Approach:	They propose an Impact-based Metric for GEC using PARAllel data, IMPARA . IMPRA computes correction impacts computed by parallel data comprising pairs of grammatical/ungrammatically-spaced sentences.
Outcome:	The proposed method can perform evaluations that fit different domains and correction styles.

Copied to clipboard

Masahiro Kaneko, Youmi Ma, Yuki Wata, Naoaki Okazaki

Robustness in NLP

Challenge:	Large Language Models (LLMs) are trained on large-scale web data, which makes it difficult to grasp the contribution of each text.
Approach:	They propose a membership-inference attack method that uses only the input text to detect leaks.
Outcome:	The proposed method performs on par with existing likelihood-based methods even without likelihoods.

Controlling Grammatical Error Correction Using Word Edit Rate (P19-2)

Copied to clipboard

Kengo Hotate, Masahiro Kaneko, Satoru Katsumata, Mamoru Komachi

Gender Bias in Masked Language Models for Multiple Languages (2022.naacl-main)

Challenge:	Existing models for grammatical error correction only consider the single degree of correction suited for training corpus.
Approach:	They propose a neural grammar error correction method that can control the degree of correction by using new training data annotated with word edit rate.
Outcome:	The proposed method improves correction accuracy by using training data annotated with word edit rate.

Copied to clipboard

Masahiro Kaneko, Aizhan Imankulova, Danushka Bollegala, Naoaki Okazaki

Likelihood-based Mitigation of Evaluation Bias in Large Language Models (2024.findings-acl)

Challenge:	Masked Language Models (MLMs) pre-trained by predicting masked tokens on large corpora have been used successfully in natural language processing tasks for a variety of languages.
Approach:	They propose to use English attribute word lists to evaluate bias in eight languages without manually annotating data.
Outcome:	The proposed model significantly correlates with the existing English datasets for gender bias.

Copied to clipboard

Masanari Ohi, Masahiro Kaneko, Ryuto Koike, Mengsay Loem, Naoaki Okazaki

Investigating How Pre-training Data Leakage Affects Models’ Reproduction and Detection Capabilities (2025.emnlp-main)

Challenge:	Large Language Models (LLMs) are widely used to evaluate natural language generation tasks as automated metrics.
Approach:	They propose to use LLMs to evaluate sentences with higher likelihoods and lower likelihoods to mitigate the likelihood bias.
Outcome:	The proposed method overrates sentences with higher likelihoods while underrating sentences with lower likelihoods.

Copied to clipboard

Masahiro Kaneko, Timothy Baldwin

Comparison of Grammatical Error Correction Using Back-Translation Models (2021.naacl-srw)

Challenge:	Existing studies do not examine how leaked instances in training datasets influence LLMs’ output and detection capabilities.
Approach:	They conduct an experimental survey to examine the relationship between data leakage in training datasets and its effects on the generation and detection by Large Language Models (LLMs).
Outcome:	The results show that enhancing leakage detection through few-shot learning can help mitigate the impact of the leakage rate in the training data on detection performance.

Copied to clipboard

Aomi Koyama, Kengo Hotate, Masahiro Kaneko, Mamoru Komachi

Machine Translation

Challenge:	Currently, a mainstream approach to generate pseudo data is back-translation (BT).
Approach:	They propose to use back-translation to generate pseudo data that contains grammatical and ungrammatically produced sentences.
Outcome:	The proposed methods improve or interpolate the performance of each error type compared with a single BT model with different seeds.

Cross-Corpora Evaluation and Analysis of Grammatical Error Correction Models — Is Single-Corpus Evaluation Enough? (N19-1)

Copied to clipboard

Masato Mita, Tomoya Mizumoto, Masahiro Kaneko, Ryo Nagata, Kentaro Inui

Is Human-Like Text Liked by Humans? Multilingual Human Detection and Preference Against AI (2026.acl-long)

Challenge:	Existing studies have evaluated grammatical error correction models on a single corpus, but the evaluation is incomplete because the task difficulty varies depending on the corpus and conditions such as proficiency levels of the writers and essay topics.
Approach:	They evaluate the performance of several GEC models against various learner corpora and compare their rankings against the corpus.
Outcome:	The evaluation of several models against learner corpora shows that the models’ rankings vary depending on the corpus, indicating that single-corpus evaluation is insufficient for GEC models.

Copied to clipboard

Yuxia Wang, Rui Xing, Jonibek Mansurov, Giovanni Puccetti, Zhuohan Xie, Minh Ngoc Ta, Jiahui Geng, Jinyan Su, Mervat Abassy, Saadeldine Eletter, Kareem Elozeiri, Nurkhan Laiyk, Maiya Goloburda, Tarek Mahmoud, Raj Vardhan Tomar, Alexander Aziz, Ryuto Koike, Masahiro Kaneko, Artem Shelmanov, Ekaterina Artemova, Vladislav Mikhailov, Akim Tsvigun, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, Preslav Nakov

Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020.acl-main)

Challenge:	Prior studies have shown that distinguishing text generated by Large Language Models from human-written text is challenging for humans and often no better than random guessing.
Approach:	They conduct extensive case study to determine the upper bound of human detection accuracy.
Outcome:	The findings challenge previous conclusions on human detection accuracy across languages and domains.

Copied to clipboard

Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, Kentaro Inui

Debiasing Pre-trained Contextualised Embeddings (2021.eacl-main)

Challenge:	Existing methods for incorporating a masked language model into an EncDec model have potential drawbacks when applied to GEC.
Approach:	They propose to incorporate a pre-trained masked language model (MLM) into an encoder-decoder model for grammatical error correction.
Outcome:	The proposed method achieves state-of-the-art on BEA-2019 and CoNLL-2014 benchmarks.

Copied to clipboard

Sentence Concatenation Approach to Data Augmentation for Neural Machine Translation (2021.naacl-srw)

Challenge:	a study of contextualised word embeddings shows discriminative biases are encoded in contextualised embeddables.
Approach:	They propose a fine-tuning method that can be applied at token- or sentence-levels to debias pre-trained contextualised embeddings.
Outcome:	The proposed method can be applied at token- or sentence-levels to debias pre-trained models without requiring retrains.

Copied to clipboard

Seiichiro Kondo, Kengo Hotate, Tosho Hirasawa, Masahiro Kaneko, Mamoru Komachi

Text Generation

Challenge:	Neural machine translation is known to show poor performance at long sentence translations . however, when the sentence length exceeds a certain value, the quality of NMT becomes inferior to that of statistical machine translation.
Approach:	They propose a method that uses given parallel corpora as train data to generate long sentences by concatenating two sentences at random.
Outcome:	The proposed method improves translation quality more when combined with back-translation.

Japanese-Russian TMU Neural Machine Translation System using Multilingual Model for WAT 2019 (D19-52)

Copied to clipboard

Aizhan Imankulova, Masahiro Kaneko, Mamoru Komachi

Multilinguality

Challenge:	Using parallel corpora of different language pairs as training data is effective for multilingual neural machine translation model in extremely low resource situations.
Approach:	They propose to use Japanese-English and English-Russian parallel corpora as training data for their system to improve JapaneseRussian news translation.
Outcome:	The proposed system improves translation quality for JapaneseRussian language pairs in low resource situations.

How You Prompt Matters! Even Task-Oriented Constraints in Instructions Affect LLM-Generated Text Detection (2024.findings-emnlp)

Copied to clipboard

Ryuto Koike, Masahiro Kaneko, Naoaki Okazaki

ExaGPT: Example-Based Machine-Generated Text Detection for Human Interpretability (2026.findings-acl)

Challenge:	Recent studies have presented LLM-generated-text detectors with promising performance, but they do not cover such diverse instruction patterns when creating datasets for LLM detection.
Approach:	They propose to use task-oriented constraints that would naturally be included in an instruction and are not related to detection-evasion to create detectors with large variances in detection performance.
Outcome:	The proposed detectors have a large variance in detection performance on student essay writing with task-oriented constraints, and the standard deviation is significantly larger than that on texts generated by the constraint with such a constraint.

Copied to clipboard

Ryuto Koike, Masahiro Kaneko, Ayana Niwa, Preslav Nakov, Naoaki Okazaki

Explainability & Interpretability in NLP

Challenge:	Existing interpretable detectors are not aligned with the human decision-making process and fail to offer evidence that users easily understand.
Approach:	They propose an interpretable detection approach that checks whether a text is human-written or LLM-generated by checking whether it shares more similar spans with human-generated texts.
Outcome:	ExaGPT outperforms interpretable detectors by +37.0 points at a false positive rate of 1%.

A Self-Refinement Strategy for Noise Reduction in Grammatical Error Correction (2020.findings-emnlp)

Copied to clipboard

Masato Mita, Shun Kiyono, Masahiro Kaneko, Jun Suzuki, Kentaro Inui

SOME: Reference-less Sub-Metrics Optimized for Manual Evaluations of Grammatical Error Correction (2020.coling-main)

Challenge:	Existing approaches for grammatical error correction (GEC) rely on supervised learning with manually created datasets.
Approach:	They propose to denoise GEC datasets by leveraging prediction consistency of existing models.
Outcome:	The proposed method outperforms baseline methods on CoNLL-2014, JFLEG, and BEA-2019 benchmarks.

Copied to clipboard

Ryoma Yoshimura, Masahiro Kaneko, Tomoyuki Kajiwara, Mamoru Komachi

Debiasing Isn’t Enough! – on the Effectiveness of Debiasing MLMs and Their Social Biases in Downstream Tasks (2022.coling-1)

Challenge:	Existing reference-less metrics are not optimized for manual evaluations of system outputs because no dataset exists for manual analysis.
Approach:	They propose a reference-less metric trained on manual evaluations of system outputs for grammatical error correction.
Outcome:	The proposed metric improves correlation with manual evaluation in system- and sentence-level meta-evaluation.

Copied to clipboard

Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki

Generating Diverse Corrections with Local Beam Search for Grammatical Error Correction (2020.coling-main)

Challenge:	Existing measures for social bias evaluation are inadequate for MLMs to accurately evaluate the social biases in their systems.
Approach:	They propose task-agnostic intrinsic and task-specific extrinsic social bias evaluation measures for MLMs that use different methods to re-learn social biases during fine-tuning on downstream tasks.
Outcome:	The findings highlight the limitations of existing MLM bias evaluation measures and raise concerns on the deployment of MLMs in downstream applications using those measures.

Copied to clipboard

Kengo Hotate, Masahiro Kaneko, Mamoru Komachi

ExtraPhrase: Efficient Data Augmentation for Abstractive Summarization (2022.naacl-srw)

Challenge:	Existing methods of acquiring diverse outputs focus on revising all tokens of a sentence.
Approach:	They propose a beam search method to obtain diverse outputs in a local sequence transduction task where most of the tokens in the source and target sentences overlap.
Outcome:	The proposed method generates more diverse corrections without losing accuracy in the local sequence transduction task.

Copied to clipboard

Mengsay Loem, Sho Takase, Masahiro Kaneko, Naoaki Okazaki

The Gaps between Fine Tuning and In-context Learning in Bias Evaluation and Debiasing (2025.coling-main)

Challenge:	Recent studies indicated that neural methods are governed by the scaling law for the amount of training data.
Approach:	They propose a low-cost strategy to augment training data for abstractive summarization tasks by extracting summarized text and paraphrasing it.
Outcome:	The proposed strategy outperforms back-translation and self-training and is more cost-efficient when training data is small.

Copied to clipboard

Masahiro Kaneko, Danushka Bollegala, Timothy Baldwin

Gender-preserving Debiasing for Pre-trained Word Embeddings (P19-1)

Challenge:	FT-based debiasing methods cause a performance degradation in downstream tasks . FT works by updating some or all parameters, while ICL uses prompts without modifying the model parameters.
Approach:	They propose to use ICL to customize PLMs to downstream tasks without parameter updates.
Outcome:	The proposed method lowers the performance degradation of FT-based debiasing methods compared to FT models . the proposed method improves performance on large datasets while allowing for smaller changes to PLMs .

Copied to clipboard

Gender Bias in Meta-Embeddings (2022.findings-emnlp)

Challenge:	Existing methods for debiasing word embeddings have shown discriminative biases . word embeds learnt from social media have shown to encode racist, offensive and discriminative language usage.
Approach:	They propose a method that preserves gender-related information while removing stereotypical gender biases from pre-trained word embeddings.
Outcome:	The proposed method preserves gender-related information while removing stereotypical discriminative gender biases from pre-trained word embeddings.

Copied to clipboard

Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki

Interpretability for Language Learners Using Example-Based Grammatical Error Correction (2022.acl-long)

Challenge:	Existing methods to develop meta-embeddings from source embeddings contain unfair gender-related biases, and how these influence the meta-bedding has not been studied yet.
Approach:	They propose to use multiple debiasing methods on a single source embedding to create a gender-based meta-embedding.
Outcome:	The proposed method amplifies gender biases compared to input source embeddings.

Copied to clipboard

Masahiro Kaneko, Sho Takase, Ayana Niwa, Naoaki Okazaki

Sense Embeddings are also Biased – Evaluating Social Biases in Static and Contextualised Sense Embeddings (2022.acl-long)

Challenge:	Existing neural-based GEC models mainly aim at improving accuracy, but their interpretability has not been explored.
Approach:	They propose an example-based method that generates corrections using retrieved examples.
Outcome:	The proposed method improves interpretability and supports language learners.

Copied to clipboard

Yi Zhou, Masahiro Kaneko, Danushka Bollegala

Representation Learning

Challenge:	Existing studies have evaluated social biases in word embeddings, but they are understudied.
Approach:	They propose to evaluate the social biases in sense embeddings using a benchmark dataset for word embedders.
Outcome:	The proposed measures show that even when no biases are found at word-level, there are still worrying levels of social biase at sense-level which are often ignored by the word- level bias evaluation measures.

Rectifying Belief Space via Unlearning to Harness LLMs’ Reasoning (2025.findings-acl)

Copied to clipboard

Ayana Niwa, Masahiro Kaneko, Kentaro Inui

Reasoning

Challenge:	Large Language Models (LLMs) exhibit sophisticated reasoning yet still generate incorrect answers.
Approach:	They propose a belief space rectification framework that suppresses spurious beliefs and enhances true ones to reduce erroneous reasoning and generalization.
Outcome:	The proposed framework reduces erroneous reasoning and improves generalization on three QA datasets and three LLMs.

Solving NLP Problems through Human-System Collaboration: A Discussion-based Approach (2024.findings-eacl)

Copied to clipboard

Masahiro Kaneko, Graham Neubig, Naoaki Okazaki

Explainability & Interpretability in NLP

Challenge:	Existing systems that make predictions and ask questions are unable to have a mutual exchange of opinions.
Approach:	They propose to use a dataset and computational framework to allow systems to have beneficial discussions with humans, improving the accuracy by 25 points on a natural language inference task.
Outcome:	The proposed system improves accuracy by 25 points on a natural language inference task.

Autoencoding Improves Pre-trained Word Embeddings (2020.coling-main)

Copied to clipboard

Evaluating Gender Bias of Pre-trained Language Models in Natural Language Inference by Considering All Labels (2024.lrec-main)

Challenge:	Existing work has shown that word embeddings are distributed in a narrow cone and that centering and projection can improve the accuracy of pre-trained word embeds without requiring additional training data.
Approach:	They propose to remove the top principal components from pre-trained word embeddings and center and project them onto principal component vectors to reinstate isotropy in the embeddable space.
Outcome:	The proposed method is equivalent to applying a linear autoencoder to minimize the squared L2 reconstruction error.

Copied to clipboard

Panatchakorn Anantaprayoon, Masahiro Kaneko, Naoaki Okazaki

Reducing Sequence Length by Predicting Edit Spans with Large Language Models (2023.emnlp-main)

Challenge:	Existing methods to evaluate gender bias in PLMs focus on one label out of three labels, such as neutral.
Approach:	They propose a bias evaluation method for PLMs that considers all the three labels of NLI task and then defines a measure based on the corresponding label output.
Outcome:	The proposed method can distinguish biased, incorrect inferences from non-biased incorrect infertility better than baseline, resulting in a more accurate bias evaluation.

Copied to clipboard

Masahiro Kaneko, Naoaki Okazaki

Cross-lingual Transfer Learning for Grammatical Error Correction (2020.coling-main)

Challenge:	Large Language Models (LLMs) have demonstrated remarkable performance in various tasks and gained significant attention.
Approach:	They propose to predict edit spans for local sequence transduction tasks by predicting edit span with a position of the source text and corrected tokens.
Outcome:	The proposed method reduces the length of the target sequence and the computational cost for inference by as small as 21%.

Copied to clipboard

Ikumi Yamashita, Satoru Katsumata, Masahiro Kaneko, Aizhan Imankulova, Mamoru Komachi

Balanced Multi-Factor In-Context Learning for Multilingual Large Language Models (2025.emnlp-main)

Challenge:	Existing studies on English GEC have focused on improving it, but the resources required to train the models are not sufficient.
Approach:	They investigate cross-lingual transfer learning in grammatical error correction tasks . similarities between these languages is a key factor for successfully transferring grammatikal knowledge .
Outcome:	The proposed methods improve accuracy of grammatical error correction tasks in English and Russian, but lack the resources to train models in these languages.

Copied to clipboard

Masahiro Kaneko, Alham Fikri Aji, Timothy Baldwin

ProQE: Proficiency-wise Quality Estimation dataset for Grammatical Error Correction (2022.lrec-1)

Challenge:	Existing approaches address key factors that influence multilingual ICL, but they do not integrate them into the model.
Approach:	They propose a method that quantifies and optimally balances three factors for improved example selection.
Outcome:	Experiments on mCSQA and TYDI show that the proposed method outperforms existing methods.

Copied to clipboard

Yujin Takahashi, Masahiro Kaneko, Masato Mita, Mamoru Komachi

Comparing Intrinsic Gender Bias Evaluation Measures without using Human Annotated Examples (2023.eacl-main)

Challenge:	Prior work has shown that QE models of grammatical error correction are biased toward data by learners with relatively high proficiency levels.
Approach:	They investigated whether learners' proficiency affects supervised quality estimation models of grammatical error correction (GEC) . they created a QE dataset that includes multiple proficiency levels and explored the necessity of performing proficiency-wise evaluation for QE of GEC.
Outcome:	The proposed model is based on multiple proficiency levels and can be performed in real-world scenarios.

Copied to clipboard

Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki