Papers by Sho Yokoi
Filtering Noisy Dialogue Corpora by Connectivity and Content Relatedness (2020.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale dialogue datasets contain a non-negligible number of unacceptable utterance pairs . previous studies have identified such flaws and reported that the corpus is noisy . |
| Approach: | They propose a method for scoring the quality of utterance pairs based on their connectivity and relatedness. |
| Outcome: | The proposed method has a good correlation with human judgment of dialogue quality and is applied to training data filtered by the proposed method. |
Modeling Event Salience in Narratives via Barthes’ Cardinal Functions (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for estimating event salience without annotations are prohibitively costly because they require annotators to understand the concept of event salientity. |
| Approach: | They propose to use Barthes’ definition of event salience to compute event salientity without annotations by using a pre-trained language model. |
| Outcome: | The proposed methods outperform baseline methods on folktales with event salience annotation and fine-tuned language model is key factor in improving the methods. |
Unbalanced Optimal Transport for Unbalanced Word Alignment (2023.acl-long)
Copied to clipboard
| Challenge: | Figure 1 illustrates the challenges of monolingual word alignment. |
| Approach: | They propose to use the family of optimal transport (OT) to achieve unbalanced word alignment that values alignment and null alignment on unsupervised datasets. |
| Outcome: | The proposed methods are competitive against the state-of-the-art methods on challenging datasets with high null alignment frequencies. |
Unsupervised Learning of Style-sensitive Word Vectors (P18-2)
Copied to clipboard
| Challenge: | Existing studies on what is said and how it is said focus on stylistic variations . lack of objective definitions is a major difficulty in studying style . |
| Approach: | They propose to extend the continuous bag of words embedding model to learn style-sensitive word vectors using a wider context window. |
| Outcome: | The proposed extensions contribute to the acquisition of style-sensitive word embeddings. |
Instance-Based Learning of Span Representations: A Case Study through Named Entity Recognition (2020.acl-main)
Copied to clipboard
Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi, Sho Yokoi, Tatsuki Kuribayashi, Ryuto Konno, Kentaro Inui
| Challenge: | Recent neural networks can induce good span feature representations and achieve high performance in structured prediction tasks. |
| Approach: | They propose an instance-based learning method that learns similarities between spans . they aim to build models that have high interpretability without sacrificing performance . |
| Outcome: | The proposed method improves interpretability without sacrificing performance. |
Transformer Language Models Handle Word Frequency in Prediction Head (2023.findings-acl)
Copied to clipboard
| Challenge: | Prediction head is a crucial component of Transformer language models. Despite its direct impact on prediction, its characteristics have been overlooked in previous analyses. |
| Approach: | They examine the inner workings of the prediction head, specifically the bias parameters, and quantify the effect of controlling their frequency biases on text generation. |
| Outcome: | The prediction head is a crucial component of the Transformer language models. |
Computationally Efficient Wasserstein Loss for Structured Labels (2021.eacl-srw)
Copied to clipboard
| Challenge: | Existing approaches to estimate the probability distribution of labels are based on tree-Wasserstein distance. |
| Approach: | They propose a tree-Wasserstein distance regularized LDL algorithm for hierarchical text classification tasks. |
| Outcome: | The proposed method performs well on synthetic and real-world datasets and compares favorably with the Sinkhorn algorithm in terms of computation time and memory usage. |
Norm of Word Embedding Encodes Information Gain (2023.emnlp-main)
Copied to clipboard
| Challenge: | Distributed representations of words encode lexical semantic information, but what type of information is encoded and how? |
| Approach: | They propose to use the squared norm of static word embedding to encode the information gain conveyed by the word. |
| Outcome: | The proposed model can encode the information gain of a word in a language model or neural network. |
Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings (2026.acl-long)
Copied to clipboard
| Challenge: | Text embeddings are used across a wide range of NLP tasks, including retrieval-augmented generation. |
| Approach: | They propose a metric to quantify the collapse induced by mean pooling and a simple metric for measuring how often it occurs in real models and texts. |
| Outcome: | The proposed metric measures how often the collapse occurs in real models and texts. |
Improving word mover’s distance by leveraging self-attention matrix (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Word mover's distance (WMD) computes semantic similarity via the optimal alignment between word embeddings, but does not consider word order, making it challenging to distinguish sentences with significant overlaps of similar words. |
| Approach: | They propose to incorporate the sentence structure represented by BERT’s self-attention matrix (SAM) to improve WMD and its variants in paraphrase identification. |
| Outcome: | The proposed method improves WMD and its variants in paraphrase identification with near-equivalent performance in semantic textual similarity. |
Quantifying Lexical Semantic Shift via Unbalanced Optimal Transport (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for detecting semantic change only measure the level of individual usage instances. |
| Approach: | They propose to use unbalanced optimal transport to capture semantic change through excess and deficit in the alignment between usage instances. |
| Outcome: | The proposed method captures semantic change through excess and deficit in the alignment between usage instances. |
Incorporating Residual and Normalization Layers into Analysis of Masked Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Transformer architecture is composed of multi-head attention, which has been extensively analyzed. |
| Approach: | They extended the scope of the analysis of Transformers from solely the attention patterns to the whole attention block, i.e., multi-head attention, residual connection, and layer normalization. |
| Outcome: | The proposed method incorporates the whole attention block, i.e., multi-head attention, residual connection, and layer normalization into the analysis. |
Attention is Not Only a Weight: Analyzing Transformers with Vector Norms (2020.emnlp-main)
Copied to clipboard
| Challenge: | Attention is a key component of Transformers, which have achieved considerable success in natural language processing. |
| Approach: | They propose to integrate attention weights and the norm of transformed input vectors into a norm-based analysis that incorporates the norm. |
| Outcome: | The proposed analysis shows that attention weights alone determine the output of attention and that reasonable word alignment can be extracted from attention mechanisms of Transformers. |
Instance-Based Neural Dependency Parsing (2021.tacl-1)
Copied to clipboard
Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi, Sho Yokoi, Tatsuki Kuribayashi, Masashi Yoshikawa, Kentaro Inui
| Challenge: | Existing models that use instance-based inference for dependency parsing are difficult to understand for humans. |
| Approach: | They develop neural models that adopt an interpretable inference process for dependency parsing. |
| Outcome: | The proposed models achieve competitive accuracy with standard neural models and have plausibility of instance-based explanations. |
Pointwise HSIC: A Linear-Time Kernelized Co-occurrence Norm for Sparse Linguistic Expressions (D18-1)
Copied to clipboard
| Challenge: | Empirically, PHSIC is learned thousands of times faster than an RNN-based PMI while outperforming PMI in accuracy. |
| Approach: | They propose a new kernel-based co-occurrence measure that can be applied to sparse linguistic expressions with a very short learning time. |
| Outcome: | The proposed measure can be applied to sparse linguistic expressions with a very short learning time, and is called the pointwise HSIC. |
Word Rotator’s Distance (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to measure textual similarity are inconsistent with the word alignment and are empirically inferior to the simple cosine similarity between general-purpose sentence vectors. |
| Approach: | They propose to decouple word vectors into their norm and direction and then grow the norm and directions of word vector. |
| Outcome: | The proposed methods outperform alignment-based approaches on several benchmarks and strong baselines on the semantic textual similarity task. |
Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Words (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Embedding a sentence into a point in a highdimensional continuous space plays a foundational role in the natural language processing. |
| Approach: | They propose to use contrastive loss to fine-tune sentences by inverse word frequency . they also show that more informative words receive greater weight than less informative ones . |
| Outcome: | The proposed method improves the performance of sentence embeddings by weighing them based on information-theoretic quantities. |
On Entity Identification in Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work has shed light on the internal mechanisms of language models that can recall factual knowledge composed of entities and relations. |
| Approach: | They propose a framework analogous to clustering quality metrics to analyze the correspondence between entities and their mentions. |
| Outcome: | The proposed framework is analogous to clustering quality metrics. |
Subspace Representations for Soft Set Operations and Sentence Similarities (2024.naacl-long)
Copied to clipboard
| Challenge: | Embedding-based word representations are crucial for capturing the semantic meanings of individual words. |
| Approach: | They propose to embed word sets and corresponding set operations within pre-trained word embedding spaces. |
| Outcome: | The proposed representations outperform vector-based representations in sentence similarity and set retrieval tasks. |
Timesteps of Mamba Align with Human Reading Times (2026.findings-acl)
Copied to clipboard
| Challenge: | In Mamba, the recurrent state transition at each layer conceptually takes some duration of time, the discretization timestep t, determined dynamically in response to the input. |
| Approach: | They propose to align per-word processing time in a popular state-space language model Mamba with human reading time using a naturalistic reading dataset. |
| Outcome: | The proposed model can predict reading times comparable to baselines such as word frequency and GPT-2 surprisal and significant even when they are controlled for. |