Papers by Justin Vasselli
Multilingual Dialogue Generation and Localization with Dialogue Act Scripting (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to training or evaluating non-English dialogue datasets often introduce artifacts that reduce their naturalness and cultural appropriateness. |
| Approach: | They propose a structured framework for encoding, localizing, and generating multilingual dialogues from abstract intent representations. |
| Outcome: | The proposed framework outperforms translation models in Italian, German, and Chinese on cultural relevance, coherence, and situational appropriateness. |
How to Make the Most of LLMs’ Grammatical Knowledge for Acceptability Judgments (2025.naacl-long)
Copied to clipboard
Yusuke Ide, Yuto Nishida, Justin Vasselli, Miyu Oba, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe
| Challenge: | Conventional approaches compare sentence probabilities directly, but large language models (LLMs) provide nuanced evaluation methods using prompts and templates. |
| Approach: | They propose to derive acceptability judgments from large language models using prompts and templates to comprehensively evaluate their grammatical knowledge. |
| Outcome: | The proposed methods excel in different linguistic phenomena, suggesting they access different aspects of the LLMs’ grammatical knowledge. |
Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries (2025.findings-acl)
Copied to clipboard
Haruki Sakajo, Yusuke Ide, Justin Vasselli, Yusuke Sakai, Yingtao Tian, Hidetaka Kamigaito, Taro Watanabe
| Challenge: | Existing approaches to cross-lingual vocabulary transfer face challenges when dealing with low-resource languages. |
| Approach: | They propose a dictionary-based crosslingual vocabulary transfer method that leverages bilingual dictionaries, which are available for many languages thanks to descriptive linguists. |
| Outcome: | The proposed method outperforms existing methods for low-resource languages. |
Measuring Linguistic Competence of LLMs on Indigenous Languages of the Americas (2026.eacl-short)
Copied to clipboard
| Challenge: | Existing benchmarks for linguistic knowledge of Indigenous languages of the Americas focus on high- and medium-resource languages with substantial digital presence. |
| Approach: | They propose a framework for probing large language models’ linguistic knowledge of Indigenous languages of the Americas using zero-shot prompting and few-shot probing. |
| Outcome: | The proposed framework evaluates models from five major families on 13 Indigenous languages including Bribri, Guarani, and Nahuatl. |
Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction (2025.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation metrics for Grammatical error correction lack explainability . lack of explainability hinders researchers from analyzing strengths and weaknesses of models . |
| Approach: | They propose to assign sentence-level scores to individual edits to improve GEC performance . they use Shapley values, from cooperative game theory, to compute contribution of each edit . |
| Outcome: | The proposed method shows that the evaluation metrics are consistent across edits and human evaluations. |
Measuring the Robustness of Reference-Free Dialogue Evaluation Systems (2025.coling-main)
Copied to clipboard
| Challenge: | Advancements in dialogue systems powered by large language models have outpaced the development of reliable evaluation metrics. |
| Approach: | They propose a benchmark to evaluate the robustness of reference-free dialogue metrics against four categories of adversarial attacks. |
| Outcome: | The proposed benchmarks show that the two axes of reliability are not always aligned . the findings motivate the development of nuanced evaluation frameworks to address real-world dialogue challenges. |
Beyond Film Subtitles: Is YouTube the Best Approximation of Spoken Vocabulary? (2025.coling-main)
Copied to clipboard
Adam Nohejl, Frederikus Hudi, Eunike Andriani Kardinata, Shintaro Ozaki, Maria Angelica Riera Machin, Hongyu Sun, Justin Vasselli, Taro Watanabe
| Challenge: | Word frequency is a key variable in psycholinguistics, useful for modeling human familiarity with words . a recent study shows that frequency from YouTube subtitles is comparable to and often better than the best available resources. |
| Approach: | They use YouTube subtitles to construct frequency norms for five languages . they find they are comparable to and often better than the best currently available resources . |
| Outcome: | The proposed method improves on the best currently available resources for Chinese, English, Indonesian, Japanese, and Spanish. |
Translating Movie Subtitles by Large Language Models using Movie-meta Information (2025.acl-srw)
Copied to clipboard
| Challenge: | Large language models (LLMs) have advanced natural language processing by understanding, generating, and manipulating texts. |
| Approach: | They propose to use movie subtitle prompts to improve translation accuracy by incorporating movie meta-information into the models. |
| Outcome: | The proposed prompts improve translation accuracy and reduce computational effort. |
Dynamic Meta-Metrics: Source-Sentence Conditioned Weighting for MT Evaluation (2026.acl-srw)
Copied to clipboard
| Challenge: | Rather than relying on a single static ensemble or language-specific weighting, DMM adapts the metric combination based on properties of the source segment. |
| Approach: | They propose a framework for machine translation evaluation that learns source-sentence conditioned combinations of existing metrics. |
| Outcome: | The proposed framework outperforms linear and Gaussian process-based ensembles across multiple language pairs and introducing soft conditioning yields gains over linear models. |
CoAM: Corpus of All-Type Multiword Expressions (2025.acl-long)
Copied to clipboard
Yusuke Ide, Joshua Tanner, Adam Nohejl, Jacob Hoffman, Justin Vasselli, Hidetaka Kamigaito, Taro Watanabe
| Challenge: | Existing datasets for multiword expressions are inconsistently annotated, limited to a single type of MWE, or limited in size. |
| Approach: | They propose to use a new interface to generate MWE annotations for the first time in a dataset of MWE identification. |
| Outcome: | The proposed model outperforms existing models on the DiMSUM dataset. |