Papers by Jun Araki
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent studies have shown that language models capture different types of knowledge regarding facts or commonsense knowledge. |
| Approach: | They examine how language models can be calibrated to make their confidence scores correlate better with the likelihood of correctness. |
| Outcome: | The proposed calibration methods improve confidence scores on QA tasks and improve accuracy. |
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Language models (LMs) capture factual knowledge by filling in the blanks of cloze-style prompts. |
| Approach: | They propose a code-switching-based method to improve the ability of multilingual LMs to access knowledge and verify its effectiveness on several benchmark languages. |
| Outcome: | The proposed method improves the ability of multilingual LMs to access knowledge and verify its effectiveness on several benchmark languages. |
ToolDreamer: Instilling LLM Reasoning Into Tool Retrievers (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing retrieval models rank tools based on similarity between query and tool description (TD) Existing tools are not conditioned to learn tool-to-tool relationships (middle). |
| Approach: | They propose a framework that conditions retrieval models to fetch tools based on hypothetical (synthetic) TD generated using an LLM. |
| Outcome: | The proposed framework improves the performance of sparse and dense retrievers with and without training, showcasing its flexibility. |
DelucionQA: Detecting Hallucinations in Domain-specific Question Answering (2023.findings-emnlp)
Copied to clipboard
Mobashir Sadat, Zhengyu Zhou, Lukas Lange, Jun Araki, Arsalan Gundroo, Bingqing Wang, Rakesh Menon, Md Parvez, Zhe Feng
| Challenge: | Hallucination is a well-known phenomenon in text generated by large language models . state-of-the-art LLMs still have a number of weaknesses, including the tendency to generate hallucinatory statements without considering the factuality . |
| Approach: | They propose a dataset that captures hallucinations made by retrieval-augmented LLMs . they propose to use these methods to help detect hallucinosity in QA tasks . |
| Outcome: | The proposed method captures hallucinations made by retrieval-augmented LLMs for QA tasks. |
Generalizing Natural Language Analysis through Span-relation Representations (2020.acl-main)
Copied to clipboard
| Challenge: | a large number of natural language processing tasks are generated with specially designed architectures. |
| Approach: | They propose to represent a wide variety of tasks in a single unified format . they perform extensive experiments to demonstrate benefits of multi-task learning . |
| Outcome: | The proposed model performs comparable to state-of-the-art models on 10 tasks . it also shows that it can analyze differences and similarities in how the model handles different tasks compared to other models . |
SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent work shows promising results when prompting pre-trained language models, but in low-resource domains, the domain gap between the pre-training data and the downstream task is too large. |
| Approach: | They propose a method for prompting pre-trained language models using domain-specific keywords with a trainable gated prompt. |
| Outcome: | The proposed prompting method outperforms state-of-the-art prompting methods on three text classification benchmarks and shows that it reduces the need for domain-specific language model pre-training. |
Understanding and Improving Zero-shot Multi-hop Reasoning in Generative Question Answering (2022.coling-1)
Copied to clipboard
| Challenge: | Generative question answering (QA) models generate answers to complex questions, but their mechanism for doing so is still poorly understood. |
| Approach: | They decompose multi-hop questions into multiple corresponding single-hop question chains and find marked inconsistency in QA models’ answers on these pairs of ostensibly identical question chains. |
| Outcome: | The proposed models lack zero-shot multi-hop reasoning ability when trained on single-hop questions and on logical forms. |
Open-Domain Event Detection using Distant Supervision (C18-1)
Copied to clipboard
| Challenge: | Existing work on restricted domains and event annotation has limited coverage of events. |
| Approach: | They propose a distant supervision method that generates high-quality training data . they use a manually annotated corpus as a model to investigate events in various domains . |
| Outcome: | The proposed method outperforms supervised models in a manually annotated event corpus despite no direct supervision . |
Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer (2022.emnlp-main)
Copied to clipboard
| Challenge: | eschewing separate architecture and training for knowledge-intensive tasks is cumbersome . end-to-end training only based on supervision from the end task is awkward . |
| Approach: | They propose a single Transformer that performs retrieval as attention and end-to-end training solely based on supervision from the end QA task. |
| Outcome: | The proposed model outperforms state-of-the-art retrievers and readers on in-domain datasets. |
How Can We Know What Language Models Know? (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent work examines knowledge contained in language models by having the LM fill in the blanks of prompts such as “Obama is a __ by profession”. |
| Approach: | They propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. |
| Outcome: | The proposed methods improve accuracy from 31.1% to 39.6% on the LAMA benchmark for extracting relational knowledge from LMs. |
CoAug: Combining Augmentation of Labels and Labelling Rules (2023.findings-acl)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) tasks require large labeled datasets to perform well. |
| Approach: | They propose a co-augmentation framework that bootstraps predictions from each model to improve few-shot models and rule-augmentation models by bootstrapping them. |
| Outcome: | The proposed model outperforms strong weak-supervision-based models by 6.5 F1 points . the proposed model can learn from limited labeled data and perform better on small datasets . |
Explicitly Capturing Relations between Entity Mentions via Graph Neural Networks for Domain-specific Named Entity Recognition (2021.acl-short)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is well studied for the general domain, but the performance is still moderate for specialized domains. |
| Approach: | They propose to explicitly connect entity mentions based on global coreference relations and local dependency relations to build better entity mention representations. |
| Outcome: | The proposed system improves the NER performance even with a tiny amount of labeled data. |
A Textual Dataset for Situated Proactive Response Selection (2023.acl-long)
Copied to clipboard
| Challenge: | Recent data-driven conversational models can return fluent, consistent, and informative responses to many kinds of requests and utterances in task-oriented scenarios. |
| Approach: | They propose a task of proactive response selection based on situational information and a dataset of 1.7k English conversation examples that include situational background information and for each conversation a set of responses. |
| Outcome: | The proposed model can only provide fluent, consistent, and informative responses to a set of 1.7k English conversation examples and is not easy to perform for strong neural models. |