Papers by Qiyue Gao
Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing evaluation methods do not provide insight into how well a language model captures distinct linguistic skills essential for language understanding and reasoning. |
| Approach: | They propose a new format of NLI benchmark for evaluation of broad-coverage linguistic phenomena using a set of datasets and an evaluation procedure for diagnosing how well a language model captures reasoning skills. |
| Outcome: | The proposed model can diagnose model behavior and verify model learning quality. |
DISCO: Distilling Counterfactuals with Large Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | high-quality counterfactual data is scarce for most tasks and not easily generated at scale. |
| Approach: | They propose a method for automatically generating high-quality counterfactual data at scale . they use a large general language model to generate phrasal perturbations and filter them . |
| Outcome: | The proposed method is task-agnostic and can be applied to the task of natural language inference. |
Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation (2025.findings-acl)
Copied to clipboard
Qiyue Gao, Xinyu Pi, Kevin Liu, Junrong Chen, Ruolan Yang, Xinqi Huang, Xinyu Fang, Lu Sun, Gautham Kishore, Bo Ai, Stone Tao, Mengyang Liu, Jiaxi Yang, Chao-Jung Lai, Chuanyang Jin, Jiannan Xiang, Benhao Huang, Zeming Chen, David Danks, Hao Su, Tianmin Shu, Ziqiao Ma, Lianhui Qin, Zhiting Hu
| Challenge: | Recent studies have evaluated and shown limitations in specific capabilities such as visual understanding, but a systematic evaluation of VLMs’ fundamental WM abilities remains absent. |
| Approach: | They propose a framework that assesses perception and prediction to provide an atomic evaluation of VLMs as WMs. |
| Outcome: | The proposed framework assesses perception and prediction abilities on 15 latest VLMs and compares them to human-level models. |
NeuralLog: Natural Language Inference with Joint Neural and Logical Reasoning (2021.starsem-1)
Copied to clipboard
| Challenge: | Currently, symbolic and deep learning approaches to NLI are receiving less attention. |
| Approach: | They propose a symbolic-based inference framework that integrates symbolic reasoning and semantic formalism to solve NLI tasks. |
| Outcome: | The proposed framework improves accuracy on the NLI task and on the SICK and MED datasets. |