Papers by Jiannan Xiang
Learning to Stop: A Simple yet Effective Approach to Urban Vision-Language Navigation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models treat STOP as other actions, which leads to undesirable behaviors that the agent fails to stop at the destination. |
| Approach: | They propose a policy module that differentiates STOP from other actions . they propose 'learning to stop' module that can be used to train an agent to follow natural language instructions in real-world environments. |
| Outcome: | The proposed model outperforms the baseline on a challenging urban VLN dataset Touchdown by 6.89%. |
Investigating Data Variance in Evaluations of Automatic Machine Translation Metrics (2022.findings-acl)
Copied to clipboard
| Challenge: | Current evaluation methods focus on one dataset, e.g., Newstest dataset in each year’s WMT Metrics Shared Task. |
| Approach: | They propose to use a single dataset to evaluate the performance of automatic translation metrics. |
| Outcome: | The results show that the rankings of metrics vary when the evaluation is conducted on different datasets. |
Visualizing the Relationship Between Encoded Linguistic Information and Task Performance (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that encoding more syntactic information does not lead to better performance. |
| Approach: | They propose a method to optimize pareto-optimal models by formalizing it as a multi-objective optimization problem. |
| Outcome: | The proposed method is better than a baseline method on two NLP tasks. |
Assessing Dialogue Systems with Distribution Distances (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics focus on turnlevel quality, which is not well suited for open-end dialogue tasks. |
| Approach: | They propose to measure the performance of a dialogue system by computing the distributionwise distance between its generated conversations and real-world conversations. |
| Outcome: | The proposed metrics correlate better with human judgments than existing metrics on dialogue systems. |
Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation (2025.findings-acl)
Copied to clipboard
Qiyue Gao, Xinyu Pi, Kevin Liu, Junrong Chen, Ruolan Yang, Xinqi Huang, Xinyu Fang, Lu Sun, Gautham Kishore, Bo Ai, Stone Tao, Mengyang Liu, Jiaxi Yang, Chao-Jung Lai, Chuanyang Jin, Jiannan Xiang, Benhao Huang, Zeming Chen, David Danks, Hao Su, Tianmin Shu, Ziqiao Ma, Lianhui Qin, Zhiting Hu
| Challenge: | Recent studies have evaluated and shown limitations in specific capabilities such as visual understanding, but a systematic evaluation of VLMs’ fundamental WM abilities remains absent. |
| Approach: | They propose a framework that assesses perception and prediction to provide an atomic evaluation of VLMs as WMs. |
| Outcome: | The proposed framework assesses perception and prediction abilities on 15 latest VLMs and compares them to human-level models. |
ASDOT: Any-Shot Data-to-Text Generation with Pretrained Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to data-to-text generation require limited training examples . a data-based approach is based on a set of pre-trained language models with optional finetuning. |
| Approach: | They propose a data-to-text generation task that makes use of any given (or no) examples. |
| Outcome: | The proposed approach improves on baselines on a dataset with zero/few/full-shot settings. |