Papers by Zi-Yi Dou
FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigation (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing speaker-follower models are follower-agnostic and fail to take state of follower into account. |
| Approach: | They propose a speaker-follower model that is constantly updated given follower feedback . they optimize the speaker and obtain its training signals by evaluating the follower on labeled data . |
| Outcome: | The proposed model outperforms strong baseline models on room-to-room and room-across-room datasets. |
Word Alignment by Fine-tuning Embeddings on Parallel Corpora (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing work on word alignment has focused on unsupervised learning on parallel text. |
| Approach: | They propose to combine pre-trained contextualized word embeddings with multilingually trained language models to achieve competitive results on word alignment tasks. |
| Outcome: | The proposed model outperforms state-of-the-art models on five language pairs and can train multilingual word aligners that can obtain robust performance on different language pairs. |
Gender Biases in Automatic Evaluation Metrics for Image Captioning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained evaluation metrics can perpetuate and amplify biases, causing inability to differentiate between biased and unbiased generations. |
| Approach: | They conduct a systematic study of gender biases in image captioning tasks . they show that pretrained models perpetuate and amplify biase . |
| Outcome: | The proposed model-based evaluation metrics have shown good correlations with human judgments in language generation tasks. |
Masked Path Modeling for Vision-and-Language Navigation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | A major challenge in vision-and-language navigation is the limited available training data, which hinders the models’ ability to generalize effectively. |
| Approach: | They propose a masked path modeling objective that pretrains an agent using self-collected data for subsequent navigation tasks. |
| Outcome: | The proposed model pretrains an agent using self-collected data for subsequent navigation tasks eliminating the need for external tools. |
ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos (2023.emnlp-main)
Copied to clipboard
Te-Lin Wu, Zi-Yi Dou, Qingyuan Hu, Yu Hou, Nischal Chandra, Marjorie Freedman, Ralph Weischedel, Nanyun Peng
| Challenge: | despite its importance, there are few datasets that cover multimodal counterfactual reasoning . a dataset focusing on this area is limited because of its limited coverage over synthetic environments . |
| Approach: | They develop a video question answering dataset that provides questions on multimodal reasoning . they ask questions about counterfactual hypotheses over visual events . |
| Outcome: | The proposed dataset shows a significant performance gap between models and humans . it provides questions that span physical, social, and temporal dimensions . |
VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods focus on object hallucinations, focusing on object outputs . current evaluation methods struggle to address subtle semantic distinctions between outputs and reference data . |
| Approach: | They propose a multi-dimensional benchmark covering objects, attributes, and relations . they propose metric that generalizes CHAIR metric and incorporates faithfulness and coverage . |
| Outcome: | The proposed evaluation framework is more comprehensive and better correlated with humans than existing evaluation methods. |
RefSum: Refactoring Neural Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for text summarization are limited by limitations of reranking or stacking. |
| Approach: | They propose a framework that provides a unified view of text summarization and summaries combination. |
| Outcome: | The proposed method can be used by researchers as an off-the-shelf tool to achieve further performance improvements. |
CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation methods for text summarization systems are limited to in-domain setting, where supervised pre-trained models are evaluated on the same dataset. |
| Approach: | They propose to use a cross-dataset evaluation approach to evaluate different summarization systems in a multi-domain setting. |
| Outcome: | The proposed model can be used to evaluate text summarization systems on different datasets. |
Dynamic Oracle for Neural Machine Translation in Decoding Phase (L18-1)
Copied to clipboard
| Challenge: | Existing methods to improve NMT performance but there is a discrepancy between training and inference when decoding. |
| Approach: | They propose to use Scheduled Sampling to reduce the discrepancy between training and inference in NMT when decoding to mitigate the discrépancy. |
| Outcome: | The proposed methods improve translation quality over standard NMT system. |
GSum: A General Framework for Guided Neural Abstractive Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Abstractive summarization models are flexible, but they can be difficult to control. |
| Approach: | They propose a general and extensible guided summarization framework that takes different kinds of guidance as input and perform experiments across different varieties. |
| Outcome: | The proposed framework can generate more faithful summaries and different types of guidance generate qualitatively different summary. |
Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks (D19-1)
Copied to clipboard
| Challenge: | Existing methods to learn general representations of text can achieve sub-optimal performance in low-resource scenarios. |
| Approach: | They propose to use language model pre-training and multi-task learning to learn robust representations but these methods can achieve sub-optimal performance in low-resource scenarios. |
| Outcome: | The proposed model outperforms strong baselines on the GLUE benchmark and can be adapted to new tasks efficiently and effectively. |
Information Aggregation for Multi-Head Attention with Routing-by-Agreement (N19-1)
Copied to clipboard
| Challenge: | Existing studies focus on extracting informative or distinct partial-representations from different subspaces, while few studies have paid attention to the aggregation of the extracted partial-Representations. |
| Approach: | They propose to use a routing-by-agreement algorithm to improve multi-head attention by iteratively updating the proportion of how much a part should be assigned to a whole based on agreement between parts and wholes. |
| Outcome: | The proposed algorithm improves the information aggregation for multi-head attention over the standard linear transformation on linguistic probing and machine translation tasks. |
Domain Differential Adaptation for Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Neural networks are data hungry and domain sensitive, so it is difficult to obtain labeled data for every domain. |
| Approach: | They propose a framework for domain adaptation where we model the difference between domains instead of smoothing over them. |
| Outcome: | The proposed framework improves on domain adaptation in multiple experimental settings. |
compare-mt: A Tool for Holistic Comparison of Language Generation Systems (N19-4)
Copied to clipboard
| Challenge: | Unlike machine translation, natural language outputs are nuanced and there are no clear yes/no distinctions about whether they are correct or not. |
| Approach: | They describe compare-mt, a tool for holistic analysis and comparison of the results of systems for language generation tasks such as machine translation. |
| Outcome: | The compare-mt tool is an open-source pure-python package that has already proven useful to generate analyses that have been used in our papers. |
Re-ReST: Reflection-Reinforced Self-Training for Language Agents (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to fine tune language agents with reasoning-action trajectories require high-quality model-generated samples, which are hard to obtain for challenging language agent tasks. |
| Approach: | They propose a method to employ reflection during inference without ground-truth feedback to improve agents more autonomously. |
| Outcome: | The proposed method improves self-training performance on open-source language agents by 7.6% and 14.1% respectively. |
Dynamic Data Selection and Weighting for Iterative Back-Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results demonstrate that our methods achieve improvements of up to 1.8 BLEU points over competitive baselines. |
| Approach: | They propose a data selection and weighting strategy to iterate back-translation models and apply it to it . they use a target language to back-transcribe monolingual data, which is of high quality and reflect the target domain. |
| Outcome: | The proposed approach achieves 1.8 BLEU points over baselines on domain adaptation, low-resource, and high-resourced MT settings and on two language pairs. |
Improving Pre-trained Vision-and-Language Embeddings for Phrase Grounding (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the phrase grounding ability of pretrained vision-and-language models, but it is unclear how they can be used for phrase ground. |
| Approach: | They propose to extract phrase-region pairs from pre-trained vision-and-language embeddings and propose four fine-tuning objectives to improve model phrase grounding ability using image-caption data without any supervised grounding signals. |
| Outcome: | The proposed model outperforms baseline models in weakly-supervised and supervised phrase grounding settings on two representative datasets and shows that it is possible to achieve better phrase groundability without sacrificing representation generality. |
Unsupervised Domain Adaptation for Neural Machine Translation with Domain-Aware Feature Embeddings (D19-1)
Copied to clipboard
| Challenge: | Recent studies have focused on domain adaptation for neural machine translation systems where in-domain data is scarce or nonexistent. |
| Approach: | They propose an approach that adapts models with domain-aware feature embeddings, which are learned via an auxiliary language modeling task. |
| Outcome: | The proposed model performs better in multiple experimental settings and with back translation. |
Medical Vision-Language Pre-Training for Brain Abnormalities (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing vision-language models lack expertise for medical applications due to the scarcity and complexity of data. |
| Approach: | They propose a pipeline to collect medical image-text aligned data for pretraining from public resources such as PubMed and build a high-performance vision-language model tailored to specific medical tasks. |
| Outcome: | The proposed model is based on a large brain image-text dataset and will be released to the public. |
Uncertainty Calibration for Tool-Using Language Agents (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language agents are increasingly used to perform tasks and interact with a variety of external tools to achieve specific, goal-oriented objectives. |
| Approach: | They propose a tool calibration tool called ProbeCal which recalibrates the internal probabilities of tool-using language agents to better reflect the actual effectiveness of tool. |
| Outcome: | The proposed model significantly improves off-the-shelf language models in tool-using applications. |
Exploiting Deep Representations for Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Neural machine translation models typically implement encoder and decoder as multiple layers, but only the top layers are leveraged in the subsequent process, which misses the opportunity to exploit useful information embedded in other layers. |
| Approach: | They propose to expose all of these signals with layer aggregation and multi-layer attention mechanisms and introduce an auxiliary regularization term to encourage different layers to capture diverse information. |
| Outcome: | The proposed approach exposes all of these signals with layer aggregation and multi-layer attention mechanisms on widely-used translation datasets. |
Unsupervised Bilingual Lexicon Induction via Latent Variable Models (D18-1)
Copied to clipboard
| Challenge: | Existing methods for generating bilingual lexicons use parallel corpora or bilingual dictionaries. |
| Approach: | They propose a model which builds cross-lingual dictionaries using latent variable models and adversarial training with no parallel corpora. |
| Outcome: | The proposed model outperforms state-of-the-art models on several language pairs and reaches competitive performance. |