Papers by Fei Sha
A Probabilistic Model for Joint Learning of Word Embeddings from Texts and Images (D18-1)
Copied to clipboard
| Challenge: | Existing approaches combine language and perception to infer word embeddings . however, the embeddables produced by such models do not reflect the actual word representations. |
| Approach: | They propose a probabilistic model that integrates linguistic and perceptual inputs to explain observed word-context pairs in a text corpus. |
| Outcome: | The proposed model achieves competitive or stronger results on tasks of assessing pairwise word similarity and image/caption retrieval compared to other state-of-the-art models. |
Learning to Represent Image and Text with Denotation Graph (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in learning representations of visual and language information have been a problem with many applications. |
| Approach: | They propose to extract visual expressions from images aligned with linguistic expressions that describe the images to learn representations from implicit expressions. |
| Outcome: | The proposed representations lead to stronger empirical results on downstream tasks of cross-modal image retrieval, referring expression, and compositional attribute-object recognition. |
A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Psychologists have documented several ways in which humans’ inferences deviate from the rules of logic. |
| Approach: | They focus on syllogisms, which are inferences from two simple premises, and show that larger models are more logical than smaller ones. |
| Outcome: | The results show that language models often mimic human biases, but overcome them in some cases. |
ReadTwice: Reading Very Large Documents with Memories (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to model long-range dependencies in text are limited to 512 tokens . however, the amount of compute in attention depends quadratically on the number of tokens in an input text passage. |
| Approach: | They propose a technique that summarises text into a memory table to be used in a second read of the text. |
| Outcome: | The proposed method outperforms models of comparable size on several question answering datasets and sets a new state of the art on the NarrativeQA task, with questions about entire books. |
Multi-Task Learning for Sequence Tagging: An Empirical Study (C18-1)
Copied to clipboard
| Challenge: | Existing work on "pairwise" MTL has been validated in sequence tagging but key issues remain about its effectiveness. |
| Approach: | They propose three general multi-task learning approaches on 11 sequence tagging tasks. |
| Outcome: | The proposed approaches improve on 11 sequence tagging tasks. |
Generate-and-Retrieve: Use Your Predictions to Improve Retrieval for Semantic Parsing (2022.coling-1)
Copied to clipboard
Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie, Panupong Pasupat, Peter Shaw, Linlu Qiu, Sumit Sanghai, Fei Sha
| Challenge: | Existing retrieval techniques for semantic parsing use similarity of query and exemplar inputs . Existing work suggests that appending training samples to training samples improves performance . |
| Approach: | They propose a retrieval procedure that retrieves exemplars for which outputs are similar . existing retrieval techniques are based on similarity of query and exemplar inputs . |
| Outcome: | Existing retrieval techniques rely on similarity of query and exemplar inputs . they retrieve exemplars with similar outputs and generate a final prediction . |
Systematic Generalization on gSCAN: What is Nearly Solved and What is Next? (2021.emnlp-main)
Copied to clipboard
| Challenge: | a general-purpose Transformer-based model with crossmodal attention solves most of the systematic generalization problems . current models are data inefficient given the narrow scope of commands in gSCAN . |
| Approach: | They propose to use a Transformer-based model with cross-modal attention to solve gSCAN . they propose to generate data to incorporate relations between objects in the visual environment . |
| Outcome: | The proposed model outperforms specialized approaches on most splits, and is data inefficient given the narrow scope of commands. |
BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps (2020.acl-main)
Copied to clipboard
| Challenge: | Existing state-of-the-art VLN agents do not generalize well for long navigation tasks. |
| Approach: | They propose a VLN agent that is learned to navigate by decomposing long instructions into shorter ones and completing them sequentially. |
| Outcome: | The proposed agent can follow long instructions better than existing ones, but it does not generalize well. |
DOCENT: Learning Self-Supervised Entity Representations from Large Document Collections (2021.eacl-main)
Copied to clipboard
Yury Zemlyanskiy, Sudeep Gandhe, Ruining He, Bhargav Kanagal, Anirudh Ravula, Juraj Gottweis, Fei Sha, Ilya Eckstein
| Challenge: | Using pre-trained models, we learn to jointly predict words and entities from multiple text sources without any human supervision. |
| Approach: | They propose to learn rich self-supervised entity representations from large amounts of associated text. |
| Outcome: | The proposed models outperform baseline models on downstream tasks in the TV-Movies domain, and scale to very large corpora. |
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022.emnlp-main)
Copied to clipboard
Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, Kristina Toutanova
| Challenge: | Pre-trained language models struggle on out-of-distribution compositional generalization . recent work shows considerable improvements on many NLP tasks from model scaling . |
| Approach: | They evaluate encoder-decoder models up to 11B parameters and decoder-only models up 540B parameters . they compare scaling curves for fine-tuning, prompt tuning, and in-context learning methods . |
| Outcome: | The proposed scaling methods improve compositional generalization on many tasks . fine-tuning generally has flat or negative scaling curves on out-of-distribution compositional . larger models are better at modeling the syntax of the output space, the study finds . |
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference (2023.findings-acl)
Copied to clipboard
Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit Sanghai, Fei Sha, William Cohen
| Challenge: | Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model . however, the architecture used for FiD was not designed for retrieval augmented models . |
| Approach: | They propose to make FiD a modified retrieval-augmented language model with a large decoder and memory bandwidth constraints to alleviate memory bandwidth limitations. |
| Outcome: | The proposed architecture outperforms existing models on knowledge-intensive tasks even on large models on many knowledge-based tasks. |
Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets (N18-1)
Copied to clipboard
| Challenge: | Visual question answering datasets are a form of (visual) Turing test that artificial intelligence should strive to achieve. |
| Approach: | They propose automatic procedures to remedy design deficiencies in visual question answering datasets . they propose to use a set of decoys to re-construct decoying answers for two popular Visual QA datasets. |
| Outcome: | The proposed procedures improve the performance of the proposed datasets. |
The Impact of Depth on Compositional Generalization in Transformer Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | In this paper, we test the hypothesis that deeper transformers generalize more compositionally. |
| Approach: | They propose to add layers to transformers to generalize more compositionally . they propose to fine-tune the models so that the total number of parameters is constant . |
| Outcome: | The proposed model generalizes more compositionally than shallower models, but returns diminish . the proposed model can be made shallower without sacrificing performance . |
Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs (2025.emnlp-main)
Copied to clipboard
Hexiang Tan, Fei Sun, Sha Liu, Du Su, Qi Cao, Xin Chen, Jingang Wang, Xunliang Cai, Yuanzhuo Wang, Huawei Shen, Xueqi Cheng
| Challenge: | Existing detection methods fail to account for **self-consistent error** . study identifies self-consistency errors and evaluates them . |
| Approach: | They propose a method that fuses hidden state evidence from an external verifier LLM to detect self-consistent errors. |
| Outcome: | The proposed method significantly enhances performance on self-consistent errors across three LLM families. |
Visually Grounded Concept Composition (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to visual grounding do not explicitly model compositional structures of text expressions. |
| Approach: | They propose a concept-relation Graph and a composition neural network to combine CRGs . they propose to align CRG-based concepts with images to learn visually grounded concepts . |
| Outcome: | The proposed model can model grounded concepts forming at sentence level and word level. |
Graders Should Cheat: Privileged Information Enables Expert-Level Automated Evaluations (2025.emnlp-main)
Copied to clipboard
| Challenge: | a lack of trust in graders on graduate-level physics and Olympiad-level math makes them unreliable grader. |
| Approach: | They propose to use a grader LM to evaluate the candidate LMs. |
| Outcome: | The proposed approach outperforms human graders on *RewardBench* and human expert grader on Olympiad-level math problems. |
Improving Compositional Generalization with Latent Structure and Data Augmentation (2022.naacl-main)
Copied to clipboard
| Challenge: | Generic unstructured neural networks struggle on out-of-distribution compositional generalization. |
| Approach: | They propose a method to recombinate examples from a model called Compositional Structure Learner and add them to a pre-trained sequence-to-sequence model. |
| Outcome: | The proposed model is even stronger than a T5-CSL ensemble on two real world compositional generalization tasks. |