Papers by Weijia Shi
Nonparametric Masked Language Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. |
| Approach: | They introduce a nonparametric masked language model that replaces a softmax with a distribution over every phrase in a reference corpus and uses an in-batch approximation to train it. |
| Outcome: | The proposed model outperforms larger parametric models on 16 tasks including classification, fact probing and question answering. |
DESCGEN: A Distantly Supervised Datasetfor Generating Entity Descriptions (2021.acl-long)
Copied to clipboard
| Challenge: | Short textual descriptions of entities provide summaries of their key attributes but generating entity descriptions can be challenging since information is scattered across multiple sources with varied content and style. |
| Approach: | They propose to generate an entity summary description from 37K entities from Wikipedia and Fandom, paired with nine evidence documents on average. |
| Outcome: | The proposed task is entity-centric, more abstractive, and covers a wide range of domains. |
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2024.naacl-short)
Copied to clipboard
| Challenge: | Language models (LMs) often struggle to pay enough attention to the input context, and generate texts that are unfaithful or contain hallucinations. |
| Approach: | They propose a context-aware decoding technique that amplifies the difference between the output probabilities when a model is used with and without context. |
| Outcome: | The proposed model significantly improves faithfulness of different LM families including OPT, GPT, LLaMA, and FLAN-T5 for summarization tasks. |
Design Challenges in Low-resource Cross-lingual Entity Linking (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing techniques for grounding mentions of entities in a foreign language do not rise to the challenges introduced by text in low-resource languages (LRL) and fail to generalize to text not taken from Wikipedia, on which they are usually trained. |
| Approach: | They propose a cross-lingual XEL technique that uses search engines to locate and search for foreign language entries in Wikipedia. |
| Outcome: | The proposed system shows an increase of 25% in gold candidate recall and 13% in end-to-end linking accuracy over state-of-the-art baselines. |
Data Swarms: Optimizable Generation of Synthetic Evaluation Data (2026.findings-acl)
Copied to clipboard
| Challenge: | Extensive experiments demonstrate that Data Swarms outperforms eight data generation baselines across five evaluation objectives. |
| Approach: | They propose an algorithm to optimize the generation of synthetic evaluation data and advance quantitative desiderata of LLM evaluation. |
| Outcome: | The proposed algorithm outperforms baseline evaluations and Adversarial Swarms generates harder data while learning from such data. |
s1: Simple test-time scaling (2025.emnlp-main)
Copied to clipboard
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candes, Tatsunori Hashimoto
| Challenge: | OpenAI’s o1 model showed this capability but did not publicly share its methodology, leading to many replication efforts. |
| Approach: | They curate a small dataset s1K with 1,000 reasoning questions based on three criteria we validate through ablations: difficulty, diversity, and quality. |
| Outcome: | The proposed model exceeds o1-preview on competition math questions by up to 27% (MATH and AIME24). |
REPLUG: Retrieval-Augmented Black-Box Language Models (2024.naacl-long)
Copied to clipboard
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, Wen-tau Yih
| Challenge: | Existing retrieval-augmented language models require access to internal representations to enhance performance. |
| Approach: | They introduce a retrieval-augmented language modeling framework that treats the language model as a black box and augments it with a tuneable retrieval model. |
| Outcome: | The proposed framework improves performance on language modeling tasks by 6.3% and 5.1%. |
Nearest Neighbor Zero-Shot Inference (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using non-parametric memory for retrieval-augmented language models yields significant performance boosts over strong zeroshot baselines. |
| Approach: | They propose a retrieval-augmented language model with fuzzy verbalizers that expands the verbalizes that define different end-task class labels. |
| Outcome: | The proposed model outperforms non-retrieval-augmented language models on perplexity-based evaluations but gains transfer marginally . the main challenge is to achieve coverage of the verbalizer tokens that define the different end-task class labels. |
One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2023.findings-acl)
Copied to clipboard
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, Tao Yu
| Challenge: | a new method for embedding text is developed for tasks that require specialized encoders . INSTRUCTOR is a single embedder that can generate text embeddables tailored to different tasks and domains based on instruction finetuning . |
| Approach: | They introduce a new method for computing text embeddings given task instructions . they first annotate instructions for 330 diverse tasks and train INSTRUCTOR on this multitask mixture . |
| Outcome: | The proposed method improves on 70 embedding evaluation tasks with fewer parameters than the previous best model. |
Instruction-tuned Language Models are Better Knowledge Learners (2024.acl-long)
Copied to clipboard
Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Lin, Wen-tau Yih, Srini Iyer
| Challenge: | Large language models store factual knowledge in parameters, but it can become outdated as the work evolves . pre-instruction-tuning improves ability of LLMs to absorb knowledge from new documents . |
| Approach: | They propose a method that instruction-tunes on questions prior to training on documents . they propose to use QA pairs to update factual knowledge of large language models . |
| Outcome: | The proposed method outperforms instruction-tuning on documents by 17.8%. |
Getting MoRE out of Mixture of Language Model Reasoning Experts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) have poor generalizability on question types beyond those seen in the prompt. |
| Approach: | They propose a framework that integrates specialized language models to generalize across question types that require distinct reasoning abilities. |
| Outcome: | The proposed framework gives higher accuracy than any single specialized model on a collection of 12 QA datasets from four reasoning types. |
Examining Gender Bias in Languages with Grammatical Gender (D19-1)
Copied to clipboard
| Challenge: | Existing studies on gender bias in word embeddings focus on English . however, these studies cannot be extended to languages with morphological agreement on gender . |
| Approach: | They propose new metrics to evaluate gender bias in word embeddings of English and Spanish . they extend existing approaches to mitigate gender bias while preserving original embeddables . |
| Outcome: | The proposed methods reduce gender bias while preserving the original embeddings. |
Retrofitting Contextualized Word Embeddings with Paraphrases (D19-1)
Copied to clipboard
| Challenge: | Contextualized word embeddings can be useful for downstream applications, but they can be over-sensitive to contexts. |
| Approach: | They propose a method to retrofit contextualized word embeddings with paraphrases to minimize the variance of word representations on paraphrased contexts. |
| Outcome: | The proposed method improves on sentence classification and inference tasks. |
When One LLM Drools, Multi-LLM Collaboration Rules (2026.acl-long)
Copied to clipboard
Shangbin Feng, Wenxuan Ding, Alisa Liu, Zifeng Wang, Weijia Shi, Yike Wang, Shannon Zejiang Shen, Xiaochuang Han, Hunter Lang, Chen-Yu Lee, Tomas Pfister, Yejin Choi, Yulia Tsvetkov
| Challenge: | a single general-purpose LLM is not enough to produce a reliable output, argues this paper . a multi-LLM collaboration approach addresses reliability, democratization, and pluralism . |
| Approach: | They argue that a single general-purpose LLM is not enough to produce a reliable output . they organize existing multi-LLM collaboration methods into a hierarchy based on access and information exchange . |
| Outcome: | The proposed method addresses reliability, democratization, and pluralism challenges a single LLM fails to produce a reliable output. |
Teaching LLMs to Abstain across Languages via Multilingual Feedback (2024.emnlp-main)
Copied to clipboard
Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Orevaoghene Ahia, Shuyue Stella Li, Vidhisha Balachandran, Sunayana Sitaram, Yulia Tsvetkov
| Challenge: | Existing studies on LLM abstention focus on English, but they show that it can reduce the accuracy of the model by 20.5% . |
| Approach: | They propose to teach LLMs to abstain in the face of knowledge gaps by generating multiple feedback items in related languages. |
| Outcome: | Extensive experiments show that the proposed approach outperforms baselines and achieves 9.2% improvement for low-resource languages. |
Toward Human Readable Prompt Tuning: Kubrick’s The Shining is a good movie, and a good prompt too? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models can perform downstream tasks in a zero-shot fashion, given natural language prompts that specify the desired behavior. |
| Approach: | They propose a human readable prompt tuning method that incorporates a fluency constraint to find a distribution of effective and fluent prompts. |
| Outcome: | The proposed method outperforms baselines by 7.0% across three tasks. |
RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing QA models are not robust to variations in question constraints, but can be made more robust by tuning on clusters of related questions. |
| Approach: | They introduce RoMQA, the first benchmark for robust, multi-evidence, multianswer question answering (QA) RoMQ contains clusters of related questions that are derived from the Wikidata knowledge graph . |
| Outcome: | The proposed model is the first benchmark for robust, multi-evidence, multianswer question answering (QA) compared to prior QA datasets, it has more human-written questions that require reasoning over more evidence text and have, on average, many more correct answers. |
Cross-lingual Entity Alignment with Incidental Supervision (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods to match entities in multilingual knowledge graphs are insufficient, resulting in inconsistent seed alignment between KGs. |
| Approach: | They propose a model that integrates multilingual KGs and monolingual text corpora in a shared embedding scheme and a self-learning based alignment learning process to induce correspondence between entities and lexemes. |
| Outcome: | The proposed model significantly outperforms state-of-the-art methods on benchmark datasets and significantly outpersts existing methods. |