Papers by Zexuan Zhong
Structured Pruning Learns Compact and Accurate Models (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models have high costs in terms of storage, memory, and computation time. |
| Approach: | They propose a task-specific structured pruning method CoFi which provides highly parallelizable subnetworks and matches distillation methods in both accuracy and latency. |
| Outcome: | The proposed method matches the distillation methods in accuracy and latency without resorting to unlabeled data. |
Should You Mask 15% in Masked Language Modeling? (2023.eacl-main)
Copied to clipboard
| Challenge: | Masked language models (MLMs) traditionally mask 15% of tokens due to the belief that more masking would leave insufficient context to learn good representations. |
| Approach: | They revisit the 15% masking rate of MLMs to examine the role of masking in linguistic training. |
| Outcome: | The proposed masking rate outperforms BERT-large size models on GLUE and SQUAD while maintaining 95% accuracy. |
Simple Entity-Centric Questions Challenge Dense Retrievers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Open-domain question answering has exploded in popularity due to the success of dense retrieval models. |
| Approach: | They construct a set of simple, entity-rich questions based on facts from Wikidata and test their models against supervised datasets. |
| Outcome: | The proposed model outperforms sparse retrieval methods on open-domain question answering datasets by a large margin. |
MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for retraining from scratch are limited and only work on the recall of edited facts. |
| Approach: | They propose a benchmark method that allows users to ask multi-hop questions to assess whether edited models correctly answer questions where the answer should change as an entailed consequence of edited facts. |
| Outcome: | The proposed method outperforms existing models and scales well with LLMs (up to 175B) it is based on a memory-based approach that stores all edited facts externally while prompting the language model iteratively to generate answers consistent with the edited facts. |
REST: Retrieval-Based Speculative Decoding (2024.naacl-long)
Copied to clipboard
| Challenge: | Retrieval-based speculative decoding (REST) is a new language model generation algorithm . it uses existing knowledge to generate draft tokens, allowing for seamless integration and acceleration of any language model. |
| Approach: | They propose a new algorithm that uses a draft language model to generate tokens from existing knowledge. |
| Outcome: | The proposed method achieves a speedup of 1.62 to 2.36 on code or text generation. |
Retrieval-based Language Models and Applications (2023.acl-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we will provide a comprehensive overview of retrieval-based language models. |
| Approach: | This tutorial will provide a comprehensive overview of recent advances in retrieval-based language models. |
| Outcome: | This tutorial will provide a comprehensive overview of recent advances in retrieval-based language models. |
Poisoning Retrieval Corpora by Injecting Adversarial Passages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Dense retrievers have outperformed traditional lexical methods in a range of information retrieval tasks, but to what extent can they be safely deployed in real-world applications? |
| Approach: | They propose a method where a malicious user injects a small number of adversarial passages into a retrieval corpus to maximize similarity with a set of training queries. |
| Outcome: | The proposed attack fools retrieval systems into returning top results for queries not seen by the attacker. |
Privacy Implications of Retrieval-Based Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | a study of retrieval-based language models shows improved interpretability, factuality, and adaptability compared to parametric counterparts . kNN-LMs are more susceptible to leaking private information from their private datastore than parametric models . |
| Approach: | They present the first study of privacy risks in retrieval-based language models . they aim to strike a balance between utility and privacy in domains where privacy is of concern . |
| Outcome: | The proposed methods improve interpretability, factuality, and adaptability compared to parametric models . the study finds that kNN-LMs are more susceptible to leaking private data than parametric ones . |
CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Developing long-context LLMs with robust long-text capabilities is underdeveloped due to a lack of benchmarks. |
| Approach: | They propose a Chinese benchmark for evaluating long-context LLMs with Chinese capabilities. |
| Outcome: | The proposed model is based on 6 open-source LLMs and 2 commercial ones. |
Training Language Models with Memory Augmentation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for training memory-augmented language models only introduce mem-ories at testing time or represent them using a separately trained encoder. |
| Approach: | They propose a training approach that directly takes in-batch examples as accessible memory and new methods for memory construction and data batching that are used for adapting to different sets of memories at testing time. |
| Outcome: | The proposed approach reduces perplexity from 18.70 to 15.37 on multiple language modeling and machine translation benchmarks. |
A Frustratingly Easy Approach for Entity and Relation Extraction (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing work on end-to-end relation extraction models combine two tasks: named entity recognition and relation extraction. |
| Approach: | They propose a pipelined approach for entity and relation extraction that uses two independent encoders to construct the relation model. |
| Outcome: | The proposed approach achieves an 8.16 speedup with a slight reduction in accuracy on standard benchmarks. |
SemRegex: A Semantics-Based Approach for Generating Regular Expressions from Natural Language Specifications (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to generate programs from natural language do not address program aliasing . semantically equivalent programs may have many syntactically different forms . |
| Approach: | They propose a semantics-based approach to generate regular expressions from natural language. |
| Outcome: | The proposed approach improves on three public datasets. |
Factual Probing Is [MASK]: Learning vs. Learning to Recall (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for factual probing can interpret the model’s prediction accuracy as a lower bound on the amount of factual information it encodes. |
| Approach: | They propose a method which directly optimizes in continuous embedding space and can predict an additional 6.4% of facts in the LAMA benchmark. |
| Outcome: | The proposed method outperforms the best previous prompt method by 6.4% on the LAMA benchmark. |