Papers by Xinying Song
Token Dropping for Efficient BERT Pretraining (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to accelerate pretraining of transformer-based models are computationally expensive and degrade performance on downstream tasks. |
| Approach: | They propose a "token dropping" method to accelerate the pretraining of transformer-based models by 25% . they leverage the already built-in masked language modeling loss to identify unimportant tokens with practically no computational overhead. |
| Outcome: | The proposed method reduces the pretraining cost of BERT models by 25% while achieving similar overall performance on downstream tasks. |
Bring Invariant to Variant: A Contrastive Prompt-based Framework for Temporal Knowledge Graph Forecasting (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for temporal knowledge graph forecasting are insufficient structural contexts to learn effective representations. |
| Approach: | They propose a Contrastive Prompt-based framework with Entity background information for TKG forecasting that brings time-invariant entity background information to time-variant structural information. |
| Outcome: | The proposed framework is effective and stays competitive in inference with limited structural information. |
Fast WordPiece Tokenization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for tokenization of text are not efficient, but they are based on Aho-Corasick's algorithm. |
| Approach: | They propose an efficient algorithm for WordPiece tokenization using a longest-match-first strategy . they propose an algorithm whose tokenization complexity is strictly O(n) |
| Outcome: | The proposed method is 8.2x faster than HuggingFace Tokenizers and 5.1x faster on average for general text tokenization. |
TimeR4 : Time-aware Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Temporal Knowledge Graph Question Answering (TKGQA) aims to answer temporal questions using knowledge in Temporal knowledge graphs (TKTs). |
| Approach: | They propose a Time-aware retrieve-rewrite-retrieve-rerank framework to integrate temporal knowledge from TKGs into Large Language Models (LLMs) to reduce temporal hallucination, they propose rewrite module to rew questions using background knowledge stored in TKG's, then implement a retrieve-rank module to retrieve semantically and temporally relevant facts from Tkgs and rerank them according to temporal constraints. |
| Outcome: | The proposed approach achieves relative gains of 47.8% and 22.5% on two datasets, underscoring its effectiveness in boosting the temporal reasoning abilities of LLMs. |
An Efficient Conversational Smart Compose System (2023.acl-demo)
Copied to clipboard
Yun Zhu, Xiayu Chen, Lei Shu, Bowen Tan, Xinying Song, Lijuan Liu, Maria Wang, Jindong Chen, Ning Ruan
| Challenge: | a cloud-based smart compose system is designed to improve human-to-human conversation efficiency. |
| Approach: | They propose a cloud-based smart compose system to improve conversation efficiency . they propose heuristics to achieve the best trade-off between quality and latency . |
| Outcome: | The proposed system reduces latency without losing composing quality further. |