Papers by Zhuowen Tu
DocKD: Knowledge Distillation from LLMs for Open-World Document Understanding Models (2024.emnlp-main)
Copied to clipboard
Sungnyun Kim, Haofu Liao, Srikar Appalaraju, Peng Tang, Zhuowen Tu, Ravi Satzoda, R. Manmatha, Vijay Mahadevan, Stefano Soatto
| Challenge: | Existing methods for visual document understanding are limited by training on a small-scale, curated document dataset, compromising generalizability of VDU models to diverse documents. |
| Approach: | They propose a framework that integrates external document knowledge into the data generation process. |
| Outcome: | The proposed framework produces high-quality annotations and surpasses direct knowledge distillation approach. |
When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual language models are widely used to extend NLP systems to low-resource languages. |
| Approach: | They pre-train over 10,000 monolingual and multilingual language models for over 250 languages including multiple language families that are under-studied in NLP. |
| Outcome: | The results show that adding multilingual data improves low-resource language modeling performance, similar to increasing low-source dataset sizes by up to 33%. |
Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that convolutions have been successful in natural language learning. |
| Approach: | They propose a convolutional approach to construct relative position embeddings in self-attention layers and propose 'compact attention' they propose multiple ways to integrate convolutions into Transformer self- attention. |
| Outcome: | The proposed composite attention improves performance on multiple downstream tasks, replacing absolute position embeddings, and is more expressive than convolutions in NLP. |
The Geometry of Multilingual Language Model Representations (2022.emnlp-main)
Copied to clipboard
| Challenge: | XLM-R models encode language-sensitive information in each language, allowing them to extract features for downstream tasks and cross-lingual transfer learning. |
| Approach: | They evaluate how multilingual language models maintain a shared multilingual representation space while still encoding language-sensitive information in each language. |
| Outcome: | The proposed model can extract features for downstream tasks and cross-lingual transfer learning. |