Papers by Michael Sheng
LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement (2024.findings-acl)
Copied to clipboard
Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipalli, Michael Mahoney, Kurt Keutzer, Amir Gholami
| Challenge: | Pretrained large language models are currently state-of-the-art for solving most tasks . however, many of them are in the low-data regime, making fine-tuning challenging . a new data augmentation strategy uses a teacher LLM to augment a small seed dataset . |
| Approach: | They propose a targeted and iterative data augmentation strategy that augments a teacher LLM to fine-tune a small seed dataset by adding additional data. |
| Outcome: | The proposed approach outperforms fine-tuning and other data augmentation strategies on a small seed dataset. |
Commonsense Inference in Natural Language Processing (COIN) - Shared Task Report (D19-60)
Copied to clipboard
| Challenge: | The workshop on Commonsense Inference in NLP (COIN) evaluated text understanding systems' ability to draw inferences about facts that are not mentioned in the text, but that are assumed to be common ground. |
| Approach: | They propose to use commonsense knowledge to evaluate systems' ability to answer questions/queries about a text. |
| Outcome: | The proposed tasks evaluated systems in two contexts: Commonsense Inference and Commonsensible Inference. |
Hyperbolic Geometry is Not Necessary: Lightweight Euclidean-Based Models for Low-Dimensional Knowledge Graph Embeddings (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent knowledge graph embedding models based on hyperbolic geometry are complicated than Euclidean operations. |
| Approach: | They propose to use hyperbolic geometry to generate high-fidelity and parsimonious representations of hierarchical patterns in knowledge graphs. |
| Outcome: | The proposed models achieve state-of-the-art performance on two widely-used datasets and cost less than RotH. |
What’s Hidden in a One-layer Randomly Weighted Transformer? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing subnetworks of one-layer randomly weighted neural networks can achieve impressive performance without changing initializations. |
| Approach: | They find subnetworks within one-layer randomly weighted neural networks that can achieve impressive performance without ever modifying the initializations. |
| Outcome: | The proposed subnetworks match 98%/92% of the performance of a trained Transformersmall/base on IWSLT14/WMT14. |
Reservoir Transformers (2021.acl-long)
Copied to clipboard
| Challenge: | Using random initialization, we show that some transformers obtain impressive performance even when some of the layers are frozen. |
| Approach: | They propose to freeze transformer layers and use them to improve performance . they find that the transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. |
| Outcome: | The proposed model improves on translation and language modelling tasks even when some layers are frozen. |
MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on phrase localization uses caption-image datasets as weak supervision . existing work on supervised phrase localisation uses a large-scale annotated dataset . |
| Approach: | They develop a multimodal alignment framework to leverage more widely available caption-image datasets to model phrase relevance. |
| Outcome: | The proposed model improves on the widely-adopted Flickr30k dataset . it also improves the previous best unsupervised result by 5.56% . |