Papers by Shuhuai Ren
Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification (2021.emnlp-main)
Copied to clipboard
| Challenge: | Data augmentation aims to alleviate the overfitting issue in low-resource or class-imbalanced situations. |
| Approach: | They propose a framework called Text AutoAugment to enhance training samples . they use a Bayesian optimization algorithm to search for the best policy . |
| Outcome: | The proposed framework outperforms baseline methods on six benchmark datasets. |
LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation? (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing models for text-to-image generation have been underperforming in image-totext generation tasks. |
| Approach: | They propose a framework that uses a split BERT to create a dedicated latent space for captions and integrates a regularization module to manage varying text lengths. |
| Outcome: | The proposed framework achieves state-of-the-art performance on the MS COCO dataset with 38.2 BLEU@4 and 126.2 CIDEr . |
TempCompass: Do Video LLMs Really Understand Videos? (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks on video large language models lack a comprehensive feedback on temporal perception ability . current models cannot distinguish between different temporal aspects and are limited in task formats . |
| Approach: | They propose a benchmark to evaluate temporal perception ability of video large language models . they construct conflicting videos that share the same static content but differ in a specific temporal aspect . |
| Outcome: | The proposed benchmarks show that video large language models exhibit poor temporal perception ability. |
RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction (2025.emnlp-main)
Copied to clipboard
Yuchi Wang, Yishuo Cai, Shuhuai Ren, Sihan Yang, Linli Yao, Yuanxin Liu, Yuanxing Zhang, Pengfei Wan, Xu Sun
| Challenge: | Existing recaptioning methods suffer from inaccuracies due to missing fine-grained details. |
| Approach: | They propose a framework that refines captions through visual reconstruction using a text-to-image model and a visual reconstruction framework. |
| Outcome: | The proposed framework outperforms baselines on CapsBench and CompreCap by 10%. |
PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain (2024.findings-acl)
Copied to clipboard
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, Baobao Chang
| Challenge: | a new multimodal decision-making benchmark evaluates the integrated capabilities of multimodal large language models. |
| Approach: | They propose a multimodal decision-making benchmark for evaluating MLLMs . they propose an automatic evaluation protocol to assess 10 prevalent ML models . |
| Outcome: | The proposed benchmark improves performance of multimodal large language models in three scenarios . the model is required to integrate multiple capabilities to make accurate decisions . |
TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that TESTA reduces the number of visual tokens by 75% and thus accelerates video encoding. |
| Approach: | They propose a method to condense video semantics by aggregating similar frames and patches within each frame. |
| Outcome: | The proposed method reduces visual tokens by 75% and accelerates video encoding. |
Delving into the Openness of CLIP (2023.findings-acl)
Copied to clipboard
| Challenge: | Contrastive Language-Image Pre-training (CLIP) allows for open-vocabulary visual recognition, where the model can recognize images from an open class set in a zero-shot manner. |
| Approach: | They propose to use image classification as an image-to-text matching task instead of discrete category IDs to achieve open-vocabulary visual recognition. |
| Outcome: | The proposed model can recognize images from an open vocabulary in a zero-shot manner, but its performance deteriorates as the vocabulary expands. |
Learning Relation Alignment for Calibrated Cross-modal Retrieval (2021.acl-long)
Copied to clipboard
| Challenge: | despite advances in multimodal pre-training, cross-modal retrieval remains challenging . lack of relation consistency impairs contextualized representation of image-text pairs . |
| Approach: | They propose a new metric to quantify the relation consistency by measuring the semantic distance between linguistic and visual relations. |
| Outcome: | The proposed method boosts the performance of prevailing models on Flickr30k and MS COCO datasets by a considerable margin. |
Dynamic Knowledge Distillation for Pre-trained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods conduct knowledge distillation statically, e.g., student model aligns output distribution to teacher model on pre-defined training dataset. |
| Approach: | They propose a dynamic knowledge distillation that empowers the student to adjust the learning procedure according to its competency . they find it is promising and provide discussions on potential future directions towards more efficient methods . |
| Outcome: | The proposed method can boost student model performance while accelerating training . the proposed method reduces memory usage and accelerates model inference . |
Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to attack text classification are limited due to the complexity of the problem. |
| Approach: | They propose a greedy algorithm to generate adversarial examples that maintain lexical correctness, grammatical correctiness and semantic similarity. |
| Outcome: | The proposed algorithm maintains lexical correctness, grammatical correctity and semantic similarity well and is hard for humans to perceive. |
CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascade (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that CascadeBERT can achieve an overall 15% improvement under 4x speed-up compared with existing dynamic early exiting methods on six classification tasks. |
| Approach: | They propose a framework which emits predictions in internal layers without passing through the entire model. |
| Outcome: | The proposed framework can achieve 15% improvement under 4x speed-up compared with existing methods on six classification tasks yielding more calibrated and accurate predictions. |