Papers by Fuli Luo
Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency (2022.acl-long)
Copied to clipboard
| Challenge: | Structured pruning has been extensively studied on monolingual pre-trained models . but little attention has been paid to evaluating the effectiveness of structured pruning on multilingual models. |
| Approach: | They investigate settings, algorithms, and efficiency of structured pruning on multilingual models . authors propose a simple approach that allows training the model once and adapting to different model sizes at inference . |
| Outcome: | The proposed approach allows training the model once and adapting to different model sizes at inference. |
Cross-Modal Commentator: Automatic Machine Commenting Based on Cross-Modal Information (P19-1)
Copied to clipboard
| Challenge: | Existing work on commenting based on textual content is focused on other modalities, such as graphics and images. |
| Approach: | They propose a task to integrate multiple modalities into automatic commenting . they construct a large-scale dataset and propose 'co-attention' model to capture dependency between textual and visual information. |
| Outcome: | The proposed model can achieve better performance than baselines. |
Towards Fine-grained Text Sentiment Transfer (P19-1)
Copied to clipboard
| Challenge: | Existing methods for fine-grained text sentiment transfer only reverse the sentiment polarity of text, but they lack a robust and parallel learning algorithm. |
| Approach: | They propose a novel fine-grained text sentiment transfer task that revises a sequence to satisfy a given sentiment intensity while preserving the original semantic content. |
| Outcome: | The proposed model outperforms existing methods by a large margin in automatic evaluation and human evaluation. |
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent pretrained language models extend from millions to billions of parameters. |
| Approach: | They propose a technique which forwards on a whole network while backwarding on resetting the gradients of the non-child network during the backward process. |
| Outcome: | The proposed technique outperforms the vanilla fine-tuning technique on various downstream tasks and can achieve better generalization performance by large margins. |
Enhancing Topic-to-Essay Generation with External Commonsense Knowledge (P19-1)
Copied to clipboard
| Challenge: | Existing methods for topic-to-essay generation are insufficient for generating novel, diverse, and topic-consistent paragraph-level text with a set of topics. |
| Approach: | They propose to integrate commonsense from external knowledge base into the generator through dynamic memory mechanism and adversarial training to further improve topic-consistency. |
| Outcome: | The proposed task is more novel, diverse, and topic-consistent than existing methods in terms of both automatic and human evaluation. |
Towards Unified Prompt Tuning for Few-shot Text Classification (2022.findings-emnlp)
Copied to clipboard
Jianing Wang, Chengyu Wang, Fuli Luo, Chuanqi Tan, Minghui Qiu, Fei Yang, Qiuhui Shi, Songfang Huang, Ming Gao
| Challenge: | Prompt-based fine-tuning has boosted performance of Pre-trained Language Models (PLMs) on few-shot text classification, but PLMs are unfamiliar with prompt-style expressions during pre-training, which limits the few- shot learning performance on downstream tasks. |
| Approach: | They propose a framework for prompt-based fine-tuning that captures prompting semantics from non-target NLP datasets and propose 'Prompt-Options-Verbalizer' for joint prompt learning across different NLP tasks. |
| Outcome: | Experiments show that the proposed framework outperforms state-of-the-art prompt-based fine-tuning frameworks on few-shot text classification tasks. |
Towards Comprehensive Description Generation from Factual Attribute-value Tables (P19-1)
Copied to clipboard
| Challenge: | Existing models for comprehensive descriptions for factual attribute-value tables might suffer from missing key attributes and groundless information problems. |
| Approach: | They propose a force attention method to encourage the generator to pay more attention to uncovered attributes to avoid potential key attributes missing. |
| Outcome: | The proposed model outperforms the state-of-the-art baselines on automatic and human evaluation. |
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (2024.acl-long)
Copied to clipboard
Damai Dai, Chengqi Deng, Chenggang Zhao, R.x. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.k. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, Wenfeng Liang
| Challenge: | Mixture-of-Experts (MoE) architectures face challenges in ensuring expert specialization . despite the promising performance, scaling language models to an extremely large scale is associated with exceedingly high computational costs. |
| Approach: | They propose an architecture that allows for ultimate expert specialization by segmenting experts into mN ones and activating mK from them. |
| Outcome: | The proposed architecture achieves comparable performance with GShard with 2B parameters and computation. |
S4-Tuning: A Simple Cross-lingual Sub-network Tuning Method (2022.acl-short)
Copied to clipboard
| Challenge: | Existing multilingual pre-trained language models allow to adapt to target languages with only few labeled examples. |
| Approach: | They propose a simple cross-lingual sub-network tuning method that detects the most essential sub-netzwork for each target language and updates it during fine-tuning. |
| Outcome: | The proposed method improves on three multi-lingual tasks involving 37 different languages. |
Multi-Granularity Contrasting for Cross-Lingual Pre-Training (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to pre-training focus on embedding alignment, but they neglect the modeling of bidirectional contexts. |
| Approach: | They propose a framework to learn languageuniversal representations using multi-granularity contrasting framework . they encode semantic equivalents from different languages into similar representations . |
| Outcome: | The proposed framework can achieve significant performance gains in machine translation and cross-lingual language understanding. |
Pun-GAN: Generative Adversarial Network for Pun Generation (D19-1)
Copied to clipboard
| Challenge: | Existing methods for generating pun sentences with word senses lack large-scale corpus for supervised learning . a pun is a clever and amusing use of a word with two meanings (word senses) |
| Approach: | They propose an adversarial generative network for pun generation with a generator and a discriminator to distinguish between generated pun sentences and real sentences with specific word senses. |
| Outcome: | The proposed network generates sentences that are more ambiguous and diverse in both automatic and human evaluation. |
A Hierarchical Reinforced Sequence Operation Method for Unsupervised Text Style Transfer (P19-1)
Copied to clipboard
| Challenge: | Existing text style transfer methods face three challenges: 1) the transfer is weakly interpretable; 2) generated outputs struggle in content preservation; 3) the trade-off between content and style is intractable. |
| Approach: | They propose a hierarchical reinforced sequence operation method that proposes operation positions and alters the sentence. |
| Outcome: | The proposed method significantly outperforms existing methods on two text style transfer datasets. |
A Deep Reinforced Sequence-to-Set Model for Multi-Label Classification (P19-1)
Copied to clipboard
| Challenge: | Multi-label classification (MLC) aims to assign multiple labels to each sample. |
| Approach: | They propose a sequence-to-set model that is trained via reinforcement learning and rewards feedback independent of the label order. |
| Outcome: | The proposed model outperforms baseline models and reduces sensitivity to label order. |
Rethinking Denoised Auto-Encoding in Language Pre-Training (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models such as BERT have achieved success in learning sequence representations, but they tend to learn representations that are covariant with the noise of pre-training. |
| Approach: | They propose to train self-trained models to learn noise invariant sequence representations . they encourage consistency between original sequence and corrupted version via unsupervised instance-wise training signals. |
| Outcome: | The proposed model improves on 11 natural language understanding and cross-modal tasks and achieves 0.6% gain on GLUE benchmarks and 0.8% increment on NLVR2 . |
Learning to Control the Fine-grained Sentiment for Story Ending Generation (P19-1)
Copied to clipboard
| Challenge: | Existing studies focus on controlling the sentiment of story endings. |
| Approach: | They propose a generic and novel framework which controls fine-grained sentiment intensity for automatic story ending generation without manually annotating sentiment labels. |
| Outcome: | The proposed framework can generate story endings which meet the given sentiment intensity better. |
MAAM: A Morphology-Aware Alignment Model for Unsupervised Bilingual Lexicon Induction (P19-1)
Copied to clipboard
| Challenge: | Existing work shows that morphological variation is an intractable challenge for the unsupervised bilingual lexicon induction task. |
| Approach: | They propose a morphology-aware alignment model to alleviate the adverse effect of morphological variation by introducing grammatical information learned by the pre-trained denoising language model. |
| Outcome: | The proposed model outperforms state-of-the-art unsupervised systems and achieves competitive performance compared to supervised methods. |
Incorporating Glosses into Neural Word Sense Disambiguation (P18-1)
Copied to clipboard
| Challenge: | Existing neural networks for Word Sense Disambiguation rely on labeled data and lexical knowledge. |
| Approach: | They propose a gloss-augmented WSD neural network which integrates context and glosses of the target word into a unified framework. |
| Outcome: | The proposed model outperforms the state-of-the-art systems on several English all-words WSD datasets. |
Leveraging Gloss Knowledge in Neural Word Sense Disambiguation by Hierarchical Co-Attention (D18-1)
Copied to clipboard
| Challenge: | Existing models for Word Sense Disambiguation use labeled data, but lack gloss knowledge. |
| Approach: | They propose a co-attention mechanism to generate co-dependent representations for context and gloss . they propose to incorporate gloss knowledge into neural networks for Word Sense Disambiguation . |
| Outcome: | The proposed model achieves state-of-the-art results on standard English all-words WSD datasets. |
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work in multilingual pretraining relies on the shared vocabulary and bilingual contexts to encourage the correlation across languages. |
| Approach: | They propose to plug a cross-attention module into a Transformer encoder to explicitly build the interdependence between languages. |
| Outcome: | The proposed model outperforms existing models on XTREME and English-to-French translation datasets. |