Bag of Experts Architectures for Model Reuse in Conversational Language Understanding (N18-3)
Copied to clipboard
| Challenge: | Slot tagging is a key component of natural language understanding systems for personal digital assistants. |
| Approach: | They propose to use a bag of experts architecture to reuse domain data for slot tagging models. |
| Outcome: | Experiments with 10 domains show that the proposed models outperform baseline models by 5.06% and 12.16% when training with only 25% of the training data. |
Similar Papers
Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging (2022.findings-acl)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models (PLMs) have made extraordinary progress in most NLP tasks, but they fail to achieve state-of-the-art (SOTA) performance. |
| Approach: | They propose a Guassian HMM variant for unsupervised POS tagging that incorporates contexualized word representations into the decoder. |
| Outcome: | The proposed model outperforms state-of-the-art models on Penn Treebank and multilingual Universal Dependencies treebank v2.0. |
A Simple and Effective Usage of Word Clusters for CBOW Model (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing word clustering algorithms can be used to obtain word embeddings without additional language resources. |
| Approach: | They propose to replace infrequent input and output words with clusters to produce word embeddings. |
| Outcome: | The proposed method produces embeddings of frequent words and small amount of cluster embeddables, which can be fine-tuned on downstream tasks. |
PARE: A Simple and Strong Baseline for Monolingual and Multilingual Distantly Supervised Relation Extraction (2022.acl-short)
Copied to clipboard
| Challenge: | Recent approaches to distantly supervised relation extraction (DS-RE) encode each sentence in an entity-pair bag separately. |
| Approach: | They propose a simple baseline approach where sentences of a bag are concatenated into a passage of sentences and encoded jointly using BERT. |
| Outcome: | The proposed approach outperforms state-of-the-art models in monolingual and multilingual datasets. |
PBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embedding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing word embeddings assume fixed finite-size vocabularies, hindering their ability to provide useful word representations for out-of-vocaulary words. |
| Approach: | They propose a model that generalizes word embeddings without extra contextual information . they use the spellings of words to model subword segmentation and compute subword-based compositional word embeds. |
| Outcome: | The proposed model can generate meaningful subword segmentations without any source of explicit morphological knowledge. |
Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In recent years, pre-trained language models have become the de facto instrument for the field of natural language processing (NLP). |
| Approach: | They propose a method that injects linguistic structures into pre-trained language models in the parameter-efficient fine-tuning setting. |
| Outcome: | The proposed approach outperforms state-of-the-art methods with a comparable number of parameters. |
Parsing as Tagging (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for dependency parsing treat parse as tagging, but they are not perfect. |
| Approach: | They propose a simple yet accurate method that treats parsing as tagging . they use a sequence model with a bidirectional LSTM over BERT embeddings . |
| Outcome: | The proposed method outperforms the state-of-the-art method on universal dependency (UD) by 1.76% unlabeled attachment score (UAS) for English, 1.98% UAS for French, and 1.16% UAS in German. |
An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers (2022.acl-short)
Copied to clipboard
| Challenge: | a standard tokenizer does not cover all characters of a word but preserves key aspects of its morphological structure . a novel method to improve tokenization of pretrained language models is proposed . |
| Approach: | They propose a method to improve the tokenization of pretrained language models . they use the vocabulary of a standard tokenizer but preserves morphological structure . |
| Outcome: | The proposed method improves tokenization of pretrained language models on morphological gold segmentations and text classification tasks. |
TAP4LLM: Table Provider on Sampling, Augmenting, and Packing Semi-structured Data for Large Language Model Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing solutions for table reasoning tasks are mainly tested on small tables and face scalability issues and struggle with complex queries due to incomplete or dispersed data across different table sections. |
| Approach: | They propose a table reasoning pre-processor suite that can be used to leverage large language models (LLMs) in table-based tasks. |
| Outcome: | The proposed method improves LLMs’ reasoning capabilities in various tabular tasks and enhances interaction between LLM and tabular data by employing effective pre-processing. |
Breaking ReLU Barrier: Generalized MoEfication for Dense Pretrained Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to convert pretrained dense models to MoEs are limited to ReLU-based models with natural sparsity. |
| Approach: | They propose a G-MoEfication approach for arbitrary dense models where activation sparsity assumptions no longer hold. |
| Outcome: | The proposed method reduces the inference cost associated with dense models by sparsely activating experts. |
Connecting Language and Knowledge with Heterogeneous Representations for Neural Relation Extraction (N19-1)
Copied to clipboard
| Challenge: | Knowledge Bases (KBs) require constant updating to reflect changes to the world they represent. |
| Approach: | They propose a framework that unifies learning of RE and KBE models . the framework is based on a relation extraction task that uses a KB relation to a phrase . |
| Outcome: | The proposed framework unifies learning of RE and KBE models, leading to significant improvements over the state-of-the-art RE framework. |