Tracr-Injection: Distilling Algorithms into Pre-trained Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent efforts to characterize symbolic abilities of the transformer architecture have shown that the tasks that can be implemented in RASP are uncommon to learn from natural unsupervised data. |
| Approach: | They propose a programming language, called RASP, which can be directly compiled into transformer weights to implement these algorithms. |
| Outcome: | The proposed method improves out-of-distribution performance compared to baselines, indicating that indeed a more symbolic mechanism is taking place in the inner workings of the model. |
Similar Papers
Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | Existing methods for quantization-aware training and quantization for learning have limitations in dealing with accumulative quantization errors. |
| Approach: | They propose a method that minimizes accumulative quantization errors and outperforms baselines by distilling knowledge from a fine-tuned teacher network. |
| Outcome: | The proposed method minimizes accumulative quantization errors and outperforms baselines on the XGLUE benchmark. |
Exploring Quantization for Efficient Pre-Training of Transformer Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Quantization has proven to be effective after pre-training and during fine-tuning, but its effects on pre-trainer performance have remained unexplored. |
| Approach: | They propose a linear quantization strategy to be applied during the pre-training of Transformers to improve model efficiency and stability. |
| Outcome: | The proposed method improves model efficiency, stability, and performance while maintaining language modeling ability. |
Natural Language Generation for Effective Knowledge Distillation (D19-61)
Copied to clipboard
| Challenge: | Knowledge distillation can transfer knowledge from deep language representation models to shallow word embedding-based neural networks. |
| Approach: | They propose to build an unlabeled transfer dataset to enable effective knowledge transfer . they hypothesize that this principled, general approach outperforms rule-based techniques . |
| Outcome: | The proposed method outperforms OpenAI GPT on four datasets in sentiment classification, sentence similarity, and linguistic acceptability. |
TRACE: Training and Inference-Time Interpretability Analysis for Language Models (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Existing tools for interpretability analysis of transformer models are post hoc, rely on scalar metrics or require nontrivial integration effort. |
| Approach: | They propose a modular toolkit for training and inference-time interpretability analysis of transformer models. |
| Outcome: | Experiments with autoregressive transformers show that TRACE reveals developmental phenomena overlooked by traditional scalar metrics such as loss or accuracy. |
Synthetic Pre-Training Tasks for Neural Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | toxicity and bias can be addressed by pre-training with synthetic resources . BLEU scores are used to compare methods with real-world data . |
| Approach: | They propose several ways to generate obfuscated data from large parallel corpus and concatenating phrase pairs from small word-aligned corpus with synthetic parallel data without real human language corpora. |
| Outcome: | The proposed methods can be used to generate obfuscated data or synthetic parallel data without real human language corpora even with high levels of oblication. |
DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to extract word embeddings from training datasets are not efficient for training other models. |
| Approach: | They propose a method to distill a training dataset into a textual model by combining a small number of informative synthetic samples. |
| Outcome: | The proposed method outperforms existing methods on training datasets and language models. |
Choose Your Transformer: Improved Transferability Estimation of Transformer Models on Classification Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for NLP tasks require fine-tuning, but it is computationally infeasible. |
| Approach: | They propose an approach that inexpensively estimates a ranking of the expected performance of a given set of transformer language models for a specific task. |
| Outcome: | The proposed model improves the Pearson correlation coefficient between the true model ranks and the estimate. |
General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained language models are currently used for many NLP tasks . however, inference for these models requires significant computational resources . |
| Approach: | They propose to use a shared text encoder to amortize the computational cost of inference over multiple tasks. |
| Outcome: | The proposed method reduces the size of the extracted representations by a factor of 16 to store them for later use. |
Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models (2023.findings-acl)
Copied to clipboard
Borui Wang, Qiuyuan Huang, Budhaditya Deb, Aaron Halfaker, Liqun Shao, Daniel McDuff, Ahmed Hassan Awadallah, Dragomir Radev, Jianfeng Gao
| Challenge: | Existing pre-trained language models that ignore the logical structures underlying natural language text often lack the ability to capture and encode key logical information in the input sequences. |
| Approach: | They propose to construct logic-aware input embeddings for transformer language models through logic detection, logic mapping and hierarchical logical projections and then develop a new modeling paradigm that can upgrade existing transformer language model into logical transformers to boost their performance. |
| Outcome: | The proposed model can achieve superior performance on four important and challenging tasks. |
Attribute Injection for Pretrained Language Models: A New Benchmark and an Efficient Method (2022.coling-1)
Copied to clipboard
| Challenge: | Recent models rely on pretrained language models that use metadata as inputs . however, these methods are either nontrivial or cost-ineffective . |
| Approach: | They propose a benchmark for evaluating attribute injection models using eight datasets . they extend adapters to include attributes independently of or jointly with the text . |
| Outcome: | The proposed method outperforms previous methods and achieves state-of-the-art performance on all datasets. |