Papers with EWC
Unsupervised Pretraining for Neural Machine Translation Using Elastic Weight Consolidation (P19-2)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) uses sequence to sequence architectures, but requires a huge amount of parallel data. |
| Approach: | They use Elastic Weight Consolidation to regularize weights of two language models . they then fine-tune the model on parallel data to avoid forgetting the original task . |
| Outcome: | The proposed method achieves BLEU scores similar to the previous work, but is slower and requires less training data. |
Elastic weight consolidation for better bias inoculation (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that the lack of suitable inductive biases in sentence-pair classification models can cause misclassifications on training datasets. |
| Approach: | They propose to use elastic weight consolidation (EWC) to fine-tune models to mitigate biases while being less susceptible to catastrophic forgetting. |
| Outcome: | The proposed model improves on fact verification and stress tests while maintaining the original task accuracy. |
Mitigating the Diminishing Effect of Elastic Weight Consolidation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing work addresses catastrophic forgetting in sequential training by fine-tuning pre-trained language models on different datasets. |
| Approach: | They propose to rescale the components of EWC to mitigate catastrophic forgetting by mixing new and old training data and retraining the model from scratch. |
| Outcome: | The proposed method requires smaller values for the trade-off parameters to achieve comparable results to EWC on natural language inference and fact-checking tasks. |