Papers with copying
Making a Point: Pointer-Generator Transformers for Disjoint Vocabularies (2020.aacl-srw)
Copied to clipboard
| Challenge: | Existing neural models rely on an overlap between source and target vocabularies to perform sequence-to-sequence tasks. |
| Approach: | They propose a pointer-generator transformer model for disjoint vocabularies that does not rely on an overlap between source and target vocs. |
| Outcome: | The proposed model outperforms a standard pointer-generator transformer by an average of 5.1 WER over 15 languages. |
Language Models “Grok” to Copy (2025.naacl-short)
Copied to clipboard
| Challenge: | We examine the pre-training dynamics of language models, focusing on their ability to copy text from preceding context. |
| Approach: | They propose that Transformer-based language models develop copying abilities similarly to grokking . they argue that the connection between groking and context copying can improve in-context performance. |
| Outcome: | The proposed model development is similar to grokking, but the speed is independent of tokens trained. |
Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale (2023.acl-long)
Copied to clipboard
| Challenge: | 70% of attention heads and 20% of the feed forward networks can be removed with minimal decline in task performance. |
| Approach: | They propose to investigate whether in-context learning is not uniform across all components of a large language model. |
| Outcome: | The proposed model can remove 70% of attention heads and 20% of feed forward networks with minimal decline in task performance. |