Papers by Itay Levy
Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens (2022.naacl-main)
Copied to clipboard
| Challenge: | Standard pre-trained language models do not see the characters that compose each token's string representation. |
| Approach: | They probe the embedding layer of pretrained language models and show that models learn the internal character composition of whole word and subword tokens without seeing the characters coupled with the tokens. |
| Outcome: | The embedding layers of RoBERTa and GPT2 hold enough information to accurately spell up to a third of the vocabulary and reach high character ngram overlap across all token types. |
Diverse Demonstrations Improve In-context Compositional Generalization (2023.acl-long)
Copied to clipboard
| Challenge: | In-context learning has shown great success in i.i.d semantic parsing splits . however, in compositional generalization, selecting similar demonstrations is insufficient . |
| Approach: | They propose a method to select diverse demonstrations that collectively cover all the structures required in the output program and encourage the model to generalize to new structures from these demonstrations. |
| Outcome: | The proposed method improves performance across three compositional generalization datasets and finetuning. |