Papers by Uri Alon
On the Expressivity Role of LayerNorm in Transformers’ Attention (2023.findings-acl)
Copied to clipboard
| Challenge: | Layer Normalization (LayerNorm) is an inherent component in all Transformer-based models. |
| Approach: | They propose to use LayerNorm to normalize the activations during the forward pass and their gradients during the backward pass. |
| Outcome: | The proposed model is able to express the multi-head attention layer that follows it in a d-1 space and scales to the same norm of d. |
Learning and Evaluating a Differentially Private Pre-trained Language Model (2021.findings-emnlp)
Copied to clipboard
Shlomo Hoory, Amir Feder, Avichai Tendler, Sofia Erell, Alon Peled-Cohen, Itay Laish, Hootan Nakhost, Uri Stemmer, Ayelet Benjamini, Avinatan Hassidim, Yossi Matias
| Challenge: | Contextual language models have improved performance but can lead to information leakage . |
| Approach: | They propose a differentially-private word-piece algorithm that allows training a tailored domain-specific vocabulary while maintaining privacy. |
| Outcome: | The proposed model can guarantee privacy while maintaining good model performance. |
In-Context Learning with Long-Context Models: An In-Depth Exploration (2025.naacl-long)
Copied to clipboard
| Challenge: | In-context learning is limited by context length, but it can be used for many tasks. |
| Approach: | They study the behavior of in-context learning at an extreme context length . example retrieval shows excellent performance at low context lengths but has diminished gains . |
| Outcome: | The proposed model can perform many tasks with reasonable accuracy when a few examples are provided in-context. |
Language Models of Code are Few-Shot Commonsense Learners (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to generate graphs using pre-trained language models hinder their ability to generate them correctly. |
| Approach: | They propose to frame structured commonsense reasoning tasks as code generation tasks instead of serializing the output graph as a flat list of nodes and edges. |
| Outcome: | The proposed approach outperforms natural-language LMs in three natural language tasks even when the downstream task does not involve source code at all. |
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code (2023.emnlp-main)
Copied to clipboard
| Challenge: | NLCode generates long expressions and statements rather than a single next-token . evaluating and comparing different models has remained a challenge . |
| Approach: | They propose a code-generating evaluation metric built on BERTScore . they use five language-specific pretrained models to evaluate their code . |
| Outcome: | The proposed evaluation metric achieves higher correlation with human preference and functional correctness than existing metrics across four programming languages. |