Papers by Pratyusha Sharma
Grokking of Hierarchical Structure in Vanilla Transformers (2023.acl-short)
Copied to clipboard
| Challenge: | a recent study has shown that neural sequence models like transformers can generalize hierarchically when training for extended periods. |
| Approach: | They show that transformers can learn to generalize hierarchically after long training periods . they call this phenomenon structural grokking, which exhibits inverted U-shaped scaling in model depth . |
| Outcome: | The proposed model generalizes better than both very deep and very shallow models on multiple datasets. |
Pseudointelligence: A Unifying Lens on Language Model Evaluation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies claim that language models surpass human performance on new benchmarks within a few years. |
| Approach: | They propose a framework for model evaluation that casts as a dynamic interaction between a model and a learned evaluator. |
| Outcome: | The proposed framework can be used to reason about two case studies in language model evaluation, and analyze existing evaluation methods. |
Pushdown Layers: Encoding Recursive Structure in Transformer Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pushdown Layers model recursive state via stack tape that tracks estimated depths of tokens in incremental parsing . pushdown layers are drop-in replacement for standard self-attention . recursion is a key component of many aspects of intelligent behavior, authors say . |
| Approach: | They propose a self-attention layer that models recursive state via a stack tape . Pushdown Layers is a drop-in replacement for standard self- attention . |
| Outcome: | The proposed self-attention layer improves on parse tasks with a recursive-state model . it can model recursion using a stack tape that tracks estimated depths of tokens . |
Skill Induction and Planning with Latent Language (2022.acl-long)
Copied to clipboard
| Challenge: | a framework for learning hierarchical policies from demonstrations is lacking . we use sparse annotations to guide the discovery of reusable skills . |
| Approach: | They propose a framework for learning hierarchical policies from demonstrations using sparse annotations. |
| Outcome: | The proposed model outperforms other models with access to ground-truth plans in the ALFRED simulation environment. |