Papers by Amanda Bertsch
Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions (2025.emnlp-main)
Copied to clipboard
Emmy Liu, Amanda Bertsch, Lintang Sutawika, Lindia Tjuatja, Patrick Fernandes, Lara Marinov, Michael Chen, Shreya Singhal, Carolin Lawrence, Aditi Raghunathan, Kiril Gashteovski, Graham Neubig
| Challenge: | Language model performance is largely dependent on pretraining decisions, but scaling laws based on only these two aspects do not always explain downstream task performance. |
| Approach: | They meta-analyze 92 open-source pretrained models to quantify their impact on performance. |
| Outcome: | The framework lays a foundation for more systematic investigation of how model development choices shape final capabilities. |
To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Natural language processing (NLP) is in a period of disruptive change that is impacting our methodologies, funding sources, and public perception. |
| Approach: | They conduct interviews with 26 NLP researchers of varying seniority, research area, institution, and social identity to identify cyclical patterns in the field and new shifts without historical parallel . they conclude by discussing shared visions, concerns, and hopes for the future of NLP . |
| Outcome: | The authors identify cyclical patterns in the field, as well as new shifts without historical parallel, including changes in benchmark culture and software infrastructure. |
Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attention (2025.acl-long)
Copied to clipboard
| Challenge: | Many-shot in-context learning shifts computational burden from training-time to inference-time, making deployment of many-shot ICL challenging to justify in-practice. |
| Approach: | They propose a method for retrieval-based many-shot in-context learning that uses blocks-sparse attention and retrieval of cached demonstrations to achieve comparable per-example latency to finetuning. |
| Outcome: | The proposed method achieves comparable per-example latency to finetuning while maintaining on average >95% of the best method’s accuracy across strong ICL and finetuned baselines. |
He Said, She Said: Style Transfer for Shifting the Perspective of Dialogues (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on style transfer has focused on controlling formality, authorial style, and sentiment of text. |
| Approach: | They propose a style transfer task that reframes a dialogue from informal first person to formal third person rephrasing . they use a dataset to annotate dialogues from a text summarization corpus . |
| Outcome: | The proposed task improves the performance of extractive models on a dialogue summarization dataset. |
Prompt2Model: Generating Deployable Models from Natural Language Instructions (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a step backward from traditional special-purpose NLP models . they require extensive computational resources for deployment and can be gated behind APIs . |
| Approach: | They propose a general-purpose method that takes a natural language task description and uses it to train a special-purpose model. |
| Outcome: | The proposed method outperforms a strong LLM by 20% while being 700 times smaller. |
In-Context Learning with Long-Context Models: An In-Depth Exploration (2025.naacl-long)
Copied to clipboard
| Challenge: | In-context learning is limited by context length, but it can be used for many tasks. |
| Approach: | They study the behavior of in-context learning at an extreme context length . example retrieval shows excellent performance at low context lengths but has diminished gains . |
| Outcome: | The proposed model can perform many tasks with reasonable accuracy when a few examples are provided in-context. |
FicSim: A Dataset for Multi-Faceted Semantic Similarity in Long-Form Fiction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | evaluating the usefulness of language models for literary-domain tasks remains challenging due to the cost of fine-grained annotation for long-form texts and data contamination concerns inherent in using public-domain literature. |
| Approach: | They use a dataset of long-form, recently written fiction to evaluate embedding models . they prioritize author agency and rely on continual, informed author consent . |
| Outcome: | The proposed dataset of long-form, recently written fiction is compared with existing models on this task. |