Papers by Jakob Foerster
HelloFresh: LLM Evalutions on Streams of Real-World Human Editorial Actions across X Community Notes and Wikipedia edits (2024.findings-acl)
Copied to clipboard
Tim Franzmeyer, Aleksandar Shtedritski, Samuel Albanie, Philip Torr, Joao F. Henriques, Jakob Foerster
| Challenge: | a better understanding of LLM capabilities on real world tasks is vital for safe development and deployment. |
| Approach: | They propose a new LLM called HelloFresh that uses real-world data to measure performance . they backtest the model and find it yields a temporally consistent ranking . |
| Outcome: | The proposed benchmarks outperform static evaluation data and test data on Wikipedia pages. |
Seeded self-play for language learning (D19-64)
Copied to clipboard
| Challenge: | Current methods for learning human language are too data inefficient to learn it in this way. |
| Approach: | They propose to train a meta-learning agent in simulation to interact with populations of pre-trained agents, each with their own distinct communication protocol. |
| Outcome: | The proposed algorithm minimizes the number of on-policy interactions while learning human language while minimizing the number on-political interactions. |