Papers by Nader Akoury
A Framework for Exploring Player Perceptions of LLM-Generated Dialogue in Commercial Video Games (2023.findings-emnlp)
Copied to clipboard
| Challenge: | evaluating the player experience in a roleplaying game augmented with LLM-generated dialogue remains a major challenge. |
| Approach: | They propose a dynamic evaluation framework for the dialogue management systems that govern the task-oriented dialogue often found in roleplaying video games. |
| Outcome: | The proposed framework directly evaluates the performance of LLM-generated dialogue in a role-playing game with 28 players. |
Syntactically Supervised Transformers for Faster Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Standard decoders for neural machine translation generate a single token per timestep, which slows inference . a series of controlled experiments demonstrates that SynST decodes sentences 5x faster than the baseline autoregressive Transformer. |
| Approach: | They propose a syntactically supervised Transformer that generates all target tokens in one shot . synST is a variant of the Transformer architecture that autoregressively predicts a chunked parse tree . |
| Outcome: | The proposed method decodes sentences 5x faster than the baseline method on En-De and En-Fr datasets while achieving higher BLEU scores. |
STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets lack rich enough contexts to guide models and evaluations are unreliable for long-form creative text. |
| Approach: | They propose a dataset and evaluation platform built from STORIUM . their dataset contains 6K lengthy stories with fine-grained natural language annotations . |
| Outcome: | The proposed model can be used to generate 6K long stories with fine-grained natural language annotations and a user-generated dataset. |
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on open-ended text generation tasks because they are difficult to evaluate automatically. |
| Approach: | They conduct a survey of 45 open-ended text generation papers to determine whether models are reproducible . they then run story evaluation experiments with AMT workers and English teachers . |
| Outcome: | The results show that AMT workers and English teachers perform better when shown model-generated output alongside human-generated references. |