Papers by Yuandong Tian
Learning Personalized Alignment for Evaluating Open-ended Text Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Traditional evaluation metrics rely heavily on lexical similarity with human-written references, showing poor correlation with human judgments and failing to account for alignment with the diversity of human preferences. |
| Approach: | They propose an interpretable evaluation framework that evaluates alignment with specific human preferences by providing detailed comments and fine-grained scoring. |
| Outcome: | The proposed framework outperforms GPT-4 in Kendall correlation and accuracy with zero-shot reviewers. |
Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge (2025.emnlp-main)
Copied to clipboard
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason E Weston, Sainbayar Sukhbaatar
| Challenge: | Existing methods for improving large language models have focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. |
| Approach: | They propose an iterative Meta-Rewarding step where the model judges its own judgements and uses that feedback to refine its judgment skills. |
| Outcome: | The proposed model improves Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2 and 20.6% to 29.1% on Arena-Hard. |
CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication (P19-1)
Copied to clipboard
Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, Devi Parikh
| Challenge: | a goal-driven collaborative drawing task combines language, perception, and actions in a partially observable environment . et al., 1990: 138K messages exchanged between human players. |
| Approach: | They propose a goal-driven collaborative task that combines language, perception, and action . they collect a clip art dataset and use it to build an image-drawing game between two agents . |
| Outcome: | The proposed task integrates language, perception, and action in a virtual world . it is based on a dataset of 10K dialogs and 138K messages exchanged between humans . |
To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planning (2024.emnlp-demo)
Copied to clipboard
Da Ju, Song Jiang, Andrew Cohen, Aaron Foss, Sasha Mitts, Arman Zharmagambetov, Brandon Amos, Xian Li, Justine Kao, Maryam Fazel-Zarandi, Yuandong Tian
| Challenge: | a new system that takes natural language requests from users generates and trains optimal travel plans . a user can provide instructions and an agent provides optimal solutions . the system takes 5seconds to reply to the user request with guaranteed itineraries . |
| Approach: | They propose a real-time demo system that takes natural language requests from users . it translates requests to symbolic form and produces optimal travel itineraries with LLM . |
| Outcome: | The proposed system produces optimal travel itineraries with mixed integer linear programming solvers. |
You Only Use Reactive Attention Slice When Retrieving From Long Context (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing retrieval techniques for language models are limited due to their reliance on lexical similarity and are computationally expensive to train. |
| Approach: | They propose a training-free and fine-tuning-free attention-based retrieval technique that uses a reaction score heuristic to quantify how an LM’s self-attention “reacts” to a user query. |
| Outcome: | The proposed approach improves QA task accuracy by 15% and inference throughput by 31% compared to embedding-based retrieval. |
Re3: Generating Longer Stories With Recursive Reprompting and Revision (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has generated short stories of several pages in length, but they are much shorter than typical short stories meant for human consumption. |
| Approach: | They propose a framework to generate long-range plot coherence and relevance by prompting a general-purpose language model and a language model. |
| Outcome: | The proposed framework generates stories of 2000-2500 words, compared to similar-length stories generated directly from the same model. |
DOC: Improving Long Story Coherence With Detailed Outline Control (2023.acl-long)
Copied to clipboard
| Challenge: | Detailed Outline Control (DOC) framework improves long-range plot coherence . human evaluations of DOC show it outperforms strong Re3 on plot cohesion, outline relevance and interestingness . |
| Approach: | They propose a Detailed Outline Control framework to improve long-range plot coherence . the detailed outliner creates a more detailed, hierarchically structured outline . they propose doc with a detailed controller to ensure the more detailed outline is respected . |
| Outcome: | The proposed framework outperforms Re3 on plot coherence, outline relevance and interestingness. |