Papers by Da Song
Benchmarking Language Models for Code Syntax Understanding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models capture the syntactic rules of natural languages without fine-tuning on syntax understanding tasks. |
| Approach: | They propose a benchmarking test to compare pre-trained language models with a large-scale dataset of programs annotated with syntactic relationships in their corresponding abstract syntax trees. |
| Outcome: | The proposed model fails to match baselines based on positional offsets and keywords. |
TESTEVAL: Benchmarking Large Language Models for Test Case Generation (2025.findings-naacl)
Copied to clipboard
Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, Lei Ma
| Challenge: | Existing methods to generate test cases using large language models are limited in their ability to generate unit test cases. |
| Approach: | They propose a test case generation benchmark that uses large language models to generate unit test cases. |
| Outcome: | The proposed test case generation benchmarks compare LLMs with commercial and open-source LLM platforms and find that they lack the ability to comprehend program logic and execution paths. |
To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planning (2024.emnlp-demo)
Copied to clipboard
Da Ju, Song Jiang, Andrew Cohen, Aaron Foss, Sasha Mitts, Arman Zharmagambetov, Brandon Amos, Xian Li, Justine Kao, Maryam Fazel-Zarandi, Yuandong Tian
| Challenge: | a new system that takes natural language requests from users generates and trains optimal travel plans . a user can provide instructions and an agent provides optimal solutions . the system takes 5seconds to reply to the user request with guaranteed itineraries . |
| Approach: | They propose a real-time demo system that takes natural language requests from users . it translates requests to symbolic form and produces optimal travel itineraries with LLM . |
| Outcome: | The proposed system produces optimal travel itineraries with mixed integer linear programming solvers. |
Trial and Error: Exploration-Based Trajectory Optimization of LLM Agents (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have become integral components in various autonomous agent systems. |
| Approach: | They propose an exploration-based trajectory optimization approach that allows agents to learn from their exploration failures. |
| Outcome: | The proposed method outperforms baseline methods on three complex tasks by a large margin. |