Papers by Tianhao Shen
Automated Progressive Red Teaming (2025.coling-main)
Copied to clipboard
| Challenge: | Automated red teaming (ART) is effective but time-consuming, costly and lacks scalability. |
| Approach: | They propose an automated red teaming framework that generates adversarial prompts to expose LLM vulnerabilities. |
| Outcome: | The proposed framework explores and exploits LLM vulnerabilities through multi-round interactions. |
X-RiSAWOZ: High-Quality End-to-End Multilingual Dialogue Datasets and Few-shot Agents (2023.findings-acl)
Copied to clipboard
Mehrad Moradshahi, Tianhao Shen, Kalika Bali, Monojit Choudhury, Gael de Chalendar, Anmol Goel, Sungkyun Kim, Prashant Kodali, Ponnurangam Kumaraguru, Nasredine Semmar, Sina Semnani, Jiwon Seo, Vivek Seshadri, Manish Shrivastava, Michael Sun, Aditya Yadavalli, Chaobin You, Deyi Xiong, Monica Lam
| Challenge: | X-RiSAWOZ dataset has more than 18,000 human-verified dialogue utterances for each language . Xiaoping and Xinhui are the main challenges for task-oriented dialogue research . |
| Approach: | They develop a toolkit to accelerate the post-editing of a new language dataset after translation . their dataset, code, and toolkit are released open-source . |
| Outcome: | The proposed toolkit accelerates the post-editing of a new language dataset after translation. |
CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation (2024.acl-long)
Copied to clipboard
| Challenge: | CharacterEval is a benchmark for comprehensive RPCA assessment in Chinese . authors show that Chinese LLMs exhibit more promising capabilities than GPT-4 in role-playing conversation. |
| Approach: | They propose a Chinese benchmark for comprehensive RPCA assessment . they use a dataset of Chinese role-playing dialogues and character profiles . |
| Outcome: | The proposed benchmark demonstrates that Chinese LLMs exhibit more promising capabilities than GPT-4 in Chinese role-playing conversation. |
ChatSOP: An SOP-Guided MCTS Planning Framework for Controllable LLM Dialogue Agents (2025.acl-long)
Copied to clipboard
Zhigen Li, Jianxiang Peng, Yanmeng Wang, Yong Cao, Tianhao Shen, Minghui Zhang, Linxi Su, Shang Wu, Yihang Wu, YuQian Wang, Ye Wang, Wei Hu, Jianfeng Li, Shaojun Wang, Jing Xiao, Deyi Xiong
| Challenge: | Existing models that use Large Language Models (LLMs) show superior performance in various tasks, but lack of controllability leads to unfocused conversations or task failure. |
| Approach: | They propose a standard operating procedure (SOP) framework to regulate dialogue flow by integrating Chain of Thought reasoning and supervised fine-tuning for SOP prediction. |
| Outcome: | The proposed method achieves a 27.95% improvement in action accuracy compared to baseline models based on GPT-3.5 and also shows notable gains for open-source models. |
C²RBench: A Chinese Complex Reasoning Benchmark for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks often fail to capture complex multi-step reasoning demands inherent in real-world scenarios. |
| Approach: | They propose a benchmark to evaluate multi-step, multimodal advanced reasoning of large language models. |
| Outcome: | The proposed benchmark exceeds existing benchmarks in cognitive complexity and accuracy by over 90% . it features 1,115 carefully curated Chinese tasks organized into eight domain-specific subsets . evaluations of 20 LLMs and 24 multimodal large language models reveal critical performance gaps . |
ChatMusician: Understanding and Generating Music Intrinsically with LLM (2024.findings-acl)
Copied to clipboard
Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, Liumeng Xue, Ziyang Ma, Qin Liu, Tianyu Zheng, Yizhi Li, Yinghao Ma, Yiming Liang, Xiaowei Chi, Ruibo Liu, Zili Wang, Chenghua Lin, Qifeng Liu, Tao Jiang, Wenhao Huang, Wenhu Chen, Jie Fu, Emmanouil Benetos, Gus Xia, Roger Dannenberg, Wei Xue, Shiyin Kang, Yike Guo
| Challenge: | Despite LLMs' impressive capabilities in musical knowledge, music reasoning remains an unsolved task. |
| Approach: | They propose an open-source large language model (LLM) that integrates intrinsic musical abilities into LLaMA2 and GPT-3.5. |
| Outcome: | The proposed model can understand and generate music with a pure text tokenizer without external multi-modal neural structures or tokenizers. |
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022.emnlp-demos)
Copied to clipboard
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo F . R. Ribeiro, Lewis Tunstall, Li Zhang, Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou
| Challenge: | Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. |
| Approach: | They propose to use the Generation, Evaluation, and Metrics Benchmark to integrate new evaluation methods into existing evaluations. |
| Outcome: | The proposed evaluation infrastructure bridges the gap between the advantages of leaderboards and in-depth and evolving evaluations by allowing model developers to benefit from each other's work. |
Recovering Gold from Black Sand: Multilingual Dense Passage Retrieval with Hard and False Negative Samples (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to recover and utilize negative samples have not been effective on all languages. |
| Approach: | They propose a multilingual dense passage retrieval framework to recover and utilize negative samples. |
| Outcome: | The proposed framework outperforms strong sparse, dense and hybrid baselines on a high-quality dataset covering eleven typologically diverse languages. |
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (2024.findings-acl)
Copied to clipboard
| Challenge: | OpenCodeInterpreter-33B provides a high level of performance for code generation, executing, and iterative refinement. |
| Approach: | They propose a family of open-source code systems for generating, executing, and iteratively refining code. |
| Outcome: | The OpenCodeInterpreter-33B performs well on humanEval, MBPP, and EvalPlus benchmarks. |