Papers by Andrew Liu
Open-ended Knowledge Tracing for Computer Science Education (2022.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge tracing (KT) is a method used to estimate student mastery of concepts/skills/knowledge components from their responses to questions and to predict future performance. |
| Approach: | They propose a student knowledge-guided code generation approach that combines program synthesis methods with student knowledge tracing methods to solve the OKT problem. |
| Outcome: | The proposed method is based on a student knowledge-guided code generation approach and validates on coding questions. |
AlignBench: Benchmarking Chinese Alignment of Large Language Models (2024.acl-long)
Copied to clipboard
Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Andrew Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Xiaotao Gu, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, Jie Tang
| Challenge: | Effective evaluation of alignment for emerging Chinese LLMs is still significantly lacking, calling for real-scenario grounded, open-ended, challenging and automatic evaluations tailored for alignment. |
| Approach: | They propose a multi-dimensional benchmark for evaluating LLMs’ alignment in Chinese with 8 main categories, 683 real-scenario rooted queries and corresponding human verified references. |
| Outcome: | The benchmark uses a human-in-the-loop data curation pipeline, 683 real-scenario rooted queries and human verified references. |
Cross-Modal Discrete Representation Learning (2022.acl-long)
Copied to clipboard
| Challenge: | a new framework for learning representations from multimodal data is proposed . the proposed framework uses discretized embedding vectors to capture finer levels of granularity . |
| Approach: | They propose a self-supervised representation learning framework that captures finer levels of granularity across different modalities. |
| Outcome: | The proposed representation can capture finer levels of granularity across different modalities . it can be used on cross-modal retrieval tasks without direct supervision . |
ClinicalTrialsHub: Bridging Registries and Literature for Comprehensive Clinical Trial Access (2026.eacl-demo)
Copied to clipboard
Jiwoo Park, Ruoqi Liu, Avani Jagdale, Andrew Srisuwananukorn, Jing Zhao, Lang Li, Ping Zhang, Sachin Kumar
| Challenge: | ClinicalTrialsHub consolidates clinical trial data from ClinicalTrial.gov and augments it by extracting and structuring trial-relevant information from PubMed. |
| Approach: | They propose a search-focused platform that consolidates PubMed data and extracts structured trial information. |
| Outcome: | ClinicalTrialsHub increases access to structured clinical trial data by 83.8% compared to ClinicalTrial.gov alone. |
Eeyore: Realistic Depression Simulation via Expert-in-the-Loop Supervised and Preference Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been explored for mental healthcare training and therapy client simulation, but they fail to authentically capture diverse client traits and psychological conditions. |
| Approach: | They propose an 8B model optimized for realistic depression simulation with expert input at every stage. |
| Outcome: | The model outperforms GPT-4o in linguistic authenticity and profile adherence. |
AutoCT: Automating Interpretable Clinical Trial Prediction with LLM Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Clinical trials are expensive and time-consuming, and accurate trial prediction is key to advancing medical treatments. |
| Approach: | They propose a framework that combines reasoning capabilities of large language models with the explainability of classical machine learning to generate, evaluate, and refine tabular features without human input. |
| Outcome: | The proposed framework performs better than SOTA methods on clinical trial prediction tasks within a limited number of iterations. |
RATIONALYST: Pre-training Process-Supervision for Improving Reasoning (2025.acl-long)
Copied to clipboard
Dongwei Jiang, Guoxuan Wang, Yining Lu, Andrew Wang, Jingyu Zhang, Chuyu Liu, Benjamin Van Durme, Daniel Khashabi
| Challenge: | RATIONALYST is a model for process-supervision of reasoning based on pretraining on rationale annotations extracted from unlabeled data. |
| Approach: | They propose a model for process-supervision of reasoning based on pre-training on rationale annotations extracted from unlabeled data. |
| Outcome: | RATIONALYST improves reasoning accuracy by 3.9% on representative reasoning benchmarks. |
Boosting LLM Agents with Recursive Contemplation for Effective Deception Handling (2024.findings-acl)
Copied to clipboard
Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, Gao Huang
| Challenge: | Recent advances in large language models (LLMs) have led to significant success in using LLMs as agents. |
| Approach: | They propose a cognitive framework that incorporates first-order and second-order perspective transitions into LLMs to enhance their ability to identify and counteract deceptive information. |
| Outcome: | The proposed framework enhances LLMs’ ability to identify and counteract deceptive information without extra fine-tuning and data. |
TN-Eval: Rubric and Evaluation Protocols for Measuring the Quality of Behavioral Therapy Notes (2025.acl-industry)
Copied to clipboard
| Challenge: | Behavioral therapy notes are important for legal compliance and patient care, but quality standards for them remain underdeveloped. |
| Approach: | They propose a rubric for evaluating therapy notes across key dimensions: completeness, conciseness, faithfulness. |
| Outcome: | The proposed evaluation framework improves on therapist-written notes and LLM-generated notes. |
BrowseComp-Plus: A Fair and Disentangled Evaluation Benchmark for Deep Search Agents (2026.acl-long)
Copied to clipboard
Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Sahel Sharifymoghaddam, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Hosna Oyarhoseini, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, Jimmy Lin
| Challenge: | Existing benchmarks for deep search agents rely on blackbox web search APIs . dynamic and opaque web APIs hinder reproducibility and fair comparisons - authors . |
| Approach: | They propose a benchmark that employs a fixed corpus for controlled retrieval for deep search agents. |
| Outcome: | The new benchmark shows that agents that combine large language models with retrieval tools excel at complex, reasoning-intensive queries. |
Open-Domain Safety Policy Construction (2026.findings-eacl)
Copied to clipboard
| Challenge: | Moderation layers are core component of many products built on user-generated content. |
| Approach: | They propose a system that drafts a content moderation policy based on human-written seed domain information. |
| Outcome: | The proposed system outperforms definition-only and in-context learning baselines on openAI undesired content benchmarks and an in-house multimodal advertisement moderation benchmark. |
UniMorph 4.0: Universal Morphology (2022.lrec-1)
Copied to clipboard
Khuyagbaatar Batsuren, Omer Goldman, Salam Khalifa, Nizar Habash, Witold Kieraś, Gábor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Yustinus Ghanggo Ate, Maria Ryskina, Sabrina Mielke, Elena Budianskaya, Charbel El-Khaissi, Tiago Pimentel, Michael Gasser, William Abbott Lane, Mohit Raj, Matt Coler, Jaime Rafael Montoya Samame, Delio Siticonatzi Camaiteri, Esaú Zumaeta Rojas, Didier López Francis, Arturo Oncevay, Juan López Bautista, Gema Celeste Silva Villegas, Lucas Torroba Hennigen, Adam Ek, David Guriel, Peter Dirix, Jean-Philippe Bernardy, Andrey Scherbakov, Aziyana Bayyr-ool, Antonios Anastasopoulos, Roberto Zariquiey, Karina Sheifer, Sofya Ganieva, Hilaria Cruz, Ritván Karahóǧa, Stella Markantonatou, George Pavlidis, Matvey Plugaryov, Elena Klyachko, Ali Salehi, Candy Angulo, Jatayu Baxi, Andrew Krizhanovsky, Natalia Krizhanovskaya, Elizabeth Salesky, Clara Vania, Sardana Ivanova, Jennifer White, Rowan Hall Maudslay, Josef Valvoda, Ran Zmigrod, Paula Czarnowska, Irene Nikkarinen, Aelita Salchak, Brijesh Bhatt, Christopher Straughn, Zoey Liu, Jonathan North Washington, Yuval Pinter, Duygu Ataman, Marcin Wolinski, Totok Suhardijanto, Anna Yablonskaya, Niklas Stoehr, Hossep Dolatian, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Aryaman Arora, Richard J. Hatcher, Ritesh Kumar, Jeremiah Young, Daria Rodionova, Anastasia Yemelina, Taras Andrushko, Igor Marchenko, Polina Mashkovtseva, Alexandra Serova, Emily Prud’hommeaux, Maria Nepomniashchaya, Fausto Giunchiglia, Eleanor Chodroff, Mans Hulden, Miikka Silfverberg, Arya D. McCarthy, David Yarowsky, Ryan Cotterell, Reut Tsarfaty, Ekaterina Vylomova
| Challenge: | The Universal Morphology project provides broad-coverage instantiated morphological inflection tables for hundreds of diverse languages. |
| Approach: | They propose a language-independent feature schema for rich morphological annotation and a type-level resource of annotated data in diverse languages realizing that schema. |
| Outcome: | The proposed schema has added 66 new languages, including 24 endangered languages. |
RiTTA: Modeling Event Relations in Text-to-Audio Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing text-to-audio (TTA) generation methods have not explored audio event relation modeling, nor proposed any new framework to enhance this capability. |
| Approach: | They propose a comprehensive relation corpus covering all potential relations in real-world scenarios and a new audio event corpus encompassing commonly heard audios. |
| Outcome: | The proposed framework improves existing models’ relation modeling capability with negligible extra parameters. |
CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation (2024.acl-long)
Copied to clipboard
Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, Minlie Huang
| Challenge: | Existing models for NLP evaluations lack the ability to generate informative critiques in pointwise grading and pairwise comparison especially without references. |
| Approach: | They propose a method which can acquire pointwise grading critiques with pseudo references and revise these critiques via multi-path prompting to obtain informative evaluation data in different tasks and settings. |
| Outcome: | The proposed method outperforms all open-source models and even GPT-4 in system-level correlations of pointwise grading. |
Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models (2024.lrec-main)
Copied to clipboard
Oana Ignat, Zhijing Jin, Artem Abzaliev, Laura Biester, Santiago Castro, Naihao Deng, Xinyi Gao, Aylin Ece Gunal, Jacky He, Ashkan Kazemi, Muhammad Khalifa, Namho Koh, Andrew Lee, Siyang Liu, Do June Min, Shinka Mori, Joan C. Nwatu, Veronica Perez-Rosas, Siqi Shen, Zekun Wang, Winston Wu, Rada Mihalcea
| Challenge: | Recent advances in large language models have led to misleading public discourse that “it’s all been solved.” |
| Approach: | They identify 14 research areas encompassing 45 research directions that require new research and are not directly solvable by LLMs. |
| Outcome: | The research areas identified are 45 research directions that require new research and are not directly solvable by LLMs. |
BPID: A Benchmark for Personal Identity Deduplication (2024.emnlp-industry)
Copied to clipboard
Runhui Wang, Yefan Tao, Adit Krishnan, Luyang Kong, Xuanqing Liu, Yuqian Deng, Yunzhao Yang, Henrik Johnson, Andrew Borthwick, Shobhit Gupta, Aditi Gundlapalli, Davor Golac
| Challenge: | Data deduplication is a critical task in data management and mining, focused on consolidating duplicate records that refer to the same entity. |
| Approach: | They propose to use a dataset with 1,000,000 unlabeled synthetic PII profiles and a subset of 10,000 pairs curated and labeled as matches or non-matches. |
| Outcome: | The proposed datasets contain synthetic profiles built from publicly available sources that do not represent real individuals. |