Papers by Haoyu Zhang
Copied to clipboard
| Challenge: | Spreadsheets are characterized by their extensive two-dimensional grids, flexible layouts, and varied formatting options, which pose significant challenges for large language models (LLMs). |
| Approach: | They propose a structural-anchor-based compression, inverse index translation, and data-format-aware aggregation module to compress spreadsheets effectively. |
| Outcome: | The proposed method outperforms the existing model in GPT4 and achieves a state-of-the-art 78.9% F1 score. |
Copied to clipboard
| Challenge: | Prior work synthesizes tool-use LLM datasets by first generating a user query, then complex tool-using annotations like DFS. |
| Approach: | They propose an agentic framework that synthesizes user queries and generates valid tool-use chains . they propose a dataset with more complex tool use, lower cost, and almost 100% pass rate . |
| Outcome: | Experiments show that tools trained on ToolGrad outperform expensive baseline datasets and proprietary LLMs. |
Copied to clipboard
| Challenge: | Existing approaches to RAG neglect system state variables, resulting in poor performance and erroneous knowledge accumulation. |
| Approach: | They propose a framework that incorporates a Turing Complete System to manage state variables and manage retrieval halting. |
| Outcome: | The proposed framework improves on seven real-world healthcare datasets and shows that it is more accurate than existing methods. |
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) have shown promise in MER, but their internal decision-making mechanisms under modality conflict and missingness remain underexplored. |
| Approach: | They propose a multimodal large language model that can detect and control modality conflicts and missing subsets by a lightweight mechanism that detects and controls modality conflict. |
| Outcome: | The proposed framework improves performance across settings, showing it can handle conflict and missing behaviors. |
Copied to clipboard
| Challenge: | HermEs is a spreadsheet formula prediction language that is difficult for Excel users without programming experience to master. |
| Approach: | They propose a hierarchical approach to formula prediction via HiEraRchical forMulet ExpanSion . they propose generating formulas in a fixed order using hierarchically generated formulas . |
| Outcome: | The proposed approach improves formula prediction accuracy by guaranteeing correct grammar and streamlining token-level decoding with high-level Formulet. |
Copied to clipboard
| Challenge: | Existing methods for grounding video frames with dense annotations require enormous amount of human effort. |
| Approach: | They propose to ground natural language in video frames with only one frame labeled . they propose an end-to-end model that eliminates interference of irrelevant frames . |
| Outcome: | The proposed model can ground natural language in all video frames with only one frame labeled . the proposed model eliminates interference of irrelevant frames based on branch search and cropping techniques . |
Copied to clipboard
| Challenge: | In this work, we focus on the semantic classification of events in context to help machines gain a deeper understanding of events. |
| Approach: | They propose to integrate event semantics into downstream tasks to help machines understand events better. |
| Outcome: | The proposed model improves the understanding of events in context. |
Copied to clipboard
| Challenge: | Existing studies on graph learning on text-attributed graphs have been limited by memory cost and underutilization of relationships between nodes and words. |
| Approach: | They propose a Node Representation Update Pre-training Architecture based on Co-modeling text and graph to learn representations of papers and words simultaneously. |
| Outcome: | The proposed model outperforms baselines on the ogbn-arxiv benchmark dataset. |
Copied to clipboard
| Challenge: | Instruction tuning has enabled large language models to achieve remarkable performance, yet its success heavily depends on the availability of high-quality instruction-response pairs. |
| Approach: | They propose a mutual alignment framework which enforces coherence between instructions and responses through mutual constraints. |
| Outcome: | The proposed framework generalizes well across model architectures and sizes, achieving state-of-the-art performance on LLaMA, Mistral, and Qwen models across diverse benchmarks. |
Copied to clipboard
| Challenge: | Existing models of event processing do not understand the essentiality of step events towards a goal event. |
| Approach: | They propose to deconstruct a goal event into a discrete representation of finer-grained (step) events, which are not equally important to the goal. |
| Outcome: | The proposed model can understand the essentiality of different step events towards a goal event. |
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive performance but face high computational costs and latency, limiting their deployment in resource-constrained settings. |
| Approach: | They propose a robust RAG framework for large language models via Margin-aware Preference Optimization to enhance the accuracy and reliability of SLMs. |
| Outcome: | The proposed framework surpasses state-of-the-art benchmarks on three open-domain question answering tasks. |
Copied to clipboard
| Challenge: | Generating high-quality long-form survey articles poses significant challenges to AI Agent systems. |
| Approach: | They propose a hierarchically modular agent system for long-form survey generation . they use atomic models to implement skeleton initialization, digest construction, and skelet refinement . human evaluations demonstrate system surpasses representative baselines . |
| Outcome: | The proposed system surpasses representative baselines in both content depth and length, highlighting the strength of MCP-based modular planning. |
Copied to clipboard
| Challenge: | Most venture capital investments fail, while a few deliver outsized returns. |
| Approach: | They propose a framework that synthesizes relational evidence across sources . they propose combining information-gain-driven retriever and knowledge base to ground reasoning . |
| Outcome: | The proposed framework achieves +5.9% F1 and +22.1% Precision@5 over state-of-the-art baselines. |
Copied to clipboard
| Challenge: | Previously, CLIP was only regarded as a powerful visual encoder. |
| Approach: | They propose a parameter-efficient fine-tuning strategy to boost CLIP's few-shot performance on a visual entailment task without introducing any additional pre-training procedure. |
| Outcome: | The proposed strategy achieves competitive zero/few-shot results on visual question answering and visual entailment tasks without introducing any additional pre-training procedure. |
Copied to clipboard
| Challenge: | Multimodal Sentiment Analysis (MSA) is effective when using rich information from multiple sources, but the potential sentiment-irrelevant information across modalities may hinder the performance from being further improved. |
| Approach: | They propose an Adaptive Language-guided Multimodal Transformer (ALMT) that learns an irrelevance/conflict-suppressing representation from visual and audio features under guidance of language features at different scales. |
| Outcome: | The proposed model achieves state-of-the-art on several popular datasets and an abundance of ablation shows the effectiveness of the proposed model. |
Copied to clipboard
| Challenge: | Large language models are increasingly employed to empower autonomous agents to simulate human behavior. |
| Approach: | They propose to evaluate LLM-driven agents through multi-turn interactions using a bottom-up approach to create diverse social scenarios constructed from extensive scripts. |
| Outcome: | The proposed model evaluates LLM-driven agents through multi-turn interactions emphasizing goal completion and implicit reasoning. |
Copied to clipboard
| Challenge: | Prior work has explored the selection of examples for in-context learning, neglecting the internal relationships between examples and exist an inconsistency between training and inference. |
| Approach: | They propose a sequential-aware method that leverages the LLM’s feedback on varying context, aiding in capturing inter-relationships and sequential information among examples. |
| Outcome: | Experiments on 23 NLP tasks show that Se2 surpasses baselines and achieves 42% relative improvement over random selection. |
Copied to clipboard
| Challenge: | Evaluating the performance of LLMs in multi-turn interactions presents significant challenges due to the complexity and variability of user behavior. |
| Approach: | They propose a benchmark framework for assessing LLMs’ function-calling capabilities in multi-turn dialogues. |
| Outcome: | The proposed framework is based on a dataset derived from popular mobile apps and anonymized user logs. |
Copied to clipboard
| Challenge: | Document retrieval techniques are used to compute semantic similarity between a query and documents, but the scalar similarity fails to reflect enough information, hindering the interpretation of retrieval results. |
| Approach: | They propose a method which improves the global document-query similarity through contrastive learning and integrates well-designed fusion and decoding modules. |
| Outcome: | The proposed method improves the global document-query similarity through contrastive learning and integrates well-designed fusion and decoding modules. |
Copied to clipboard
| Challenge: | Language models such as GPT and Llama have shown remarkable ability on diverse natural language tasks, yet their performance on complex table tasks is suboptimal. |
| Approach: | They propose a generator-validator paradigm to iteratively generate-then-validate training data from language models to fine-tune stronger Table-Specialist models that can specialize in a given task, without using manually-labeled data. |
| Outcome: | The proposed model outperforms vanilla language models on diverse table tasks and can match or surpass GPT-4 level quality. |
Copied to clipboard
| Challenge: | Logical table-to-text generation requires models to derive logical-level facts from table records via logical inference. |
| Approach: | They propose a pretrained logical form generator framework to improve generation fidelity . they use a dataset to test the logical inference accuracy of the framework . |
| Outcome: | The proposed framework outperforms baselines on LOGICNLG and CONTLOG on two benchmarks. |
Copied to clipboard
| Challenge: | We present a new information extraction system that can construct temporal event graphs from news documents. |
| Approach: | They propose a temporal event graph extraction system that can extract news documents . they extend the system from sentence-level event extraction to cross-document cross-media event extraction . |
| Outcome: | The proposed system can extract temporal event graphs from news documents in multiple languages and multiple data modalities. |
Copied to clipboard
| Challenge: | Large Language Models lack the capacity to formulate global strategies due to latency and availability constraints. |
| Approach: | They propose a framework to internalize the strategic oversight of large models into intrinsic Latent Guidance by synthesizing a query-conditioned Latent Guide. |
| Outcome: | The proposed framework outperforms strong baselines on mathematical and coding benchmarks with negligible inference latency. |
Copied to clipboard
| Challenge: | Existing methods for event prediction are incomplete and noisy. |
| Approach: | They propose to use news-related event schemas to extract newsworthy events . they build a demo website and include a video demonstrating the framework . |
| Outcome: | The proposed framework can be applied to a wide variety of newsworthy scenarios. |
Copied to clipboard
| Challenge: | Large language models (LLMs) often produce unnecessarily long explanations that reduce efficiency. |
| Approach: | They propose a length-aware reward that selectively penalizes insignificance tokens . they also propose 'dynamic length control' that encourages more detailed reasoning . |
| Outcome: | The proposed method reduces response length while maintaining correctness, the authors show . it selectively penalizes insignificance tokens while maintaining accuracy . |
Copied to clipboard
| Challenge: | Existing large language models focus on causal coherence, neglecting the complex story arcs and orchestration inherent in human narratives. |
| Approach: | They propose a high-dimensional framework for narrative orchestration that unifies human and model perspectives while jointly characterizing narrative function and structure in a common space. |
| Outcome: | The proposed framework unifies human and model perspectives while jointly characterizing narrative function and structure in a common space. |
Copied to clipboard
| Challenge: | Existing FET noise learning methods rely on prediction distributions in instance-independent manner, which causes confirmation bias. |
| Approach: | They propose a clustering-based loss correction framework to address confirmation bias in FET . they first train a coarse backbone model as a feature extractor and noise estimator . |
| Outcome: | The proposed framework achieves the best performance over existing systems on three public datasets and is stable to hyperparameters. |
Copied to clipboard
| Challenge: | Existing methods for hierarchical text classification are limited and lack holistic structural information. |
| Approach: | They propose a hierarchy-aware global model with two variants that learn hierarchy-based label embeddings through an encoder and conduct inductive fusion of label-alike text features. |
| Outcome: | The proposed model improves on three benchmark datasets. |
Copied to clipboard
| Challenge: | Existing methods for converting unstructured text into structured Knowledge Graphs (KGs) have limitations such as large amount of noise, inaccurate knowledge, and hallucination . |
| Approach: | They propose a GraphJudge framework to reduce noise in real-world documents . they propose Graphjudge to fine-tune a LLM as a graph judge to enhance quality . |
| Outcome: | The proposed framework eliminates noise in real-world documents and improves the quality of generated KGs. |
Copied to clipboard
| Challenge: | Existing persona-based dialogue models use crowd-sourced data, such as the PersonaChat . however, the cost of such datasets is limited, and the model is not robust. |
| Approach: | They propose to disentangle persona-based dialogue generation into two sub-tasks by using a BERT-over-BERT model. |
| Outcome: | The proposed model outperforms baselines in response quality and persona consistency under different limited data settings. |
Copied to clipboard
| Challenge: | We introduce AI-Press, an automated news drafting and polishing system based on multi-agent collaboration and Retrieval-Augmented Generation. |
| Approach: | They introduce AI-Press, an automated news drafting and polishing system based on multi-agent collaboration and Retrieval-Augmented Generation. |
| Outcome: | The proposed system generates public responses considering demographic distributions. |
Copied to clipboard
| Challenge: | Existing auto-regressive pre-trained language models are challenged by recent emerging numerical reasoning datasets due to the error-prone implicit calculation. |
| Approach: | They propose a pre-computation tool to pre-compute aggregation/arithmetic results for the table in advance, so they are handy and readily available for PLMs to answer numerical reasoning questions. |
| Outcome: | The proposed model improves on TAT-QA and T5 and BART-large on multiple benchmarks. |
Copied to clipboard
| Challenge: | Recent efforts to learn explainable legal case retrieval models fail to provide faithful and interpretable explanations for legal cases. |
| Approach: | They propose a framework that uses logic rules to explain legal case retrieval results . they extend benchmarks of LeCaRD and ELAM with manually annotated logic rules . |
| Outcome: | The proposed framework is able to provide faithful explanations for legal case retrieval. |
Copied to clipboard
| Challenge: | Existing knowledge graph construction frameworks require predefined schemas, limiting their scalability and domain coverage. |
| Approach: | They propose a framework for fully autonomous knowledge graph construction that eliminates the need for predefined schemas. |
| Outcome: | The proposed framework outperforms state-of-the-art models on multi-hop QA tasks and enhances LLM factuality. |
Copied to clipboard
| Challenge: | Genereal domain large models lack nuanced multimodal understanding of social media . general domain models focus more on text than other modalities, which is not consistent with real-world user habits. |
| Approach: | They propose a Large Vision Language Model for Social Media Processing that combines five key capabilities to understand and generate real social media behavior. |
| Outcome: | The proposed model achieves state-of-the-art performance in multiple social media tasks. |
Copied to clipboard
| Challenge: | Understanding natural language involves recognizing how multiple event mentions structurally and temporally interact with each other. |
| Approach: | They propose a joint constrained learning framework that enforces logical constraints within and across multiple temporal and subevent relations of events by converting constraints into differentiable learning objectives. |
| Outcome: | The proposed framework outperforms SOTA methods on benchmarks for temporal relation extraction and event hierarchy construction. |
Copied to clipboard
| Challenge: | Existing methods that ignore the decompositionality of complex questions are not suitable for complex question semantic parsing. |
| Approach: | They propose a hierarchical semantic parsing method which utilizes the decompositionality of complex questions for semantic paring. |
| Outcome: | The proposed method improves on a large scale complex question semantic parsing dataset. |
Copied to clipboard
| Challenge: | Existing studies on improving attribute consistency focus on incorporating attribute information in responses, but few efforts have identified the consistency relations between response and attribute profile. |
| Approach: | They propose a key-value structure information enriched BERT model to identify the profile consistency . they propose to incorporate attribute information into the generated responses . |
| Outcome: | The proposed model improves over strong baselines on downstream tasks. |
Copied to clipboard
| Challenge: | Embodied question answering requires collecting context that is distributed across multiple viewpoints . most recent vision–language models (VLMs) are constrained to a fixed and finite set of input views . |
| Approach: | They propose a training-free, test-time reasoning framework that transforms a VLM into an active viewpoint reasoner through a coarse-to-fine exploration process. |
| Outcome: | The proposed framework improves LLM-Match performance by 11.98% on four mainstream VLMs. |
Copied to clipboard
| Challenge: | Event mentions in text correspond to real-world events of varying degrees of granularity . task of subevent detection aims to resolve this granulem issue by recognizing membership of events . |
| Approach: | They propose a task of event-based text segmentation as an auxiliary task to improve learning for subevent detection. |
| Outcome: | The proposed method outperforms baseline methods on subevent detection, HiEve and IC datasets while achieving decent performance on EventSeg prediction. |
Copied to clipboard
| Challenge: | Existing studies on table reasoning focus on flat tables and hierarchical tables . a new dataset, HiTab, aims to examine numerical reasoning over hierarchic tables based on hierarchically structured tables - a strong challenge for existing baselines and a valuable benchmark for future research. |
| Approach: | They propose a hierarchical question answering and natural language generation dataset to study hierarchic tables. |
| Outcome: | The proposed model shows that it is effective in QA and natural language generation over hierarchical tables. |
Copied to clipboard
| Challenge: | Existing methods for motion understanding lack precise alignment between motion and modalities . existing methods lack precise semantics and a mismatch between motion, text . |
| Approach: | They propose a modality-shared codebook that enables unified representation learning and precise alignment between motion and linguistic modalities. |
| Outcome: | The proposed model surpasses current state-of-the-art methods in many areas . it enables unified representation learning and precise alignment of motion and modalities . |
Copied to clipboard
| Challenge: | Tabular data are crucial in many fields and their understanding by large language models (LLMs) under high parameter efficiency paradigm is important. |
| Approach: | They propose a module that uses 2D LoRA to encode low-rank information on cell positions to improve table serialization and representation of two-dimensional structured information within a one-dimensional sequence. |
| Outcome: | Experiments on four tabular-related datasets show that TableLoRA outperforms vanilla LoRA and surpasses table encoding methods tested in control. |
Copied to clipboard
| Challenge: | Pre-trained language models have demonstrated superior performance on NLP tasks . however, when the training domain and testing domain are taken from different distributions, the deployed model often violates this assumption. |
| Approach: | They propose a Stable Test-time Adaptation Framework to stabilize the adaptation process. |
| Outcome: | The proposed framework boosts model robustness to noise distribution shifts while minimizing error accumulation and catastrophic forgetting. |
Copied to clipboard
| Challenge: | Existing persona-based dialogue models generate human-like responses but can hardly avoid the generation of inconsistent persona words. |
| Approach: | They propose a framework that deletes inconsistent words from a generated response prototype and further rewrites it to a personality-consistent one. |
| Outcome: | The proposed framework achieves good performance on the persona-chat dataset. |
Copied to clipboard
| Challenge: | Existing approaches to event causality identification have primarily utilized linguistic patterns and multi-hop relational inference, risking false causality . |
| Approach: | They propose to use the Rubin Causal Model to identify event causality by generating a twin from existing corpora. |
| Outcome: | The proposed method can identify causal relations more robustly than previous methods, including GPT-4, which is demonstrated on a causality benchmark, COPES-hard. |
Copied to clipboard
| Challenge: | FinWorkBench evaluates real-world enterprise-grade finance and accounting workflows . a human evaluation of GPT 5.1 Pro passes only 38.4% of workflows, a study finds . |
| Approach: | They propose a workflow construction process that combines LLM-assisted mining and expert annotation to build 172 composite workflows. |
| Outcome: | The proposed process combines expert annotation with LLM-assisted mining of workflows from authentic enterprise environments. |
Copied to clipboard
| Challenge: | Existing work on event understanding is focusing on procedural (or horizontal) tasks such as predicting the next event given an observed sequence. |
| Approach: | They propose an Analogous Process Structure Induction framework which leverages analogies among processes and conceptualization of sub-event instances to predict the whole sub- sequence of previously unseen open-domain processes. |
| Outcome: | The proposed framework can predict the whole sub-event sequence of previously unseen open-domain processes. |
Copied to clipboard
| Challenge: | Existing datasets that ignore the challenge of missing knowledge in TableQA are limited in their use. |
| Approach: | They propose to use a knowledge base as the external knowledge source for TableQA and construct a dataset with fine-grained gold evidence annotation. |
| Outcome: | The proposed model achieves remarkable performance improvements on three different settings, but still lags behind the human-level performance. |
Copied to clipboard
| Challenge: | In-context learning has been validated in large language models, but the mechanism and selection strategy for in-cont example selection lacks systematic and in-depth research. |
| Approach: | They propose a data compression approach to select in-context examples using large language models. |
| Outcome: | The proposed method shows a significant improvement of 5.90% across five real-world datasets using four language models. |
Copied to clipboard
| Challenge: | Tables store rich numerical data, but numerical reasoning over tables is still a challenge. |
| Approach: | They propose a spreadsheet formula is a valuable supervision for numerical reasoning in tables. |
| Outcome: | The proposed method outperforms state-of-the-art methods on three representative datasets of formula prediction, question answering, and cell type classification. |
Copied to clipboard
| Challenge: | Tabular data analysis is crucial in many scenarios, yet its complexity and density can make it challenging to determine the most appropriate analysis operations for a new table. |
| Approach: | They propose a tabular data analysis framework that recommends query-code-result triplets for new tables . they propose Rec-Align, a method to further improve recommendation quality . |
| Outcome: | The proposed framework achieves 77.0% top-5 recommendation recall on a dataset designed for tabular data analysis recommendation. |
Copied to clipboard
| Challenge: | Existing work on event extraction relies on labor-intensive annotation, ignoring semantic meaning of event types' labels. |
| Approach: | They propose a zero-shot event extraction approach that first identifies events with existing tools and then maps them to a given taxonomy of event types in a no-shot manner. |
| Outcome: | The proposed approach doubles the performance of previous approaches on a ACE-2005 dataset . it leverages label representations induced by pre-trained language models and maps events to the target types . |
Copied to clipboard
| Challenge: | Visual presentations are vital for effective communication, but they are limited by their complexity and lack of visual understanding. |
| Approach: | a new framework is proposed to generate high-quality visual presentations using multimodal large language models. |
| Outcome: | The proposed framework outperforms existing models in multimodal understanding and content consistency. |
Copied to clipboard
| Challenge: | Existing models for event temporal relation extraction are based on data-driven machine learning . however, TEMPREL extraction is not accurate under distribution shifts. |
| Approach: | They propose to conduct counterfactual analysis to attenuate the effects of two types of training biases: the event trigger bias and the frequent label bias. |
| Outcome: | The proposed model extracts TempRel and timelines more faithfully compared to SOTA methods . it is based on two perspectives: one is to extract genuinely based upon contextual description . the other is to provide proper uncertainty estimation and abstain from extraction when no relation is described in the text . |
Copied to clipboard
| Challenge: | Existing approaches to code generation fail to consider the quality of retrieved examples. |
| Approach: | They propose a retrieval-augmented generation method that combines existing API examples to improve complexity and readability. |
| Outcome: | The proposed method achieves up to 22% accuracy improvement over baseline methods. |
Copied to clipboard
| Challenge: | Creating spreadsheet formulas remains a tedious and error-prone task for many end-users . a novel task is proposed to generate spreadsheet formulae from a user's NL query . |
| Approach: | They propose a task to generate formulas that are grounded on a spreadsheet table given a Natural Language query as input. |
| Outcome: | The proposed task generates formulas that are grounded on a spreadsheet table, given a natural language query as input. |