Papers by Tianyu Jiang
Copied to clipboard
| Challenge: | a new task is designed to learn the prototypical uses of human-made physical objects . human beings are creative, and they create things for a reason . humans often infer that the object will be used in the most prototypical way unless told otherwise . |
| Approach: | They propose a task to learn the prototypical uses for human-made physical objects . they use frames from FrameNet to represent a set of common functions for objects based on their prototypical function . |
| Outcome: | The proposed task uses masked patterns to model prototypical uses for objects . the proposed model predicts the prototypical functions of objects and can be used to make models . |
Copied to clipboard
| Challenge: | MusicAgent integrates numerous music-related tools and an autonomous workflow to address user requirements. |
| Approach: | a new system is built to integrate music-related tools and an autonomous workflow . the system is based on large language models (LLMs) that can be used to organize and decompose requests . |
| Outcome: | the proposed system integrates numerous music-related tools and an autonomous workflow to address user requirements. |
Copied to clipboard
| Challenge: | Existing work on question-answer extraction fails to integrate incomplete utterances from dialog context for composite QA retrieval. |
| Approach: | They propose a task where questions and corresponding answers might be separated across different utterances. |
| Outcome: | The proposed methods perform well on 5 customer service datasets and set a benchmark for N-to-N DialogQAE with utterance and session level evaluation metrics. |
Copied to clipboard
| Challenge: | Several recent papers claim to have achieved human parity at sentence-level machine translation. |
| Approach: | They propose to use a dataset with rich discourse annotations to evaluate MT performance . they find that MT outputs differ fundamentally from human translations in terms of latent discourse structures. |
| Outcome: | The proposed dataset builds upon the large-scale parallel corpus BWB . it covers 15,095 entity mentions in both languages and compares them to human translations . |
Copied to clipboard
| Challenge: | Large Language Models exhibit strong capabilities in single-turn instruction following but suffer from Lost-in-Conversation (LiC) when instructions are revealed progressively in multi-turn settings, models get "Lost in Conversation" |
| Approach: | They propose a framework that encourages models to generate correct answers and judge solvability in multi-turn conversations. |
| Outcome: | The proposed framework improves models' ability to balance problem-solving with abstention . it reduces premature answering behaviors that cause lost-in-conversation (LiC) |
Copied to clipboard
| Challenge: | Currently, most research focuses on the bidding algorithms used within auction mechanisms. |
| Approach: | They propose a personalized valuation framework that integrates Large Language Models to incorporate personalized semantic preference into users valuation process. |
| Outcome: | The proposed framework incorporates Large Language Models to incorporate personalized semantic preference into users valuation process. |
Copied to clipboard
| Challenge: | Recent years have seen a paradigm shift in NLP towards using pretrained language models for a wide range of tasks. |
| Approach: | They propose to model structures as sequences of actions in autoregressive manner with PLMs . their approach allows in-structure dependencies to be learned without any loss . |
| Outcome: | The proposed approach achieves state-of-the-art on all structured prediction tasks. |
Copied to clipboard
| Challenge: | Rhetorical questions are asked not to seek information, but to persuade or signal stance . how large language models internally represent rhetorical questions remains unclear . |
| Approach: | They analyze rhetorical questions in LLM representations using linear probes on two social-media datasets with different discourse contexts. |
| Outcome: | The results show that rhetorical signals emerge early and are most stably captured by last-token representations. |
Copied to clipboard
| Challenge: | Existing methods to predict missing facts in knowledge graphs are limited in language alignment . SS-AGA uses seed alignment as an edge type to fuses all KGs as a whole graph . |
| Approach: | They propose a self-supervised adaptive graph alignment method that fuses all KGs as a whole graph by regarding alignment as 'a new edge type' they propose SS-AGA method that uses relation-aware attention weights to capture potential alignment pairs in a new paradigm. |
| Outcome: | The proposed method can predict missing facts in a knowledge graph (KG) but language alignment is scarce and new alignment identification is noisy. |
Copied to clipboard
| Challenge: | a frame-semantic parsing task is to determine which frame best captures the meaning of a word or phrase in a sentence. |
| Approach: | They propose a frame identification model that generates representations for frames and lexical units (senses) they evaluate the model on three data sets and show it consistently achieves better performance than previous systems. |
| Outcome: | The proposed model consistently outperforms previous systems on three data sets. |
Copied to clipboard
| Challenge: | Prior research has focused on learning the prototypical functions of physical objects . but many sentences refer to objects even when they are not used . |
| Approach: | They propose a task that determines whether a physical object mentioned in a sentence was used or likely will be used. |
| Outcome: | The proposed model exploits data augmentation methods and FrameNet to fine-tune a pre-trainedmodel. |
Copied to clipboard
| Challenge: | prevailing methods for machine translation are often hindered by misleading reward signals. |
| Approach: | They propose a framework that aligns large language models to human preferences . they propose 'M2PO' to correct the bias towards partial errors . |
| Outcome: | The proposed framework outperforms open-source models and achieves parity with proprietary models. |
Copied to clipboard
| Challenge: | Large language models excel at factual recall, arithmetic reasoning, multi-turn dialogue . their capacity as askers, formulating strategic, adaptive, and information-seeking questions, remains less explored . |
| Approach: | They propose a protocol for evaluating large language models as strategic question-askers . they propose entropy-based methods that filter candidates via ConceptNet and Bayesian method that tracks belief updates over semantic concepts . |
| Outcome: | The proposed method is model-agnostic and supports post hoc analysis. |
Copied to clipboard
| Challenge: | Static concreteness ratings are widely used in NLP, yet a word’s concreteness can shift with context, especially in figurative language such as metaphor, where common concrete nouns can take abstract interpretations. |
| Approach: | They conduct a layer-wise and geometric analysis of LLM hidden representations across four model families to examine how models distinguish literal vs. figurative usage. |
| Outcome: | The results show that LLMs separate literal and figurative usage in early layers and that mid-to-late layers compress concreteness into a one-dimensional direction consistent across models. |
Copied to clipboard
| Challenge: | Large language models have achieved remarkable success across a wide range of tasks, yet their performance remains heavily biased toward high-resource languages. |
| Approach: | They propose a pipeline for advancing Tibetan language modeling through multilingual continual pre-training with Tibetan, Chinese, and English. |
| Outcome: | The proposed model outperforms open-source and Tibetan-focused models on diverse tasks. |
Copied to clipboard
| Challenge: | Emotions manifest through physical experiences and bodily reactions, yet identifying such embodied emotions in text remains understudied. |
| Approach: | They propose to extend existing binary embodied emotion dataset with Ekman’s six basic emotion categories. |
| Outcome: | The proposed dataset outperforms existing methods with large language models. |
Copied to clipboard
| Challenge: | Existing work focuses on learning deep NER models with weak supervision without any human annotation. |
| Approach: | They propose a framework that can suppress the noise of the weak labels and fine-tune over the strongly labeled data. |
| Outcome: | The proposed framework outperforms existing methods on Named Entity Recognition tasks with weak supervision and weakly labeled data. |
Copied to clipboard
| Challenge: | Existing evaluation methods for visual activity recognition systems fail to capture ambiguities in verb semantics and image interpretation. |
| Approach: | They propose a framework that constructs verb sense clusters to evaluate visual activity recognition systems. |
| Outcome: | The proposed framework provides a more robust evaluation of visual activity recognition systems. |
Copied to clipboard
| Challenge: | Despite LLMs' impressive capabilities in musical knowledge, music reasoning remains an unsolved task. |
| Approach: | They propose an open-source large language model (LLM) that integrates intrinsic musical abilities into LLaMA2 and GPT-3.5. |
| Outcome: | The proposed model can understand and generate music with a pure text tokenizer without external multi-modal neural structures or tokenizers. |
Copied to clipboard
| Challenge: | enabling pre-trained models inference on ciphertext data is difficult due to the complex computations in transformer blocks. |
| Approach: | They propose an approximation approach for transformers which enables inference on ciphertext data. |
| Outcome: | The proposed approach can infer pre-trained models on encrypted data with negligible performance drop but enjoy theory-guaranteed privacy-preserving advantage. |
Copied to clipboard
| Challenge: | Recent systems enforce explicit ethical constraints, but moral judgment rarely involves such clear-cut prohibitions. |
| Approach: | They develop an emotion-induction pipeline that infuses emotion into moral situations and evaluate shifts in moral acceptability across datasets and LLMs. |
| Outcome: | The proposed pipeline can infuses emotion into moral situations and evaluate moral acceptability shifts across datasets and LLMs. |
Copied to clipboard
| Challenge: | a goal-act is an activity that represents a common reason people go to a location . recognizing goals is essential for narrative text understanding and story comprehension . |
| Approach: | They use a text corpus and semi-supervised learning to learn goal-acts for specific locations . they extract activities and locations that co-occur in goal-oriented syntactic patterns . |
| Outcome: | The proposed method outperforms baseline methods when judged against goal-acts identified by human annotators. |
Copied to clipboard
| Challenge: | ELENA is a framework for embodied emotion analysis using large vision language models . ELEna uses attention maps and a persistent bias towards the facial region . |
| Approach: | They propose a framework that utilizes large vision language models to generate ELENA . they propose to use attention maps to describe emotional reactions from body parts . |
| Outcome: | The proposed framework outperforms baseline models without fine-tuning . it uses large vision language models to generate embodied emotion narratives . |
Copied to clipboard
| Challenge: | Current Text-to-SQL reasoning models lack integrated execution feedback during generation. |
| Approach: | They propose a text-to-SQL framework that interacts with the SQL execution engine during decoding and dynamically adjusts reasoning based on execution feedback. |
| Outcome: | The proposed framework achieves 89.1% accuracy on Spider and 65.3% on BIRD at the 7B scale. |
Copied to clipboard
| Challenge: | Existing studies treat prompts as flat text, overlooking their internal structure, and different components within a prompt contribute unequally to robustness. |
| Approach: | They propose a framework that decomposes prompts into functional components and a method that selectively modifies components to expose component-wise vulnerabilities. |
| Outcome: | The proposed framework exposes component-wise vulnerabilities while ensuring linguistic plausibility through perplexity-based filtering. |
Copied to clipboard
| Challenge: | Existing tasks that aim to identify the objects in an image are object detection and image classification, but recent work has focused on more comprehensive image under- standing tasks. |
| Approach: | They propose to incorporate commonsense knowledge about physical objects into a transformer-based model that is trained to predict the actionverb for visual activity recognition. |
| Outcome: | The proposed model incorporates prototypical function knowledge about physical objects to predict the actionverb for visual activity recognition. |
Copied to clipboard
| Challenge: | Standard evaluation metrics, e.g., BLEU, TER and METEOR, focus on the quality of translations at the sentence level and do not consider discourse-level features. |
| Approach: | They propose to use a metric to take discourse coherence into consideration by categorizing discourse-related spans and calculating the similarity-based F1 measure of categorized spans. |
| Outcome: | The proposed metric possesses better selectivity and interpretability at the document-level, and is more sensitive to document- level nuances. |
Copied to clipboard
| Challenge: | a typical approach to natural language processing tasks involves selecting text spans and making decisions about them. |
| Approach: | They propose a grammar-based structured span selection model which learns to make use of partial span annotations. |
| Outcome: | The proposed model improves on two popular span prediction tasks. |
Copied to clipboard
| Challenge: | Existing Chinese preference datasets suffer from limited scale, restricted domain coverage, and insufficiently rigorous data validation. |
| Approach: | They propose an LLM-based data annotation pipeline with no human intervention to annotate Chinese preference datasets. |
| Outcome: | The proposed pipeline outperforms existing Chinese preference datasets on AlignBench and Chinese Reward Benchmark. |
Copied to clipboard
| Challenge: | Existing efforts focus on activation within ongoing dialogues, while overlooking a key real-world bottleneck. |
| Approach: | They propose a conversation starter generation system that generates personalized starters to guide users into conversation without explicit user intent. |
| Outcome: | The proposed system improves user active days by +1.84 and click-through rate by +94.25 and has been deployed in production. |
Copied to clipboard
| Challenge: | Prior work on recognizing affective events focused on producing lexical resources of verbs or event phrases with corresponding affective polarity values. |
| Approach: | They propose a BERT-based model for affective event classification and a discourse-enhanced self-training method that iteratively improves the classifier with unlabeled data. |
| Outcome: | The proposed model outperforms existing models with unlabeled data and improves recall and precision. |
Copied to clipboard
| Challenge: | Using the FrameNet lexical resource, we evaluate large language models under prompt-based inference and observe that they can perform frame identification effectively even without explicit supervision. |
| Approach: | They evaluate large language models under prompt-based inference and observe that they encode latent knowledge of frame semantics. |
| Outcome: | The proposed model can generate coherent frame definitions while generalizing well to out-of-domain benchmarks. |
Copied to clipboard
| Challenge: | Verbal multiword expressions (VMWEs) are difficult for machine translation because their meanings are often not recoverable from their component words. |
| Approach: | They analyze the impact of verbal idioms, verb-particle constructions, and light verb constructions on machine translation quality from English to multiple languages. |
| Outcome: | The proposed system improves translation quality by focusing on verb idioms, verb-particle constructions and light verb constructions. |
Copied to clipboard
| Challenge: | Large language models struggle with natural language to first order logic (NL2FOL) translation due to logical hallucination. |
| Approach: | They propose a fine-tuning free framework to correct hidden state deviation by leveraging logical stability across logic preserving perturbations of the input. |
| Outcome: | The proposed framework improves logical consistency during inference and improves accuracy over baselines. |
Copied to clipboard
| Challenge: | a new task is needed to recognize physical manifestations of emotions in natural language . physical manifestation of emotions affects not only our mental state but also our physical state . |
| Approach: | They propose a task to recognize expressions of embodied emotion in natural language . they use body part mentions with human annotations to extract emotional manner expressions . |
| Outcome: | The proposed model can train without gold data and improve performance with gold data. |
Copied to clipboard
| Challenge: | Prior work on metonymy resolution has focused on named entities, but common nouns are also a frequent problem. |
| Approach: | They propose a dataset that combines a metonymy dataset and a chain-of-thought based prompting method for detecting metonyms using large language models. |
| Outcome: | The proposed method can detect metonymy using large language models while still struggling with nuanced semantic understanding. |
Copied to clipboard
| Challenge: | Current systems often fall short of this goal in settings where translation hinges on culturally grounded entities such as books, films, places, songs and idioms. |
| Approach: | They propose a framework that anchors supervision on a verifiable, entity-level reward signal and incorporates lightweight structural gates to stabilize optimization. |
| Outcome: | The proposed framework improves on XC-Translate and shows that it can learn a robust reasoning process rather than imitating reference translations. |
Copied to clipboard
| Challenge: | Low-resource language tokens are often routed to different experts than those activated by high-resourced inputs, which hinders their efficacy in multilingual contexts. |
| Approach: | They propose a framework to transfer specialized capabilities from high-resource languages as anchors to low-resourced languages by using a symmetric Jensen-Shannon constraint. |
| Outcome: | The proposed framework outperforms standard instruction tuning on 5 low-resource languages and 3 benchmarks. |
Copied to clipboard
| Challenge: | Existing audio-visual speech recognition systems suffer from a temporal gap . visual speech patterns captured from lip movements provide complementary information that remains inherently robust to acoustic noise. |
| Approach: | They propose a framework that deeply stacks temporal tokens across both encoding and decoding stages to bridge this temporal gap. |
| Outcome: | The proposed framework outperforms existing supervised, self-supervised, and LLM-based methods by 6.1% on LRS2 and 7.8% on LLS3. |
Copied to clipboard
| Challenge: | Metonymy and metaphor are two fundamental linguistic phenomena in figurative language that involve concept mapping. |
| Approach: | They propose a framework that transforms a literal sentence into three figurative variants . they propose 'metonymic, metaphoric, and hybrid' datasets that can be used to map metonymy and metaphor . |
| Outcome: | The proposed framework transforms a literal sentence into three figurative variants . hybrid examples yield the largest gains on metonymy tasks, the study shows . |
Copied to clipboard
| Challenge: | Visual metonymy is a form of indirect representation in which an image evokes a concept not by depicting it directly, but by presenting visually associated cues that invite the viewer to infer the intended meaning. |
| Approach: | They propose a pipeline grounded in semiotic theory that leverages large language models and text-to-image models to generate metonymic visual representations. |
| Outcome: | The proposed pipeline exploits large language models and text-to-image models to generate metonymic visual representations. |
Copied to clipboard
| Challenge: | Existing model editing methods focus on single-round editing and often face significant challenges in sequential model editing. |
| Approach: | They propose a model editing method that optimizes the target layer’s hidden states using the model’s original weights to prevent model failure. |
| Outcome: | The proposed method outperforms existing model editing methods and is available on the open-source platform 4open.science. |