Papers by Da Yin
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022.emnlp-main)
Copied to clipboard
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, Jiawei Han
| Challenge: | Existing evaluation frameworks for natural language generation are dominated by similarity-based metrics. |
| Approach: | They propose a multi-dimensional evaluator for natural language generation that integrates multiple dimensions into one evaluer. |
| Outcome: | The proposed evaluator improves on three typical NLG tasks and improves with external knowledge. |
GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that Pre-trained Language Models store relational knowledge and utilize it for performing downstream tasks. |
| Approach: | They propose a benchmark dataset for probing the diversity of relational knowledge in multilingual PLMs. |
| Outcome: | The proposed dataset contains 3125 prompts in English, Chinese, Hindi, Persian, and Swahili . larger multilingual PLMs variants do not store geo-diverse concepts better than its smaller variant . |
Agent Lumos: Unified and Modular Training for Open-Source Language Agents (2024.acl-long)
Copied to clipboard
Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, Bill Yuchen Lin
| Challenge: | Lumos is a framework for training open-source agents on complex interactive tasks. |
| Approach: | They propose a framework for training open-source LLM-based agents called Lumos . Lumos features a learnable, unified and modular architecture with a planning module that learns high-level subgoal generation and a grounding module trained to translate these into the actions using various tools in the execution module. |
| Outcome: | The framework outperforms open-source agents on QA and web tasks. |
Everything Has a Cause: Leveraging Causal Inference in Legal Text Analysis (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing studies focus on analyzing structured data, while mining causal relationship among factors from unstructured data is of great importance. |
| Approach: | They propose a graph-based causal inference framework which builds causal graphs from fact descriptions without much human involvement. |
| Outcome: | The proposed framework can capture nuance from fact descriptions among confusing charges and provide explainable discrimination in few-shot settings. |
The Magic of IF: Investigating Causal Reasoning Abilities in Large Language Models of Code (2023.findings-acl)
Copied to clipboard
| Challenge: | entailment a) |
| Approach: | entailment : We want to explore whether Code-LLMs with code prompts are better . encoding a code prompt is better than text-only LLMs, they say . |
| Outcome: | entailment : Our results show that Code-LLMs with code prompts are better compared to text-only LLMs. |
SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models that capture compositional sentiment semantics are difficult to handle when the sentences are complex. |
| Approach: | They propose a variant of BERT that captures compositional sentiment semantics . they demonstrate that SentiBERT can be applied to other sentiment analysis tasks . |
| Outcome: | The proposed model is better than baseline approaches in capturing negation and contrast . it can be applied to other sentiment analysis tasks and emotion classification tasks . |
Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Generally, commonsense knowledge is correlated with culture and geographic locations and is only shared locally. |
| Approach: | They construct a Geo-Diverse Visual Commonsense Reasoning dataset to test vision-and-language models’ ability to understand cultural and geo-location-specific commonsense. |
| Outcome: | The proposed models perform better in non-Western regions including East Asia, South Asia, and Africa than in the Western regions. |
KPEval: Towards Fine-Grained Semantic-Based Keyphrase Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods for keyphrase extraction and generation rely on exact matching with human references. |
| Approach: | They propose a framework for evaluation that includes four critical aspects: reference agreement, faithfulness, diversity, utility and semantic-based metrics. |
| Outcome: | The proposed evaluation framework correlates better with human preferences than previously proposed metrics. |
Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for instruction tuning do not include associating instructions with existing datasets. |
| Approach: | They propose a dynamic growth paradigm for the automatic curation of instruction-tuning data . they use existing datasets to automatically construct instruction-uning datasets . |
| Outcome: | The proposed model reduces the API cost for generating instructions and provides high-quality data. |
QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization (2021.naacl-main)
Copied to clipboard
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, Dragomir Radev
| Challenge: | Existing work on meeting summarization tasks is limited to short summaries that cover all the content of a meeting. |
| Approach: | They propose a query-based multi-domain meeting summarization task that generates a single short summary of meetings based on a transcript. |
| Outcome: | The proposed task is based on 1,808 query-summary pairs over 232 meetings in multiple domains. |
How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Text-to-image generative models can generate high-quality photo-realistic images conditional on natural language text descriptions in a zero-shot fashion. |
| Approach: | They propose an Ethical NaTural Language Interventions in Text-to-Image GENeration benchmark dataset to evaluate the change in image generation conditional on ethical interventions across three social axes – gender, skin color, and culture. |
| Outcome: | The proposed model generations cover diverse social groups while preserving image quality. |
Guiding Through Complexity: What Makes Good Supervision for Hard Reasoning Tasks? (2025.naacl-long)
Copied to clipboard
| Challenge: | Using weak teacher models to effectively supervise LLMs can improve performance on hard reasoning tasks. |
| Approach: | They propose two data-driven supervision strategies that offer supervision data at different quality levels upon tasks of varying complexity. |
| Outcome: | The proposed methods outperform "perfectly correct" supervision on harder subtasks even when the outcome error rate is high. |
Things not Written in Text: Exploring Spatial Commonsense from Visual Signals (2022.acl-long)
Copied to clipboard
| Challenge: | Pretrained language models fail in many NLP tasks, but are ineffective in spatial commonsense reasoning. |
| Approach: | They propose a spatial commonsense benchmark that focuses on relative scales of objects and the positional relationship between people and objects under different actions. |
| Outcome: | The proposed framework outperforms pretrained models in answering spatial questions. |
What Does BERT with Vision Look At? (2020.acl-main)
Copied to clipboard
| Challenge: | Pre-trained visual grounded language models have improved performance on vision-and-language tasks but what they learn during pre-training remains unclear. |
| Approach: | They show that attention heads of visual grounded language models actively ground elements of language to image regions. |
| Outcome: | The attention heads of a visual grounded language model can ground elements to image regions, demonstrating their ability to detect syntactic relations between non-entity words and image regions. |
Trial and Error: Exploration-Based Trajectory Optimization of LLM Agents (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have become integral components in various autonomous agent systems. |
| Approach: | They propose an exploration-based trajectory optimization approach that allows agents to learn from their exploration failures. |
| Outcome: | The proposed method outperforms baseline methods on three complex tasks by a large margin. |