Papers by Semih Yavuz
Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Cross-lingual transfer of language models trained on high-resource languages such as English has been limited due to the high cost of obtaining non-English conversational data. |
| Approach: | They introduce a parallel and large-scale multilingual conversation dataset that is used for cross-lingual alignment pretraining by translating the English-only Schema-Guided Dialogue dataset into 105 other languages. |
| Outcome: | The proposed model performs well on slot-filling and intent classification tasks, and is able to perform well in other languages. |
FOLIO: Natural Language Reasoning with First-Order Logic (2024.emnlp-main)
Copied to clipboard
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, Dragomir Radev
| Challenge: | Existing benchmarks for logical reasoning in large language models lack language naturalness or limited complexity. |
| Approach: | They propose to use first-order logic annotations to evaluate logical reasoning capabilities of large language models. |
| Outcome: | The proposed dataset evaluates the FOL reasoning ability of supervised fine-tuning on medium-sized language models. |
DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | Retrievalaugmented LLMs have been used to ground LLM in external knowledge . a gap exists in the current landscape regarding the effectiveness of grounding LLM on heterogeneous knowledge sources. |
| Approach: | They propose a model that uses symbolic language to generate symbolic queries . they use a dataset that is generated using predefined reasoning chains and human annotation . |
| Outcome: | The proposed model outperforms previous approaches by a significant margin in QA tasks over text. |
P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chains (2024.findings-emnlp)
Copied to clipboard
Simeng Han, Aaron Yu, Rui Shen, Zhenting Qi, Martin Riddell, Wenfei Zhou, Yujie Qiao, Yilun Zhao, Semih Yavuz, Ye Liu, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Dragomir Radev, Rex Ying, Arman Cohan
| Challenge: | Existing methods on understanding the capabilities of LLMs in logical reasoning rely on binary entailment classification or synthetically derived rationales. |
| Approach: | They propose to annotate a human-annotated dataset consisting of diverse and complex reasoning chains for a set of realistic logical reasoning stories also written by humans. |
| Outcome: | The proposed model outperforms existing methods on understanding the capabilities of LLMs in logical reasoning by 10% or more. |
Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Phrases are fundamental linguistic units through which humans convey semantics. |
| Approach: | They assess the capacity of API-based large language models to comprehend phrase semantics . they use three human-annotated datasets to analyze their results . |
| Outcome: | The proposed model outperforms embedding-based methods in phrase semantic reasoning tasks . the proposed model does not show significant advantage over fine-tuned methods . |
What It Takes to Achieve 100% Condition Accuracy on WikiSQL (D18-1)
Copied to clipboard
| Challenge: | despite of its simplicity, none of the publicly reported structured query generation models can achieve an accuracy beyond 62%, which is far from enough for practical use. |
| Approach: | They propose a model that can achieve 88.6% condition accuracy on WikiSQL . they ask: why is the accuracy still low for such simple queries? |
| Outcome: | The proposed solution can reach up to 88.6% condition accuracy on the WikiSQL dataset. |
Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Task-adaptive pre-training (TAPT) and Self-training can be complementary with simple TFS protocol. |
| Approach: | They propose to use task-adaptive pre-training and self-training to combine TAPT and ST with a simple TFS protocol to achieve strong combined gains across six datasets. |
| Outcome: | The proposed method can achieve strong combined gains across six datasets covering sentiment classification, paraphrase identification, natural language inference, named entity recognition and dialogue slot classification. |
Simple Data Augmentation with the Mask Token Improves Domain Adaptation for Dialog Act Tagging (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on DA tagging focus on human-human social conversations, which is less applicable for task-oriented setting. |
| Approach: | They propose a controllable mechanism that augments text input by leveraging the pre-trained Mask token from BERT model. |
| Outcome: | The proposed mechanism augments text input by leveraging the pre-trained Mask token from BERT model. |
Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors (2023.acl-long)
Copied to clipboard
Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, Greg Durrett
| Challenge: | Abstractive summarization systems still include factual errors in generated summaries despite recent improvements in factuality detection . |
| Approach: | They aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. |
| Outcome: | The proposed method improves on the ChatGPT-based model and shows that it is not superior for all error types. |
Modeling Multi-hop Question Answering as Single Sequence Prediction (2022.acl-long)
Copied to clipboard
| Challenge: | Existing generative question answering models that leverage passage retrieval with a pre-trained transformer are not effective for multihop QA. |
| Approach: | They propose a generative approach that explicitly models the reasoning process to resolve the answer for multi-hop questions by encoding cross-passage interactions. |
| Outcome: | The proposed model improves on two multi-hop QA datasets and is interpretable. |
CaLcs: Continuously Approximating Longest Common Subsequence for Sequence Level Optimization (D18-1)
Copied to clipboard
| Challenge: | Maximum-likelihood estimation (MLE) is widely used for text-generation based natural language processing applications. |
| Approach: | They propose a method to train models with maximum-likelihood estimation using a differentiable surrogate of longest common subsequence measure that captures sequence-level structure similarity. |
| Outcome: | Experimental results show that the proposed approach improves on the current MLE approach for downstream tasks like text summarization and machine translation. |
Dense Hierarchical Retrieval for Open-domain Question Answering (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Dense neural text retrieval has achieved promising results on open-domain Question Answering (QA) current dense retrievers require splitting documents into short passages that usually contain local, partial and sometimes biased context, and may yield inaccurate and misleading hidden representations, thus deteriorating the final retrieval result. |
| Approach: | They propose a hierarchical framework which can generate accurate dense representations of passages by utilizing both macroscopic semantics in the document and microscopic specific to each passage. |
| Outcome: | The proposed framework significantly outperforms the original dense passage retriever and helps an end-to-end QA system outperfect the strong baselines on multiple open-domain QA benchmarks. |
DialSQL: Dialogue Based Structured Query Generation (P18-1)
Copied to clipboard
| Challenge: | Recent advances in deep learning and semantic parsing have improved the translation accuracy of natural language questions to structured queries. |
| Approach: | They propose a dialogue-based structured query generation framework that leverages human intelligence to boost performance of existing algorithms via user interaction. |
| Outcome: | The proposed framework improves on a WikiSQL dataset from 61.3% to 69.0% using only 2.4 validation questions per dialogue. |
SharPT: Shared Latent Space Prompt Tuning (2023.findings-eacl)
Copied to clipboard
| Challenge: | Prompt tuning is an efficient method for adapting large language models, but it is difficult and expensive to identify the source task that provides optimal prompts. |
| Approach: | They propose to learn a shared latent space which captures a set of basis skills from a mixture of source tasks and then transfer them to target tasks. |
| Outcome: | The proposed method outperforms previous methods on NLI, sentence completion, QA, conference resolution, word sense disambiguation and on various model scales. |
RNG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering (2022.acl-long)
Copied to clipboard
| Challenge: | Existing KBQA approaches struggle with generalization of unseen KB schema items . Rank-and-generate approach solves coverage issue with strong generalization . |
| Approach: | They propose a Rank-and-Generate approach for KBQA that uses a generation model to generalize to unseen KB schema items. |
| Outcome: | The proposed approach outperforms the prior state-of-the-art on GrailQA and WebQSP datasets. |
Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset (D19-1)
Copied to clipboard
Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, Andy Cedilnik
| Challenge: | a lack of high quality conversational data is limiting progress in dialog systems . we present a dataset of 13,215 task-based dialogs . |
| Approach: | They propose a task-based dialog dataset which includes 13,215 task-related dialogs . they use a two-person, spoken "Wizard of Oz" approach and a "self-dialog" approach . |
| Outcome: | The taskmaster-1 dataset contains 13,215 task-based dialogs comprising six domains. |
Unsupervised Paraphrasing with Pretrained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Paraphrase generation has benefited from recent advances in the design of training objectives and model architectures, but previous studies focused on supervised methods that require a large amount of labeled data that is costly to collect. |
| Approach: | They propose a transfer learning approach that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. |
| Outcome: | The proposed model performs state-of-the-art on the Quora Question Pair and ParaNMT datasets and is robust to domain shift between the two datasets. |
Few-shot Unified Question Answering: Tuning Models or Prompts? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Question-answering (QA) tasks investigate specific question types, knowledge domains, or reasoning skills, leading to specialized models catering to specific categories of QA tasks. |
| Approach: | They propose to use model and prompt tuning for unified QA in a low-resource setting to overcome drawbacks of unified models. |
| Outcome: | The proposed model and prompt tuning paradigms outperform model tuning in a few-shot setting with a good initialization and achieve a significant performance boost from pre-training in 'low-resource' setting. |
Uni-Parser: Unified Semantic Parser for Question Answering on Knowledge Base and Database (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches on semantic parsing suffer from exponential growth of logical form candidates and can hardly generalize to unseen data. |
| Approach: | They propose a unified semantic parser for question answering on KB and DB . they define the primitive as the essential element in their framework . |
| Outcome: | The proposed framework can predict logical forms by altering and composing top-ranked primitives with different operations. |
Global Relation Embedding for Relation Extraction (N18-1)
Copied to clipboard
| Challenge: | Existing methods to extract textual relations with distant supervision are limited by their reliance on supervised training data. |
| Approach: | They propose to embed relations with global statistics of relations to combat the wrong labeling problem of distant supervision. |
| Outcome: | The proposed method is more robust to training noise introduced by distant supervision and improves relation extraction models. |
HPE: Answering Complex Questions over Text by Hybrid Question Parsing and Execution (2023.findings-emnlp)
Copied to clipboard
| Challenge: | End-to-end neural networks excel at answering natural language questions but fail on complex ones . a proposed framework for question parsing and execution on textual QA is designed to combine the strengths of neural and symbolic methods. |
| Approach: | They propose a framework for question parsing and execution on textual QA . they parse questions into an intermediate representation and use deterministic rules to translate them . |
| Outcome: | The proposed framework outperforms existing methods in supervised, few-shot, and zero-shot settings while preserving its underlying reasoning process. |
Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings (2025.acl-long)
Copied to clipboard
| Challenge: | Contextual evaluation is challenging for state-of-the-art judge models . evaluation criteria are often conditional and dependent on practitioner priorities . |
| Approach: | They propose a judge benchmark that evaluates large language models as judges in contexts . they use human annotations and model-based perturbations to build the benchmark . |
| Outcome: | The proposed benchmark aims to evaluate large language models in contexts with 2,000 challenging response pairs. |
Unlocking Anticipatory Text Generation: A Constrained Approach for Large Language Models Decoding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have shown a powerful ability for text generation, but undesired behaviors such as toxicity and hallucinations can manifest. |
| Approach: | They propose to formalize text generation as a future-constrained generation problem to minimize undesirable behaviors and enforce faithfulness to instructions. |
| Outcome: | The proposed approach is effective across three tasks, including keyword-constrained generation, toxicity reduction, and factual correctness in question-answering. |
Monotonic Infinite Lookback Attention for Simultaneous Machine Translation (P19-1)
Copied to clipboard
Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Cheng Chiu, Semih Yavuz, Ruoming Pang, Wei Li, Colin Raffel
| Challenge: | Simultaneous machine translation begins to translate each source sentence before the source speaker has finished speaking, with applications to live and streaming scenarios. |
| Approach: | They propose a simultaneous translation system that learns an adaptive schedule with a neural machine translation model that attends over all source tokens read thus far. |
| Outcome: | The proposed system can achieve latency-quality trade-offs favorable to a proposed wait-k strategy for many latency values. |
Improving the Faithfulness of Abstractive Summarization via Entity Coverage Control (2022.findings-naacl)
Copied to clipboard
| Challenge: | Abstractive summarization systems have been shown to be more prone to unfaithful facts . 30% of summaries generated by pre-trained language models suffer from hallucination . |
| Approach: | They propose a method to remedy entity-level extrinsic hallucinations with Entity Coverage Control . they first compute entity coverage precision and prepend the corresponding control code . a further fine-tuning is performed to unlock zero-shot summarization . |
| Outcome: | The proposed method leads to more faithful and salient abstractive summarization in fine-tuning and zero-shot settings. |
Stage-wise Fine-tuning for Graph-to-Text Generation (2021.acl-srw)
Copied to clipboard
| Challenge: | Graph-to-text generation has benefited from pre-trained language models (PLMs) but they fail to fully utilize the structure information of the input graph. |
| Approach: | They propose a structured graph-to-text model with a two-step fine-tuning mechanism which first fine-tracks model on Wikipedia before adapting to graph- to-text generation. |
| Outcome: | The proposed model improves the performance of the English WebNLG 2017 dataset by using tree-level embeddings to capture the inter-dependency structures of the input graph. |