Papers by Thomas Scialom
Self-Attention Architectures for Answer-Agnostic Neural Question Generation (P19-1)
Copied to clipboard
| Challenge: | Neural architectures based on self-attention have attracted interest from the research community . a recent study examined the performance of Transformers on a task of Neural Question Generation . |
| Approach: | They propose to adapt Transformers to a task of Neural Question Generation without constraining the model to focus on a specific answer passage. |
| Outcome: | The proposed architectures have obtained significant improvements over the state-of-the-art in several tasks. |
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2023.acl-long)
Copied to clipboard
| Challenge: | Instruction tuning enables pretrained language models to perform new tasks from inference-time natural language descriptions without human supervision. |
| Approach: | They propose to use a dataset of natural language instructions to generate large datasets with no human supervision. |
| Outcome: | The proposed dataset outperforms open-source models on various benchmarks, and is cost-effective. |
QuestEval: Summarization Asks for Fact-based Evaluation (2021.emnlp-main)
Copied to clipboard
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, Patrick Gallinari
| Challenge: | Existing evaluation metrics for summarization evaluation are limited and do not correlate well with human judgments. |
| Approach: | They propose to extend existing evaluation metrics to include question answering models to assess whether a summary contains all relevant information in its source document. |
| Outcome: | The proposed framework significantly improves the correlation with human judgments over four evaluation dimensions. |
Fine-tuned Language Models are Continual Learners (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on large language models relies on intuition that most tasks can be described via natural language instructions. |
| Approach: | They propose that a model should be able to keep extending its knowledge without forgetting previous skills. |
| Outcome: | The proposed model can learn 8 new diverse language generation tasks while maintaining good performance on previous tasks, spanning in total of 70 datasets. |
Ask to Learn: A Study on Curiosity-driven Question Generation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing work on Question Generation focuses on generating relevant questions given text with an answer . human ability to ask questions goes beyond evaluation of reading comprehension . |
| Approach: | They propose a novel text generation task based on a conversational question-asking dataset . they investigate automated metrics to measure different properties of Curious Questions . |
| Outcome: | The proposed task is based on a conversational Question Answering dataset . the results show that humans tend to ask questions with the goal of obtaining new information . |
MLSUM: The Multilingual Summarization Corpus (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing biases in multi-lingual datasets are limiting the use of multilingual data in document summarization tasks. |
| Approach: | They present MLSUM, the first large-scale MultiLingual SUMmarization dataset. |
| Outcome: | The proposed dataset contains 1.5M+ article/summary pairs in five different languages. |
Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve Question Answering performance on non-English data are expensive and limited to evaluation set. |
| Approach: | They propose a method to improve Question Answering performance without additional annotations by leveraging Question Generation models to produce synthetic samples in a cross-lingual fashion. |
| Outcome: | The proposed method outperforms baselines on four datasets in English significantly . the proposed model outperformed baselines in english and is comparable to the validation set of the original SQuAD. |
RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question (2023.findings-acl)
Copied to clipboard
Alireza Mohammadshahi, Thomas Scialom, Majid Yazdani, Pouya Yanki, Angela Fan, James Henderson, Marzieh Saeidi
| Challenge: | Existing metrics for evaluating the quality of automatically generated questions are expensive and penalise valid questions that may not have high lexical or semantic similarity to the reference questions. |
| Approach: | They propose a question-answering and span scorer metric based on the answerability of the candidate question given the context. |
| Outcome: | The proposed metric has higher correlation with human judgment without relying on the reference question. |
Skim-Attention: Learning to Focus via Document Layout (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to document understanding have high computational and memory costs. |
| Approach: | They propose a new attention mechanism that takes advantage of the structure of a document and its layout. |
| Outcome: | The proposed attention mechanism obtains lower perplexity than previous studies while being more computationally efficient. |
TRUE: Re-evaluating Factual Consistency Evaluation (2022.naacl-main)
Copied to clipboard
Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, Yossi Matias
| Challenge: | Grounded text generation systems often generate factual inconsistencies, hindering their real-world applicability. |
| Approach: | They propose a method to assess factual consistency metrics on standardized texts . they recommend NLI and question generation-and-answering-based methods as starting points . |
| Outcome: | The proposed method is more actionable and interpretable than previous methods. |
Toward Stance-based Personas for Opinionated Dialogues (2020.findings-emnlp)
Copied to clipboard
| Challenge: | chit-chat neural models lacking specificity and coherence, argues a new study on stance-based personas . stancebased personal representations lack generalization capability, allowing agents to sustain personal points of view both within the same conversation and across different discussions. |
| Approach: | They propose to investigate stance-based persona representations and their impact on claim generation by using a conversational dataset. |
| Outcome: | The proposed dataset shows that stance-based personas grasp abstract and profound aspects of the author persona. |
Project PIAF: Building a Native French Question-Answering Dataset (2020.lrec-1)
Copied to clipboard
Rachel Keraron, Guillaume Lancrenon, Mathilde Bras, Frédéric Allary, Gilles Moyse, Thomas Scialom, Edmundo-Pavel Soriano-Morales, Jacopo Staiano
| Challenge: | a lack of data for non-English languages is limiting the development of downstream tasks such as Question Answering. |
| Approach: | They propose to collect a native French Question Answering Dataset using a participatory setup. |
| Outcome: | The proposed tool allows volunteers to participate in crowdsourced annotations in French. |
Answers Unite! Unsupervised Metrics for Reinforced Summarization Models (D19-1)
Copied to clipboard
| Challenge: | Abstractive summarization approaches based on Reinforcement Learning (RL) have been proposed to overcome classical likelihood maximization. |
| Approach: | They propose to use Reinforcement Learning to learn the model parameters through RL techniques to overcome classical likelihood maximization. |
| Outcome: | The proposed measures favor ROUGE with the additional property of not requiring reference summaries. |
Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation (2021.emnlp-main)
Copied to clipboard
Clement Rebuffel, Thomas Scialom, Laure Soulier, Benjamin Piwowarski, Sylvain Lamprier, Jacopo Staiano, Geoffrey Scoutheeten, Patrick Gallinari
| Challenge: | QuestEval is a metric used in text-to-text tasks, but its adaptation to Data-to Text tasks requires multimodal Question Generation and Answering systems, which are seldom available. |
| Approach: | They propose to build synthetic multimodal corpora enabling to train multimodal components for a data-QuestEval metric. |
| Outcome: | The proposed method obtains state-of-the-art correlations with human judgment on the WebNLG and WikiBio benchmarks. |
QACE: Asking Questions to Evaluate an Image Caption (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing metric for image captioning evaluation is based on n-gram similarity metrics but these fail to capture semantic errors in captions. |
| Approach: | They propose a new metric based on Question Answering for Caption Evaluation to evaluate image captioning based upon Question Generation and Question Answers systems. |
| Outcome: | The proposed metric is multi-modal, reference-less and explainable. |
A Multifaceted Framework to Evaluate Evasion, Content Preservation, and Misattribution in Authorship Obfuscation Techniques (2022.emnlp-main)
Copied to clipboard
| Challenge: | Authorship obfuscation techniques are often evaluated based on their ability to hide the author’s identity (evasion) while preserving the content of the original text. |
| Approach: | They propose to evaluate authorship obfuscation techniques on detection evasion and content preservation using competitive identification techniques in real-life scenarios. |
| Outcome: | The proposed method reveals key weaknesses in state-of-the-art obfuscation techniques and surprisingly competitive effectiveness from a back-translation baseline in all evaluation aspects. |
LoRaLay: A Multilingual and Multimodal Dataset for Long Range and Layout-Aware Summarization (2023.eacl-main)
Copied to clipboard
| Challenge: | Text Summarization is a popular task and a challenge for neural models. |
| Approach: | They propose to exploit visual/layout information to capture long-range dependencies in summarization models by combining layout-aware and long-reaching models. |
| Outcome: | The proposed datasets cover French, Spanish, Portuguese, and Korean languages. |