Papers by Pierre Colombo
Toward Stronger Textual Attack Detectors (2023.findings-emnlp)
Copied to clipboard
| Challenge: | despite the high performance of deep learning techniques, trained models remain vulnerable to adversarial attacks . authors present LAROUSSE, STAKEOUT and other approaches to detect adversarials . LARousSE is unsupervised, hyperparameter free and non-differentiable . |
| Approach: | They propose a framework to detect adversarial attacks and an extended benchmark to test them . they demonstrate that LAROUSSE outperforms previous methods and allows to identify interesting factor of detection rate variations. |
| Outcome: | The proposed framework outperforms existing methods and allows to identify interesting factor of detection rate variations. |
Code-switched inspired losses for spoken dialog representations (2021.emnlp-main)
Copied to clipboard
| Challenge: | We introduce new pretraining losses tailored to learn generic multilingual spoken dialogue representations . goal is to expose model to code-switched language . |
| Approach: | They propose to build a pretraining corpus of multilingual conversations in five different languages from OpenSubtitles. |
| Outcome: | The proposed models perform better in monolingual and multilingual settings. |
The importance of fillers for text representations of speech transcripts (2020.emnlp-main)
Copied to clipboard
| Challenge: | Fillers are a type of disfluency that can be a sound ("um" or "uh") filling a pause in an utterance or conversation. |
| Approach: | They propose to represent fillers with deep contextualised embeddings to improve modelling of spoken language and two downstream tasks . |
| Outcome: | The proposed representations improve modelling of spoken language and two downstream tasks, predicting a speaker’s stance and expressed confidence. |
Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern document retrieval embedding methods typically encode passages (chunks) from documents independently, often overlooking contextual information from the rest of the document. |
| Approach: | They propose a benchmark to evaluate retrieval models' ability to leverage document-wide context. |
| Outcome: | The proposed method significantly improves retrieval quality on ConTEB without sacrificing base model performance. |
Enhanced Hallucination Detection in Neural Machine Translation through Simple Detector Aggregation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) systems suffer from various pathologies, including the generation of translations that are detached from the source content, typically known as hallucinations. |
| Approach: | They propose to combine detectors and introduce a method for aggregating detectors to detect hallucinations. |
| Outcome: | The proposed method provides a promising step towards evermore reliable machine translation systems. |
Learning Disentangled Textual Representations via Statistical Measures of Similarity (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to disentangle a sensitive attribute from textual representations require training and multiple parameter updates. |
| Approach: | They propose a family of regularizers for learning disentangled representations that do not require training. |
| Outcome: | The proposed regularizers are faster and faster and achieve better results when combined with pretrained and randomly initialized text encoders. |
Hierarchical Pre-training for Sequence Labelling in Spoken Dialog (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Sequence labelling tasks like Dialog Act and Emotion/Sentiment identification are a key component of spoken dialog systems. |
| Approach: | They propose a new approach to learn generic representations adapted to spoken dialog using a hierarchical encoder based on transformer architectures. |
| Outcome: | The proposed model is model-agnostic and contains 10 datasets of various sizes. |
Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Neural machine translation models can unpredictably produce severely pathological translations, known as hallucinations, that seriously undermine user trust. |
| Approach: | They propose a fully unsupervised, plug-in detector that can be used with any attention-based NMT model. |
| Outcome: | The proposed detector outperforms existing models and is competitive with detectors that employ external models trained on millions of samples. |
A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for learning disentangled representations of textual data are difficult to implement and suffer from the degeneracy of other losses in multi-class scenarios. |
| Approach: | They propose a variational upper bound to the mutual information between an attribute and the latent code of an encoder that controls the approximation error. |
| Outcome: | The proposed method is superior on fair classification and on textual style transfer tasks. |
RainProof: An Umbrella to Shield Text Generator from Out-Of-Distribution Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Out-of-distribution (OOD) detection is a widely covered topic in classification tasks, but most methods rely on hidden features output by the encoder. |
| Approach: | They propose to leverage soft-probabilities in a black-box framework to detect OOD . they propose to use a more operational evaluation setting to enable OOD detection . |
| Outcome: | The proposed framework can access soft-predictions but not the internal states of the model. |
Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applications (2023.emnlp-main)
Copied to clipboard
| Challenge: | Instruction fine-tuned (IFT) models are gaining traction in industrial NLP to unlock task-specific performance gains and strengthen model alignment with industry requirements. |
| Approach: | They propose to use instruction fine-tuned (IFT) models to enhance the zero-shot capabilities of Large Language Models (LLMs) they also propose to leverage IFT models to analyze the trade-offs that emerge in industrial settings. |
| Outcome: | The proposed model is well adapted to new evaluation metric requirements, and offers practical insights for real-world LLM deployment. |
Affect-Driven Dialog Generation (N19-1)
Copied to clipboard
| Challenge: | Existing systems for end-to-end dialog generation focus on response quality without explicit control over affective content of the responses. |
| Approach: | They propose an affect-driven dialog system which generates emotional responses using a continuous representation of emotions. |
| Outcome: | The proposed system outperforms existing systems in terms of BLEU score and response diversity, and qualitative measures. |
From the Token to the Review: A Hierarchical Multimodal approach to Opinion Mining (D19-1)
Copied to clipboard
| Challenge: | Existing work on fine grained opinion annotations rely only on coarsely labeled opinions. |
| Approach: | They propose to use hierarchical structure of opinions to build a fine and coarse grained opinion model that exploits different views of the opinion expression. |
| Outcome: | The proposed model outperforms existing models on a recently released multimodal fine grained annotated corpus on IMDB and social networks. |
Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarking approaches assume that all systems have scores available for all tasks, which is not always practical. |
| Approach: | They propose a method to benchmark when some systems have scores missing on a task . they use a compatible partial ranking approach to impute missing data . |
| Outcome: | The proposed method is validated on 131 million scores, larger than existing benchmarks. |
Improving Multimodal fusion via Mutual Dependency Maximisation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal sentiment analysis is a trending area of research, and multimodal fusion is one of its most active topics. |
| Approach: | They propose to use modality-based penalties to measure dependency between models to improve accuracy. |
| Outcome: | The proposed methods improve accuracy on two well-known sentiment analysis datasets by 4.3 on the proposed models and by-product includes a statistical network which can interpret the high dimensional representations learnt by the model. |
Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021.emnlp-main)
Copied to clipboard
| Challenge: | metric BaryScore is used to evaluate text generation based on deep contextualized embeddings. |
| Approach: | They propose to model the layer output of deep contextualized embeddings as a probability distribution rather than a vector embeddable layer. |
| Outcome: | The proposed metric outperforms other BERT based metrics and exhibits more consistent behaviour in particular for text summarization. |
Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies on automatic story generation (ASG) rely on human criteria, but there is little research on how well they correlate with human criteria. |
| Approach: | They propose to use human criteria to evaluate automatic story generation (ASG) their paper proposes to use HANNA to quantitatively evaluate correlations between 72 automatic metrics and human criteria. |
| Outcome: | The proposed model compared human criteria with automatic criteria and found that they were significantly better than human criteria. |
Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning (2023.findings-emnlp)
Copied to clipboard
Duarte Alves, Nuno Guerreiro, João Alves, José Pombal, Ricardo Rei, José de Souza, Pierre Colombo, Andre Martins
| Challenge: | Large language models (LLMs) are a promising avenue for machine translation (MT) however, their effectiveness depends on the choice of few-shot examples and they often require extra post-processing due to overgeneration. |
| Approach: | They propose a method that incorporates few-shot examples during finetuning to improve performance on MT tasks. |
| Outcome: | The proposed method outperforms few-shot prompting while eliminating the need for in-context examples. |
Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models (2023.emnlp-main)
Copied to clipboard
Pierre Colombo, Victor Pellegrain, Malik Boudiaf, Myriam Tami, Victor Storchan, Ismail Ayed, Pablo Piantanida
| Challenge: | Proprietary and closed APIs are impacting the practical applications of natural language processing. |
| Approach: | They propose a scenario where a pre-trained model is served through a gated API . they propose 'transductive inference' that leverages statistics of unlabelled data . |
| Outcome: | The proposed model performs a few-shot classification task with unlabelled data using a gated API . the proposed model can be used to perform the task with a handful of classes . |