Papers by Haritz Puerto
MetaQA: Combining Expert Agents for Multi-Skill Question Answering (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent explosion of question-answering datasets and models has increased interest in generalization of models across multiple domains and formats. |
| Approach: | They propose to combine expert agents with a flexible and training-efficient architecture that considers questions, answer predictions, and answer-prediction confidence scores to select the best answer among a list of answer predictions. |
| Outcome: | The proposed model outperforms previous multi-agent and multi-dataset approaches and is highly data-efficient to train and adaptable to any QA format. |
UKP-SQuARE v2: Explainability and Adversarial Attacks for Trustworthy QA (2022.aacl-demo)
Copied to clipboard
Rachneet Sachdeva, Haritz Puerto, Tim Baumgärtner, Sewin Tariverdian, Hao Zhang, Kexin Wang, Hossain Shaikh Saadi, Leonardo F. R. Ribeiro, Iryna Gurevych
| Challenge: | Question Answering (QA) systems rely on deep neural networks, which are difficult to interpret by humans. |
| Approach: | They propose an interpretable model that provides an explanation infrastructure for comparing models based on saliency maps and graph-based explanations. |
| Outcome: | The proposed methods can be used to compare models based on saliency maps and graph-based explanations. |
Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large reasoning models (LRMs) are being adopted more widely as personal agents thanks to their enhanced planning skills enabled by reasoning traces (RTs). |
| Approach: | They propose to increase the budget of models with increased reasoning steps to amplify such leakage by enlarging their internal thinking to the model's internal thinking. |
| Outcome: | The proposed model can reason more verbosely and leak more in their own thinking, while improving utility but enlarges the privacy attack surface. |
Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to generate multiple independent CoTs, combining them through ensembling or other post-hoc strategies, have been shown to be effective in boosting performance. |
| Approach: | They propose a method where LLMs are fine-tuned to generate a sequence of Diverse Chains of Thought (DCoT) within a single inference step. |
| Outcome: | The proposed model can generate multiple chains of thought within a single inference step without external feedback. |
Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Membership inference attacks (MIAs) attempt to verify the membership of a data sample in the training set for a model. |
| Approach: | They propose to use membership inference attacks to verify the membership of a given data sample in a model training set. |
| Outcome: | The proposed methods are based on a new benchmark that measures the performance of membership inference attacks on large language models at a continuous scale. |
Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent prompting techniques have improved LLMs’ performance on various reasoning tasks, but there is little understanding of what triggers reasoning abilities in LLM in the inference stage. |
| Approach: | They propose a method that transforms a natural language problem into code and directly prompts the LLM using the generated code without resorting to external code execution. |
| Outcome: | The proposed method boosts multiple LLMs by 22.52 percentage points on GPT 3.5, 7.75 on Mixtral, and 16.78 on Mistral. |
UKP-SQUARE: An Online Platform for Question Answering Research (2022.acl-demo)
Copied to clipboard
Tim Baumgärtner, Kexin Wang, Rachneet Sachdeva, Gregor Geigle, Max Eichler, Clifton Poth, Hannah Sterz, Haritz Puerto, Leonardo F. R. Ribeiro, Jonas Pfeiffer, Nils Reimers, Gözde Şahin, Iryna Gurevych
| Challenge: | Recent advances in NLP and information retrieval have given rise to a diverse set of question answering tasks that are of different formats and require different model architectures and setups. |
| Approach: | They propose an extensible online QA platform that allows users to query and analyze a large collection of modern Skills via a user-friendly web interface and integrated behavioural tests. |
| Outcome: | The proposed tool allows users to query and analyze a large collection of modern Skills via a user-friendly web interface and integrated behavioural tests. |
UKP-SQuARE v3: A Platform for Multi-Agent QA Research (2023.acl-demo)
Copied to clipboard
Haritz Puerto, Tim Baumgärtner, Rachneet Sachdeva, Haishuo Fang, Hao Zhang, Sewin Tariverdian, Kexin Wang, Iryna Gurevych
| Challenge: | Current approaches to QA models are multi-dataset models, but combining expert agents can yield large performance gains over multi-agent models. |
| Approach: | They extend an online platform for QA research to support three families of multi-agent systems: agent selection, early-fusion of agents, and late-fusion. |
| Outcome: | The proposed model can be compared with multi-dataset models and achieve high inference speed and performance. |