Papers by Andre Freitas
TRACE: Training and Inference-Time Interpretability Analysis for Language Models (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Existing tools for interpretability analysis of transformer models are post hoc, rely on scalar metrics or require nontrivial integration effort. |
| Approach: | They propose a modular toolkit for training and inference-time interpretability analysis of transformer models. |
| Outcome: | Experiments with autoregressive transformers show that TRACE reveals developmental phenomena overlooked by traditional scalar metrics such as loss or accuracy. |
Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on robustness-like metamorphic relations, which limit the scope of linguistic properties they can test. |
| Approach: | They propose three new classes of metamorphic relations which address the properties of systematicity, compositionality and transitivity. |
| Outcome: | The proposed methods show that metamorphic models do not always behave according to expected linguistic properties. |
SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models are far from achieving the robustness and consistency required for safe biomedical NLI applications. |
| Approach: | They propose a framework that leverages external ontologies to instantiate diverse syllogistic arguments for biomedical NLI by identifying valid conclusions and extracting supporting evidence. |
| Outcome: | The proposed framework evaluates large language models on identifying valid conclusions and extracting supporting evidence across 28 syllogistic schemes instantiated with human genome pathways. |
Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on the application and evaluation of Large Language Models (LLMs) but LLMs are still prone to factual errors and inconsistencies in their explanations, offering limited control and interpretability for inference in complex domains. |
| Approach: | They propose an abductive-deductive framework that integrates Large Language Models with an external backward-chaining solver to refine step-wise natural language explanations. |
| Outcome: | The proposed framework improves explanations generated via in-context learning methods and Chain-of-Thought (CoT) on ethical NLI tasks while producing formal proofs describing and supporting models’ reasoning. |
Accelerating Antibiotic Discovery with Large Language Models and Knowledge Graphs (2025.acl-industry)
Copied to clipboard
| Challenge: | Antimicrobial resistance (AMR) is one of the top ten global public health threats . pharmaceutical industries face high costs (over $1 billion), long timelines, and a high failure rate . |
| Approach: | They propose an LLM-based pipeline that acts as an alert system, detecting prior evidence of antibiotic activity to prevent costly rediscoveries. |
| Outcome: | The proposed system integrates literature on organisms and chemicals into a Knowledge Graph (KG), ensuring taxonomic resolution, synonym handling, and multi-level evidence classification. |
FormalScience: Scalable Human-in-the-Loop Autoformalisation of Science with Agentic Code Generation in Lean (2026.acl-long)
Copied to clipboard
| Challenge: | Formalising informal mathematical reasoning into formally verifiable code is a significant challenge for large language models. |
| Approach: | They propose a domain-agnostic human-in-the-loop agentic pipeline to aid autoformalisation in scientific domains. |
| Outcome: | The proposed system produces syntactically correct and semantically aligned proofs for low cost. |
Consistent Autoformalization for Constructing Mathematical Libraries (2024.emnlp-main)
Copied to clipboard
| Challenge: | Autoformalization is the task of automatically translating mathematical content written in natural language to a formal language expression. |
| Approach: | They propose to use three mechanisms to improve autoformalization quality . they propose to combine most-similar retrieval augmented generation, denoising steps and auto-correction with syntax error feedback to improve syntactic, terminological and semantic control. |
| Outcome: | The proposed mechanisms can deliver syntactically, terminologically and semantically more consistent results across different models. |
Exploring the Limits of Fine-grained LLM-based Physics Inference via Premise Removal Interventions (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) perform complex mathematical reasoning in Physics where physical context requires that any symbolic manipulation satisfies complex semantics. |
| Approach: | They systematically remove crucial context from prompts to force instances where model inference may be algebraically coherent, yet unphysical. |
| Outcome: | The proposed models perform poorly in this domain, and their reasoning is not physics-informed. |
A Framework for Evaluation of Machine Reading Comprehension Gold Standards (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing literature on machine reading comprehension (MRC) data is limited on the data design of gold standards. |
| Approach: | They propose a framework to investigate linguistic features, lexical cues and ambiguity in MRC gold standards. |
| Outcome: | The proposed framework investigates the present linguistic features, required reasoning and background knowledge and factual correctness on the one hand, and the presence of lexical cues as a lower bound for the requirement of understanding on the other. |
LangVAE and LangSpace: Building and Probing for Language Model VAEs (2025.emnlp-demos)
Copied to clipboard
| Challenge: | a novel framework for modular construction of variational autoencoders (VAEs) on pre-trained large language models (LLMs) is presented. |
| Approach: | They propose a modular framework for variational autoencoders on top of pre-trained large language models. |
| Outcome: | The proposed framework can encode pre-trained language models into more compact and semantically disentangled representations. |
Empowering cross-lingual abilities of instruction-tuned large language models by translation-following demonstrations (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are often unbalanced towards English because of the imbalance in the distribution of pre-training data. |
| Approach: | They propose to empower Instruction-tuned LLMs (It-LLMs) in languages other than English by building semantic alignment between them. |
| Outcome: | The proposed model outperforms the existing model on multilingual question answering benchmarks and adapted versions of MMLU and BBH on six different languages. |
Inference to the Best Explanation in Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have found success in real-world applications, but their underlying explanatory process is still poorly understood. |
| Approach: | They propose to use a framework inspired by philosophical accounts on Inference to the Best Explanation (IBE) to advance the interpretation and evaluation of LLMs’ explanations. |
| Outcome: | The proposed framework can identify the best explanation with up to 77% accuracy (27% above random) while being intrinsically more efficient and interpretable. |
Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoders (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies on syntactic injection in Variational AutoEncoders (VAEs) are limited to LSTM-based VAEs. |
| Approach: | They propose to use latent space separation techniques to inject syntactic information into Variational AutoEncoders (VAEs) using graph-based models. |
| Outcome: | The proposed end-to-end VAE architecture can improve the organisation of the latent space, alleviating the information loss occurring in standard VAE setups, and resulting in enhanced performances on language modelling and downstream generation tasks. |
Eliciting Critical Reasoning in Retrieval-Augmented Generation via Contrastive Explanations (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies show that LLMs struggle to critically analyse RAG-based in-context information. |
| Approach: | They propose a framework that elicits critical arguments in RAG via contrastive explanations . they propose CRAG to retrieve relevant documents given a query and generate explanations that explicitly contrast relevance of passages to support the final answer. |
| Outcome: | The proposed framework improves state-of-the-art RAG models while requiring significantly fewer prompts and demonstrations and robust to perturbations in the retrieved documents. |
SemR-11: A Multi-Lingual Gold-Standard for Semantic Similarity and Relatedness for Eleven Languages (L18-1)
Copied to clipboard
| Challenge: | SemR-11 is a multi-lingual dataset for evaluating semantic similarity and relatedness for 11 languages. |
| Approach: | This paper describes a multi-lingual dataset for evaluating semantic similarity and relatedness for 11 languages. |
| Outcome: | The dataset is a multi-lingual dataset for evaluating semantic similarity and relatedness for 11 languages. |
Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networks (2024.acl-long)
Copied to clipboard
| Challenge: | Disentangled latent spaces usually have better semantic separability and geometrical properties, which leads to better interpretability and controllable data generation. |
| Approach: | They propose a flow-based invertible neural network mechanism integrated with a transformer-based language Autoencoder to deliver latent spaces with better semantic separability and geometrical properties. |
| Outcome: | The proposed model can deliver latent spaces with better separability properties compared to the current state-of-the-art models. |
Where Do LLMs Compose Meaning? A Layerwise Analysis of Compositional Robustness (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to linguistic compositional processing are limited by hallucinations, limited interpretability and a lack of semantic transparency. |
| Approach: | They propose a method that pooles token representations into linguistic constituents at various layers to improve model reliability and interpretability. |
| Outcome: | Experiments on inverse definition modelling, hypernym and synonym prediction show that semantic composition is not localised to specific layers but distributed across network depth. |
Aligning Large and Small Language Models via Chain-of-Thought Reasoning (2024.eacl-long)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) prompting empowers Large Language Models to solve complex reasoning tasks in a step-wise manner. |
| Approach: | They propose a method for aligning and transferring reasoning abilities between larger and smaller Language Models by using CoT-Demonstrations. |
| Outcome: | The proposed method outperforms baselines on question-answering and mathematical reasoning benchmarks. |
CARMA: Enhanced Compositionality in LLMs via Advanced Regularisation and Mutual Information Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) struggle with compositional generalisation, limiting their ability to systematically combine learned components to interpret novel inputs. |
| Approach: | They propose a method that enhances stability and robustness of compositional reasoning in large language models (LLMs) by regularising and preserving layer-wise stability constraints. |
| Outcome: | The proposed method reduces variability, stabilizes token representations, and improves compositional reasoning. |
Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for assessing the validity of explanations for NLI are time-consuming and prone to logical errors. |
| Approach: | They propose a framework that integrates Large Language Models and Theorem Provers to verify and refine natural language explanations through crowd-sourcing . they propose to use TPs to generate and formalise explanatory sentences and suggest potential inference strategies for NLI. |
| Outcome: | The proposed framework generates and formalises explanatory sentences and suggests potential inference strategies for NLI. |
Self-Refine Instruction-Tuning for Aligning Reasoning in Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to align reasoning abilities between Large Language Models and Smaller Language Model are supervised fine-tuning and preference optimization. |
| Approach: | They propose a method that elicits Smaller Language Models to self-improve their reasoning abilities via preference optimization. |
| Outcome: | The proposed method outperforms Instruction-tuning on commonsense and math reasoning tasks on common and math scenarios. |
Multi-Operational Mathematical Derivations in Latent Space (2024.naacl-long)
Copied to clipboard
| Challenge: | Using a symbolic engine, we investigate the possibility of approximating multiple mathematical operations in latent space for expression derivation. |
| Approach: | They propose to model mathematical operations as explicit geometric transformations by leveraging a symbolic engine and a large-scale dataset. |
| Outcome: | The proposed paradigms can be used to approximate multiple mathematical operations in latent space, while discriminating the conclusions for a single operation is achievable in the original expression encoder. |
A Symbolic Framework for Evaluating Mathematical Reasoning and Generalisation with Transformers (2024.naacl-long)
Copied to clipboard
| Challenge: | evaluating the generalisability of Transformers to out-of-distribution mathematical reasoning problems is a challenge for many open-source models. |
| Approach: | They propose a method for generating and perturbing detailed derivations of equations at scale, aided by a symbolic engine, and compare their results to sequence classification tasks. |
| Outcome: | The proposed framework outperforms GPT-4, GPT-3.5 and a canon of fine-tuned BERT models in classification tasks . perturbations to input reasoning can reduce their performance by up to 80 F1 points . |
Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies on reasoning in language models have sparked a debate on whether they can learn systematic inferential principles or merely exploit superficial patterns in the training data. |
| Approach: | They propose a method for circuit discovery aimed at interpreting syllogistic inference . they uncover a circuit involving middle-term suppression that elucidates how LMs transfer information to derive valid conclusions from premises. |
| Outcome: | The proposed method elucidates how LMs transfer information to derive valid conclusions from premises. |
Faithful and Robust LLM-Driven Theorem Proving for NLI Explanations (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that the interaction of large language models (LLMs) with theorem provers (TPs) can help verify and improve the validity of NLI explanations. |
| Approach: | They propose to use logical expressions to guide LLMs in generating structured proof sketches and to use them to improve their accuracy. |
| Outcome: | The proposed strategies improve autoformalisation, syntactic errors and explanation refinement over the state-of-the-art model. |
Dissecting Clinical Reasoning in Natural Language Inference for Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies on large language models (LLMs) have demonstrated the impact of prompting strategies and fine-tuning techniques on their reasoning capabilities. |
| Approach: | They examine four classes of prompting strategies to elicit reasoning in large language models . they then construct demonstrations using a frontier model to distil multi-step reasoning capabilities into smaller models based on Low-Rank Adaptation (LoRA). |
| Outcome: | The proposed model improves in 75% of the models on MedNLI and TREC Clinical Trials. |
To be or not to be an Integer? Encoding Variables for Mathematical Text (2022.findings-acl)
Copied to clipboard
| Challenge: | a number of natural language inference models are limited in interpreting mathematical knowledge written in Natural Language . a variable's meaning is determined exclusively by its defining type, i.e., its context . |
| Approach: | They propose a method that can create context-based representations for variables . they propose 'variable slot' approach which can be used to model variables based on their meaning . |
| Outcome: | The proposed model can be used to represent variables in natural language . it can be applied to a task of variable typing and create context-based representations for variables . |
Autoformalization in the Wild: Assessing LLMs on Real-World Mathematical Definitions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable potential in assisting with mathematical reasoning on different downstream tasks. |
| Approach: | They propose two new tools for autoformalizing real-world mathematical definitions from Wikipedia and arXiv papers. |
| Outcome: | The proposed methods improve definitions by up to 16% and undefined errors by 43%. |
MASA: LLM-Driven Multi-Agent Systems for Autoformalization (2025.emnlp-demos)
Copied to clipboard
| Challenge: | This paper presents a framework for building multi-agent systems for autoformalization driven by Large Language Models. |
| Approach: | They propose a framework for building multi-agent systems for autoformalization driven by Large Language Models. |
| Outcome: | The proposed framework leverages collaborative agents to convert natural language statements into formal representations. |
Adaptive LLM-Symbolic Reasoning via Dynamic Logical Solver Composition (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to NLP are static and require manual formalization. |
| Approach: | They propose an adaptive, multi-paradigm, neuro-symbolic inference framework that automatically identifies formal reasoning strategies from problems expressed in natural language and dynamically selects and applies specialized formal logical solvers. |
| Outcome: | The proposed framework outperforms baselines on individual and multi-paradigm reasoning tasks by 17% and 6%. |
Multi-Relational Hyperbolic Word Embeddings from Natural Language Definitions (2024.eacl-long)
Copied to clipboard
| Challenge: | a fundamental characteristic of natural language definitions is that they are widely abundant, pos-1. |
| Approach: | They propose a multi-relational model that explicitly leverages definitions' semantic structure to derive word embeddings. |
| Outcome: | The proposed model can preserve the semantic mapping required for interpretable traversal while imposing constraints on definitions while maintaining the recursive semantic structure. |
An LLM-based Knowledge Synthesis and Scientific Reasoning Framework for Biomedical Discovery (2024.acl-demos)
Copied to clipboard
Oskar Wysocki, Magdalena.wysocka@cruk.manchester.ac.uk Magdalena.wysocka@cruk.manchester.ac.uk, Danilo Carvalho, Alex Bogatu, Danilo.miranda@idiap.ch Danilo.miranda@idiap.ch, Maxime.delmas@idiap.ch Maxime.delmas@idiap.ch, Harriet.unsworth@cruk.manchester.ac.uk Harriet.unsworth@cruk.manchester.ac.uk, Andre Freitas
| Challenge: | BioLunar integrates Large Language Models (LLMs) to facilitate scientific reasoning across distributed evidence spaces. |
| Approach: | They present a tool for supporting biological analyses using Large Language Models (LLMs) it integrates large language models to facilitate scientific reasoning across distributed evidence spaces . |
| Outcome: | The proposed tool exemplifies the potential of the integration between LLMs, specialised databases and biomedical tools to support expert-level knowledge synthesis and discovery. |
Improving Chain-of-Thought Reasoning via Quasi-Symbolic Abstractions (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work suggests using logical formalisms coupled with external symbolic solvers to solve complex tasks. |
| Approach: | They propose a framework to disentangle content from logical reasoning without a complete formalisation. |
| Outcome: | The proposed methods improve CoT-based methods by up to 8% accuracy on challenging adversarial variations on both natural language and symbolic reasoning tasks. |
Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoders (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent work has struggled to achieve consistent results due to the inevitable loss of semantic information in the variational bottleneck and limited control over the decoding mechanism. |
| Approach: | They propose a model that leverages the controllability of VQVAE to guide the self-attention mechanism in Transformer-based VAEs to improve semantic control and generation. |
| Outcome: | The proposed model outperforms existing state-of-the-art VAE models in terms of control and preservation of semantic information across different tasks. |
Emergence and Localisation of Semantic Role Circuits in LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Despite displaying semantic competence, large language models’ internal mechanisms that ground abstract semantic structure remain insufficiently characterised. |
| Approach: | They propose a causal-temporal methodology that uses contrastive minimal pairs, edge-attribution circuit discovery, and training-time tracking to characterise semantic-role circuits along three dimensions. |
| Outcome: | The proposed method is applicable to any behaviour isolatable through contrastive minimal pairs, including factual recall, syntactic agreement, and logical reasoning. |
NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports (2023.emnlp-main)
Copied to clipboard
| Challenge: | Clinical trial reports (CTRs) are indispensable for the development of personalized medicine. |
| Approach: | They propose a resource to help researchers interpret clinical trial reports . they use natural language inference to compute textual entailment . |
| Outcome: | The proposed resource is the first to cover interpretation of full clinical trial reports . it includes tasks to determine inference relation between natural language statements and CTRs . |
Compartmentalised Agentic Reasoning for Clinical NLI (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models produce fluent judgments for clinical natural language inference, yet fail when the decision requires the correct inferential schema rather than surface matching. |
| Approach: | They propose a compartmentalised agentic framework that routes each premise–statement pair to a reasoning family and applies a specialised solver with explicit verification and targeted refinement. |
| Outcome: | The proposed framework improves mean accuracy from 23% with direct prompting to 57%, with the largest gains on structurally demanding reasoning types. |
PEIRCE: Unifying Material and Formal Reasoning via LLM-Driven Neuro-Symbolic Refinement (2025.acl-demo)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are capable of material inference but lack formal rigour and verifiability. |
| Approach: | They propose a framework to unify material and formal inference through an iterative conjecture–criticism process. |
| Outcome: | The proposed framework unifies material and formal inference through an iterative conjecture–criticism process. |