Papers by Mark Steedman
Open-Domain Contextual Link Prediction and its Complementarity with Entailment Graphs (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for linking knowledge graphs only use textual contexts . contextual link prediction is useful for finding context-dependent entailments . |
| Approach: | They propose a task of open-domain contextual link prediction which uses textual context and KG structure to perform link prediction. |
| Outcome: | The proposed model can ground the triples in the context of the original dataset and infer missing relations in context. |
Neutralizing Bias in LLM Reasoning using Entailment Graphs (2025.findings-acl)
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) is a foundational understanding task in language understanding. |
| Approach: | They propose a framework to construct counterfactual reasoning data and fine-tune LLMs to reduce attestation bias. |
| Outcome: | The proposed framework reduces hallucinations from attestation bias on original and bias-neutralized datasets while keeping hypotheses unchanged. |
Sources of Hallucination by Large Language Models on Inference Tasks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI) |
| Approach: | They propose to use LLMs to probe their behavior using controlled experiments. |
| Outcome: | The proposed models perform significantly worse on NLI test samples which do not conform to these biases than those which do. |
Duality of Link Prediction and Entailment Graph Induction (P19-1)
Copied to clipboard
| Challenge: | In this paper, we show that link prediction and entailment graph induction are complementary. |
| Approach: | They propose an entailment score that exploits the new facts discovered by the link prediction model and then form engorgement graphs between relations. |
| Outcome: | The proposed entailment score outperforms prior state-of-the-art results on a standard entialment dataset and the new link prediction scores show improvements over the raw link prediction score. |
Multivalent Entailment Graphs for Question Answering (2021.emnlp-main)
Copied to clipboard
Nick McKenna, Liane Guillou, Mohammad Javad Hosseini, Sander Bijl de Vroe, Mark Johnson, Mark Steedman
| Challenge: | a recent study shows that drawing inferences between open domain predicates is a necessity for true language understanding. |
| Approach: | They propose to reinterpret the Distributional Inclusion Hypothesis to model entailment between predicates of different valencies. |
| Outcome: | The proposed graphs are more useful than using the same valency evidence, the authors show . they show that drawing on evidence across valencies answers more questions than using only the same evidence. |
Align-then-Enhance: Multilingual Entailment Graph Enhancement with Soft Predicate Alignment (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to learn typed entailment graphs with predicates as nodes and enttailment relations as edges are incomplete. |
| Approach: | They propose a task to utilize entailment information from one EG to enhance another in a different language. |
| Outcome: | The proposed framework outperforms existing graphs in multilingual entailment graph enhancement tasks. |
Multi-Document Summarization with Centroid-Based Pretraining (2023.acl-short)
Copied to clipboard
| Challenge: | In Multi-Document Summarization, the input is a set of documents, and the output is its summary. |
| Approach: | They propose a novel pretraining objective that uses the ROUGE-based centroid of each document cluster as a proxy for its summary. |
| Outcome: | The proposed model is better or comparable to state-of-the-art models. |
Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to make multilingual systems expensive and tedious introduce pipeline of errors. |
| Approach: | They propose to use pre-trained multilingual models to enhance the transfer learning process by intermediate fine-tuning of pretrained multi-lingual models. |
| Outcome: | The proposed approach improves on the cross-lingual dialogue state tracking task with only 10% of the target language task data and zero-shot setup respectively. |
Data Augmentation via Dependency Tree Morphing for Low-Resource Languages (D18-1)
Copied to clipboard
| Challenge: | Lack of sizable training datasets leads to poor performance in low-resource languages. |
| Approach: | They propose two techniques to augment training sets of low-resource languages using dependency trees. |
| Outcome: | The proposed methods improve on the training datasets for low-resource languages. |
Empirical Study on Data Attributes Insufficiency of Evaluation Benchmarks for LLMs (2025.coling-main)
Copied to clipboard
| Challenge: | Existing benchmarks for evaluating large language models neglect key qualitative data attributes that can significantly impact the final rankings of LLMs. |
| Approach: | They propose a framework with three modules designed to assess diversity, redundancy, and difficulty. |
| Outcome: | The proposed framework systematically incorporates diversity, redundancy, and difficulty attributes and shows that they influence the ranking of LLMs. |
Sentence-Incremental Neural Coreference Resolution (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems for identifying spans as mentions are based on mention-pair formulations, but they are not generalized beyond pronouns. |
| Approach: | They propose a sentence-incremental neural coreference resolution system which incrementally builds clusters after marking mention boundaries in a shift-reduce method. |
| Outcome: | The proposed system outperforms state-of-the-art methods on OntoNotes and CODI-CRAC 2021 datasets and is comparable to state- of-the art methods. |
Learning Negation Scope from Syntactic Structure (2020.starsem-1)
Copied to clipboard
| Challenge: | a semi-supervised model learns the semantics of negation purely through syntactic analysis . Negation is a semantic phenomenon in natural language which varies significantly . |
| Approach: | They propose a semi-supervised model which learns negation semantics purely through syntactic analysis. |
| Outcome: | The proposed model achieves state-of-the-art on a Negation Scope Detection task without identifying individual words or extracting features beyond syntax. |
Language Models Are Poor Learners of Directional Inference (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets fail to test directionality and are infested by artefacts that can be learnt as proxy for entailments, yielding over-optimistic results. |
| Approach: | They propose a benchmark for directional predicate entailments that is extrinsic to existing training sets. |
| Outcome: | The proposed model is incompetent on directional predicate entailments, compared to engorgement graphs, but limited by sparsity. |
Extrinsic Evaluation of Machine Translation Metrics (2023.acl-long)
Copied to clipboard
| Challenge: | MT metrics are widely used to distinguish the quality of machine translation systems across relatively large test sets. |
| Approach: | They evaluate the segment-level performance of the most widely used MT metrics by correlating them with how useful they are for downstream tasks. |
| Outcome: | The MT metrics are widely used to distinguish the quality of machine translation systems across relatively large test sets. |
S2LPP: Small-to-Large Prompt Prediction across LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a small model can be used to select effective prompt templates for a larger model. |
| Approach: | They propose a method to use a smaller model to select effective prompt templates for a larger model. |
| Outcome: | The proposed method significantly reduces the cost of prompt engineering while matching performance with optimal prompts among candidates. |
Prosodic segmentation for parsing spoken dialogue (2021.acl-long)
Copied to clipboard
| Challenge: | Existing parsers struggle to parse spoken dialogue because of disfluencies and unmarked boundaries between sentence-like units (SUs). |
| Approach: | They hypothesize that prosody affects a parser that receives an entire dialogue turn as input, instead of gold standard pre-segmented SUs. |
| Outcome: | The proposed model performs better than the SU-based model on the English Switchboard corpus despite performing two tasks rather than one, and pitch and intensity features are the most important for this corpus. |
Explicit Inductive Inference using Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) suffer a signifi- cant performance drop when entailment labels disagree with the attestation label of hypothesis H. |
| Approach: | They propose a pipeline that exploits an LLM's attestation bias to do explicit inductive inference . they transform a premise into attested alternatives and aggregate the results . |
| Outcome: | The proposed pipeline improves the performance of large language models on inference tasks and alleviates the attestation bias. |
The Role of Reentrancies in Abstract Meaning Representation Parsing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) parsers make errors with respect to reentrancies, which complicates AMR parsing and requires specific transitions. |
| Approach: | They propose to categorize the types of errors AMR parsers make with respect to reentrancies and find that correcting these errors provides an in-crease of up to 5% Smatch in parsing perfor- mance and 20% in reen- trancy prediction. |
| Outcome: | The proposed formalism can predict reentrancies with 5% accuracy and 20% accuracy. |
The role of context in neural pitch accent detection in English (2020.emnlp-main)
Copied to clipboard
| Challenge: | Prosody is a rich information source in natural language, serving as a marker for phenomena such as contrast. |
| Approach: | They propose a model that uses full utterances as input and adds an LSTM layer to detect prosodic events in speech. |
| Outcome: | The proposed model improves on the American English speech in the Boston University Radio News Corpus. |
CCG Parsing Algorithm with Incremental Tree Rotation (N19-1)
Copied to clipboard
| Challenge: | Combinatory Categorial Grammar (CCG) is a mildly context sensitive grammar formalism that excels in incremental sentence processing. |
| Approach: | They propose a new incremental parsing algorithm that uses a syntactic approach . it uses right-branching constituent structures and optional constituents that adjoin on the right . |
| Outcome: | The proposed algorithm can cover the whole CCGbank with greater incrementality and accuracy than previous proposals. |
Character-Level Models versus Morphology in Semantic Role Labeling (P18-1)
Copied to clipboard
| Challenge: | Character-level models are used for high-level semantic analysis tasks such as semantic role labeling. |
| Approach: | They train character-level models that use word, character and morphology level information . they analyze how performance of characters compare to words and a variety of morphological typologies . |
| Outcome: | The results shed light on important characteristics of character-level models and their semantic capability. |
Aspectuality Across Genre: A Distributional Semantics Approach (2020.coling-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the aspectual class of verbs in English for predicting coherence relations in text and imagery, predicting links in entailment graphs and interpreting sign languages. |
| Approach: | They propose to model two elementary aspects of aspectual class, states vs. events, and telic v. atelic events, with distributional semantics. |
| Outcome: | The proposed model outperforms previous work on three datasets and shows that telicity correlates with genre and discourse goals. |
Max-Margin Incremental CCG Parsing (2020.acl-main)
Copied to clipboard
| Challenge: | a new incremental parser reduces the number of beam search violations and minimises the biggest violation. |
| Approach: | They propose to use beam search optimisation to minimise all beam search violations instead of minimising only the biggest violation. |
| Outcome: | The proposed parser outperforms existing non-incremental parsers and minimises all beam search violations instead of minimising the biggest violation. |
Node Embeddings for Graph Merging: Case of Knowledge Graph Construction (D19-53)
Copied to clipboard
| Challenge: | Combining two graphs requires merging the nodes which are counterparts of each other. In this process errors occur, resulting in incorrect merging or failure to merge. |
| Approach: | They propose to replace string similarity with vector embedding similarity to reduce errors when merging two graphs . they propose to use graph-based and word-based embeddable graph embeddances to obtain graph node embeddations. |
| Outcome: | The proposed algorithm reduces errors in merging two graphs with a corpus level one graph using string similarity and vector embedding similarity. |
Predicting accuracy on large datasets from smaller pilot data (P18-2)
Copied to clipboard
| Challenge: | obtaining training data is often the most difficult part of an NLP or ML project . obtaining data is important to estimate how much training data a system will require to achieve a target accuracy. |
| Approach: | They propose a performance extrapolation task to evaluate extrapolations on larger training sets. |
| Outcome: | The proposed method can predict accuracy on larger training datasets. |
Cross-lingual Inference with A Chinese Entailment Graph (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing work on predicate entailment detection from typed open relation triples has not been able to detect predicates. |
| Approach: | They propose a pipeline for building Chinese entailment graphs using an open relation extraction method. |
| Outcome: | The proposed pipeline outperforms monolingual and Chinese entailment graphs on a parallel dataset. |
Human Temporal Inferences Go Beyond Aspectual Class (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing work on aspectual classification in English has been motivated as a pre-requisite for Natural Language Understanding (NLU) in cases where temporal reasoning is required. |
| Approach: | They propose to classify English verb phrases into situation aspect categories by gathering crowd-sourced judgements from non-expert, native English participants. |
| Outcome: | The proposed approach uses a crowd-sourced dataset from non-expert, native English participants to examine aspectual entailments in English. |
A Usage-centric Take on Intent Understanding in E-Commerce (2024.emnlp-main)
Copied to clipboard
| Challenge: | Identifying and understanding user intents is a crucial task for E-Commerce. |
| Approach: | They propose to use intent understanding as a natural language reasoning task independent of product ontologies to identify and understand user intents. |
| Outcome: | The proposed framework can't be used to strongly align user intents with products with desirable properties and recommend useful products across diverse categories. |
Wide-Coverage Neural A* Parsing for Minimalist Grammars (P19-1)
Copied to clipboard
| Challenge: | a new parser for wide-coverage parsing uses a linguistically expressive yet highly constrained grammar . the expected time complexity of the parsers is cubic in the length of the sentence . |
| Approach: | They propose to use a linguistically expressive yet highly constrained grammar to parse a wide-coverage sentence using a bi-LSTM neural network supertagger. |
| Outcome: | The proposed algorithm recovers unbounded long distance dependencies and can recover unbundled long distance dependents. |
World Knowledge Resolves Some Aspectual Ambiguity (2025.findings-acl)
Copied to clipboard
| Challenge: | Annotating event descriptions with aspectual features is often seen as a pre-requisite to temporal reasoning, however, a recent study has shown that non-experts’ annotations of the aspectual class of English verb phrases can disagree with both expert linguistic annotations and each other. |
| Approach: | They hypothesized that people use their world knowledge to tacitly conjure their own contexts, leading to disagreement between them. |
| Outcome: | The results show that the hypothesis explains some of the disagreement, but outputs from GPT-4 are not an accurate predictor of human answers. |
Concept-Reversed Winograd Schema Challenge: Evaluating and Improving Robust Reasoning in Large Language Models via Abstraction (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent research has revealed that Large Language Models (LLMs) often experience issues with hallucinations and unreliable reasoning due to semantic associations and superficial logical chains. |
| Approach: | They propose a concept-reversed Winograd Schema Challenge dataset to evaluate the robustness of Large Language Models (LLMs) they propose Abstraction-of-Thought (AoT) method for recovering adversarial cases to normal cases using conceptual abstraction to improve LLMs’ robustness and consistency in reasoning. |
| Outcome: | The proposed method improves LLMs’ robustness and consistency in reasoning under adversarial and long-tail scenarios. |