Papers by Johan Bos
KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing LLMs lack systematic coverage of a bounded knowledge universe and compositional set-based reasoning over that universe. |
| Approach: | They propose a benchmark for multiple-choice questions based on 1,183 enumeration seeds . they use knowledge width, cardinality of required universe, reasoning depth to formalize the challenge . |
| Outcome: | The proposed benchmarks achieve only 5.26–36.88 F1 on universe enumeration and 16.00–44.19 accuracy on knowledge-grounded reasoning. |
Quantification Annotation in ISO 24617-12, Second Draft (2022.lrec-1)
Copied to clipboard
Harry Bunt, Maxime Amblard, Johan Bos, Karën Fort, Bruno Guillaume, Philippe de Groote, Chuyuan Li, Pierre Ludmann, Michel Musiol, Siyana Pavlova, Guy Perrier, Sylvain Pogodalla
| Challenge: | a project aimed at establishing an interoperable annotation schema for quantification phenomena was relaunched in early 2022 due to the Covid-19 pandemic . |
| Approach: | This paper describes the continuation of a project that aims at establishing an interoperable annotation schema for quantification phenomena as part of the ISO suite of semantic annotation standards. |
| Outcome: | The proposed schema is part of the ISO suite of semantic annotation standards known as the Semantic Annotation Framework (SemAF). |
What can we learn from Semantic Tagging? (D18-1)
Copied to clipboard
| Challenge: | a recent study shows that multi-task learning improves performance of NLP tasks by exploiting similarities between tasks. |
| Approach: | They employ semantic tagging as an auxiliary task for three NLP tasks . they compare full neural network sharing, partial neural network shared and learning what to share . |
| Outcome: | The proposed model improves for part-of-speech tagging, universal dependency parsing and natural language inference. |
Character-level Representations Improve DRS-based Semantic Parsing Even in the Age of BERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | a new method of analysis based on semantic tags demonstrates that character-level representations improve performance across a subset of selected semantic phenomena. |
| Approach: | They combine character-level and contextual language model representations to improve performance on Discourse Representation Structure parsing. |
| Outcome: | The proposed model improves performance on a subset of selected semantic phenomena. |
Scope-enhanced Compositional Semantic Parsing for DRT (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing compositional semantic parsers for DRT struggle to produce well-formed representations due to the complexity of the sentence. |
| Approach: | They propose a compositional, neurosymbolic semantic parser for DRT that uses a novel mechanism for predicting quantifier scope. |
| Outcome: | The proposed model produces well-formed outputs and performs well on complex sentences. |
What’s the Meaning of Superhuman Performance in Today’s NLU? (2023.acl-long)
Copied to clipboard
Simone Tedeschi, Johan Bos, Thierry Declerck, Jan Hajič, Daniel Hershcovich, Eduard Hovy, Alexander Koller, Simon Krek, Steven Schockaert, Rico Sennrich, Ekaterina Shutova, Roberto Navigli
| Challenge: | Recent research has focused on developing larger pretrained language models and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities. |
| Approach: | They propose to use benchmarks such as SuperGLUE and SQUAD to evaluate PLMs' abilities in language understanding, reasoning, and reading comprehension to assess their performance. |
| Outcome: | The proposed benchmarks have serious limitations affecting comparison between humans and PLMs and provide recommendations for fairer and more transparent benchmarks. |
Transparent Semantic Parsing with Universal Dependencies Using Graph Transformations (2022.coling-1)
Copied to clipboard
| Challenge: | Existing semantic parsers are based on deep learning, but rule-based approaches offer advantages . a drawback of neural semantic parses is that their output lacks explainability . |
| Approach: | They propose a method that maps a syntactic dependency tree to a formal meaning representation using a series of graph transformations. |
| Outcome: | The proposed method outperforms neural parsers in English, German, Italian and Dutch. |
MAGPIE: A Large Corpus of Potentially Idiomatic Expressions (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing corpora cover less than 5,000 instances of less than 100 different idiom types . large corpus allows for better evaluation of assumptions about idiomatic expressions . |
| Approach: | They propose to build the largest-to-date corpus of idioms for English using crowdsourcing methods. |
| Outcome: | The proposed corpus is larger than existing resources and contains rich metadata and is made publicly available. |
Pre-Trained Language-Meaning Models for Multilingual Parsing and Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have been used for tasks in computational semantics but meaning representations are not included in PLMs. |
| Approach: | They propose to include meaning representations besides natural language texts in the same model . they propose to use DRSs to improve performance of non-English tasks . |
| Outcome: | The proposed approach achieves the best performance on multilingual parsing and DRS-to-text generation tasks. |
Evaluating Scoped Meaning Representations (L18-1)
Copied to clipboard
| Challenge: | Semantic parsing offers many opportunities to improve natural language understanding . current research on open-domain semantic parsers focuses on supervised learning methods . |
| Approach: | They propose a semantically annotated parallel corpus for English, German, Italian, and Dutch . they use a matching tool to evaluate scoped meaning representations to match clauses . |
| Outcome: | The proposed method captures the semantics of negation, modals, quantification, and presupposition triggers . it compares scoped meaning representations to gold standard parsers and finds improvements . |
Input Representations for Parsing Discourse Representation Structures: Comparing English with Chinese (2021.acl-short)
Copied to clipboard
| Challenge: | Neural semantic parsers have obtained acceptable results in parsing DRSs . previous studies have focused on parse of DRS in English, but have focused only on a few languages . |
| Approach: | They propose to use character sequences as input to map meaning representations to string format. |
| Outcome: | The proposed models learn the meaning of a series of semantic phenomena by taking sentences as input and outputting the corresponding DRSs, without the aid of any extra linguistic information. |