Papers by Bernd Bohnet
On Faithfulness and Factuality in Abstractive Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing conditional text generation models produce unfaithful and unfaithed summaries . current models accomplish a high level of fluency and coherence . |
| Approach: | They propose to use pretrained models for document summarization to better understand hallucinations . they find that textual entailment measures better correlate with faithfulness . |
| Outcome: | The proposed models generate faithful and factual summaries as evaluated by humans. |
Coreference Resolution through a seq2seq Transition-Based System (2023.tacl-1)
Copied to clipboard
| Challenge: | Recent coreference resolution systems use search algorithms to identify mentions and resolve coreference. |
| Approach: | They propose a text-to-text coreference resolution system that uses a semantic paradigm to predict mentions and links jointly. |
| Outcome: | The proposed system achieves state-of-the-art accuracy on CoNLL-2012 datasets with 83.3 F1-score for English, 68.5 F1 score for Arabic, and 74.3 F1 scores for Chinese. |
The Second Multilingual Surface Realisation Shared Task (SR’19): Overview and Evaluation Results (D19-63)
Copied to clipboard
| Challenge: | EMNLP’19 Workshop on Multilingual Surface Realisation aims to stimulate the exploration of advanced neural networks for multilingual sentence generation from Universal Dependency (UD) structures. |
| Approach: | They present results from the SR'19 Shared Task, a multilingual surface realisation task organised as part of the EMNLP'19 Workshop on Multilingual Surface Realisation. |
| Outcome: | The SR'19 shared task was organised as part of the EMNLP'19 Workshop on Multilingual Surface Realisation . it consisted of two tracks with different levels of complexity . the shallow track was offered in eleven, and the deep track in three languages . |
A Comprehensive Evaluation of Tool-Assisted Generation Strategies (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Various few-shot tool-usage strategies have been proposed to overcome LMs' shortcomings. |
| Approach: | They propose to augment language models with tools to overcome their shortcomings . they find strong no-tool baselines are competitive to tool-assisted strategies . |
| Outcome: | The proposed strategies outperform those that refine incorrect outputs with tools in knowledge-retrieval tasks, the study finds . the findings suggest few-shot tool integration is still an open challenge . |
A Gold Standard Dependency Treebank for Turkish (2020.lrec-1)
Copied to clipboard
| Challenge: | Currently, Turkish treebanks are limited due to the limited number of annotated sentences in the domains of Wikipedia and ITU Web Treebanks. |
| Approach: | They propose to annotate Turkish web and Wikipedia sentences for segmentation, morphology, part-of-speech and dependency relations using tagsets and a Wikipedia section. |
| Outcome: | The proposed treebank is the largest publicly available morpho-syntactic treebank in terms of word count and has a dedicated Wikipedia section. |
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022.emnlp-demos)
Copied to clipboard
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo F . R. Ribeiro, Lewis Tunstall, Li Zhang, Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou
| Challenge: | Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. |
| Approach: | They propose to use the Generation, Evaluation, and Metrics Benchmark to integrate new evaluation methods into existing evaluations. |
| Outcome: | The proposed evaluation infrastructure bridges the gap between the advantages of leaderboards and in-depth and evolving evaluations by allowing model developers to benefit from each other's work. |
Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings (P18-1)
Copied to clipboard
| Challenge: | recurrent neural networks have produced significant advances in part-of-speech tagging accuracy . a common feature of these models is the presence of rich initial word encodings . however, word or sub-word information interacts only through subsequent recursive layers . |
| Approach: | They propose to use recurrent neural networks with sentence-level context for initial character and word-based representations. |
| Outcome: | The proposed model has the highest accuracy of all participating systems in the CoNLL 2017 task. |
A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains (2024.acl-long)
Copied to clipboard
Alon Jacovi, Yonatan Bitton, Bernd Bohnet, Jonathan Herzig, Or Honovich, Michael Tseng, Michael Collins, Roee Aharoni, Mor Geva
| Challenge: | Recent literature discusses automatic methods to evaluate reasoning to improve their correctness, but no fine-grained step-level datasets are available to enable thorough evaluation of such verification methods. |
| Approach: | They propose to benchmark automatic verifiers of complex Chain-of-Thought reasoning in open-domain question-answering settings using a dataset that includes comprehensive labels for relevance, attribution to evidence passages, and logical correctness of each reasoning step. |
| Outcome: | The proposed dataset shows that verifiers struggle at verifying reasoning chains, particularly verifying logical correctness and detecting contradictions. |
Neural Mention Detection (2020.lrec-1)
Copied to clipboard
| Challenge: | Mention detection is an important preprocessing step for downstream applications such as NER and coreference resolution. |
| Approach: | They propose and compare three approaches to mention detection using ELMO embeddings and a biaffine classifier. |
| Outcome: | The proposed model outperforms state-of-the-art models on the GENIA corpora and improves on mention recall. |
Decoding Part-of-Speech from Human EEG Signals (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that EEG signal magnitude and topography depend on word length, frequency and open vs. closed class. |
| Approach: | They propose to use EEG to predict Part-of-Speech (PoS) tags from neural signals measured at millisecond resolution during text reading. |
| Outcome: | The proposed techniques outperform linear-SVMs on PoS tagging of unigram and bigram data. |
Named Entity Recognition as Dependency Parsing (2020.acl-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a fundamental task in Natural Language Processing, concerned with identifying spans of text expressing references to entities. |
| Approach: | They propose a method to handle both types of NEs in one system by using a biaffine dependency parsing model which scores pairs of start and end tokens in a sentence. |
| Outcome: | The proposed model performs well on 8 corpora and achieves accuracy gains of up to 2.2 percentage points. |