Papers by Ben Bogin
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code (2025.coling-industry)
Copied to clipboard
Taishi Nakamura, Mayank Mishra, Simone Tedeschi, Yekun Chai, Jason T. Stillerman, Felix Friedrich, Prateek Yadav, Tanmay Laud, Vu Minh Chien, Terry Yue Zhuo, Diganta Misra, Ben Bogin, Xuan-Son Vu, Marzena Karpinska, Arnav Varma Dantuluri, Wojciech Kusa, Tommaso Furlanello, Rio Yokota, Niklas Muennighoff, Suhas Pai, Tosin Adewumi, Veronika Laippala, Xiaozhe Yao, Adalberto Barbosa Junior, Aleksandr Drozd, Jordan Clive, Kshitij Gupta, Liangyu Chen, Qi Sun, Ken Tsui, Nour Moustafa-Fahmy, Nicolo Monti, Tai Dang, Ziyang Luo, Tien-Tung Bui, Roberto Navigli, Virendra Mehta, Matthew Blumberg, Victor May, Hiep Nguyen, Sampo Pyysalo
| Challenge: | Pretrained language models are integral part of AI applications, but their high computational cost limits accessibility. |
| Approach: | They evaluate Aurora-M, a 15B parameter multilingual open-source model trained on English, Finnish, Hindi, Japanese, Vietnamese, and code. |
| Outcome: | The proposed model outperforms existing models on English, Finnish, Hindi, Japanese, Vietnamese, and code. |
Answering Questions by Meta-Reasoning over Multiple Chains of Thought (2023.emnlp-main)
Copied to clipboard
| Challenge: | Modern systems for multi-hop question answering (QA) break questions into a sequence of reasoning steps, termed chain-of-thought (CoT) Often, multiple chains are sampled and aggregated, but the intermediate steps themselves are discarded. |
| Approach: | They propose a method which prompts large language models to meta-reason over multiple chains of thought rather than aggregate their answers. |
| Outcome: | The proposed approach outperforms baselines on 7 multi-hop QA datasets. |
MedICaT: A Dataset of Medical Images, Captions, and Textual References (2020.findings-emnlp)
Copied to clipboard
Sanjay Subramanian, Lucy Lu Wang, Ben Bogin, Sachin Mehta, Madeleine van Zuylen, Sravanthi Parasa, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi
| Challenge: | Existing largescale datasets explicitly exclude compound figures . existing systems lack this ability to identify relevant subfigures . |
| Approach: | They propose a dataset of medical images in context that allows figure-to-text alignment . they use captions, inline references and manually annotated subfigures for compound figures . |
| Outcome: | The proposed dataset demonstrates the utility of inline references in image-text matching. |
Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing (P19-1)
Copied to clipboard
| Challenge: | Semantic parsing to SQL has largely ignored the structure of the database schema . a recent study used a simple DB that was observed at both training and test time. |
| Approach: | They propose a semantic parser where the schema structure is encoded with a graph neural network and used at both encoding and decoding time. |
| Outcome: | The proposed parser improves from 33.8% to 39.4%, dramatically above the current state of the art, which is at 19.7%. |
Leveraging Code to Improve In-Context Learning for Semantic Parsing (2024.naacl-long)
Copied to clipboard
| Challenge: | In-context learning is an attractive approach for semantic parsing, but learning to parse to rare domain-specific languages from a few demonstrations is challenging. |
| Approach: | They propose to use Python instead of DSLs to augment prompts with a structured domain description. |
| Outcome: | The proposed approach improves accuracy and generalization across three datasets. |
AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current language models and retrieval-augmented LMs are limited in their ability to perform tasks on the web. |
| Approach: | They propose a benchmark to evaluate language agents built on top of language models . they propose 'AssistantBench' which includes 214 tasks that can be automatically evaluated . |
| Outcome: | The proposed agent outperforms existing agents in a new benchmark for language agents on the web. |
Unobserved Local Structures Make Compositional Generalization Hard (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show sequence-to-sequence models struggle to generalize to new compositions . little is known on what makes generalization hard on a particular test instance . |
| Approach: | They propose a criterion for the difficulty of an example that is hard if it contains a local structure that was not observed at training time. |
| Outcome: | The proposed rule predicts instance-level generalization well across 5 different datasets. |
Towards an argumentative content search engine using weak supervision (C18-1)
Copied to clipboard
| Challenge: | Existing work focused on detecting claims within a small set of documents . however, pinpointing relevant claims within massive unstructured corpora, received little attention. |
| Approach: | They propose to use a weak signal to develop a query for claim–sentence detection using a large text corpus. |
| Outcome: | The proposed system outperforms previous results in terms of precision and coverage. |
SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories (2024.emnlp-main)
Copied to clipboard
Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, Tushar Khot
| Challenge: | Large Language Models (LLMs) have made significant progress in writing code, but can they be used to reproduce results from research repositories? |
| Approach: | They propose a benchmark to evaluate the capability of Large Language Models to reproduce results from research repositories. |
| Outcome: | The benchmark aims to capture the realistic challenges faced by researchers working with machine learning and natural language processing repositories. |
Global Reasoning over Database Structures for Text-to-SQL Parsing (D19-1)
Copied to clipboard
| Challenge: | Existing semantic parsers only select a set of database constants at training time . current models only consider local information, not global ones . |
| Approach: | They propose a semantic parser that globally reasons about the structure of the query to make a more contextually-informed selection of database constants. |
| Outcome: | The proposed model increases accuracy from 39.4% to 47.4% on a zero-shot semantic parsing dataset with complex databases. |
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020.findings-emnlp)
Copied to clipboard
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou
| Challenge: | Standard test sets for supervised learning evaluate in-distribution generalization but are misleading when a dataset has systematic gaps. |
| Approach: | They propose a more rigorous annotation paradigm for NLP that helps to close systematic gaps in the test data. |
| Outcome: | The proposed model performs significantly lower on contrast sets than on the original test sets—up to 25% in some cases. |
COVR: A Test-Bed for Visually Grounded Compositional Generalization with Real Images (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for visual-grounded models have focused on synthetic images . et al., 2018: compositional generalization is crucial for building models that generalize to new settings. |
| Approach: | They propose a test-bed for visually-grounded compositional generalization with real images. |
| Outcome: | The proposed test-bed enables compositional splits where models need to generalize to new concepts and compositions in a zero- or few-shot setting. |
Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering (2021.tacl-1)
Copied to clipboard
| Challenge: | Neural networks fail to generalize to out-of-distribution examples that contain new compositions. |
| Approach: | They propose a model that computes a representation and denotation for all question spans in a bottom-up, compositional manner using a CKY-style parser. |
| Outcome: | The proposed model achieves 96.1% accuracy on a challenging dataset compared to baseline models on . previous models failed to generalize to out-of-distribution examples . |
Obtaining Faithful Interpretations from Compositional Neural Networks (2020.acl-main)
Copied to clipboard
Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, Matt Gardner
| Challenge: | Neural module networks (NMNs) are a popular approach for modeling compositionality but prior work implicitly assumed that the structure of the network modules provides a faithful explanation of the model’s reasoning. |
| Approach: | They propose to use auxiliary supervision to train a model with a structured model that can understand the reasoning process and make better choices for module architecture. |
| Outcome: | The proposed models on two datasets show that the proposed models do not provide a faithful explanation of model behaviour. |
Diverse Demonstrations Improve In-context Compositional Generalization (2023.acl-long)
Copied to clipboard
| Challenge: | In-context learning has shown great success in i.i.d semantic parsing splits . however, in compositional generalization, selecting similar demonstrations is insufficient . |
| Approach: | They propose a method to select diverse demonstrations that collectively cover all the structures required in the output program and encourage the model to generalize to new structures from these demonstrations. |
| Outcome: | The proposed method improves performance across three compositional generalization datasets and finetuning. |