Papers by Ralph Weischedel
Perhaps PTLMs Should Go to School – A Task to Assess Open Book and Closed Book QA (2021.emnlp-main)
Copied to clipboard
| Challenge: | Taking the exam closed book, but having read the textbook, yields at best minor improvement (56%), suggesting that the PTLM may not have “understood” the textbook (or perhaps misundersttoo the questions). |
| Approach: | They propose to use pre-trained language models to answer questions from introductory college textbooks and hundreds of true/false statements based on review questions written by the authors. |
| Outcome: | The proposed task includes two college-level introductory texts in the social sciences (American Government 2e) and humanities (U.S. History). |
Machine-Assisted Script Curation (2021.naacl-demos)
Copied to clipboard
Manuel Ciosici, Joseph Cummings, Mitchell DeHaven, Alex Hedges, Yash Kankanampati, Dong-Ho Lee, Ralph Weischedel, Marjorie Freedman
| Challenge: | Scripts have been of interest for encoding procedural knowledge and understanding stories for over 40 years . narrative descriptions often omit common knowledge . |
| Approach: | They propose a machine-aided script creator that automates script creation with suggestions for event types, links to Wikidata, and sub-events that may have been forgotten. |
| Outcome: | The proposed system automates portions of the script creation process with suggestions for event types, links to Wikidata, and sub-events that may have been forgotten. |
ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos (2023.emnlp-main)
Copied to clipboard
Te-Lin Wu, Zi-Yi Dou, Qingyuan Hu, Yu Hou, Nischal Chandra, Marjorie Freedman, Ralph Weischedel, Nanyun Peng
| Challenge: | despite its importance, there are few datasets that cover multimodal counterfactual reasoning . a dataset focusing on this area is limited because of its limited coverage over synthetic environments . |
| Approach: | They develop a video question answering dataset that provides questions on multimodal reasoning . they ask questions about counterfactual hypotheses over visual events . |
| Outcome: | The proposed dataset shows a significant performance gap between models and humans . it provides questions that span physical, social, and temporal dimensions . |
Content Planning for Neural Story Generation with Aristotelian Rescoring (2020.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to narrative composition are plagued by difficulty in mastering structure, will veer between topics, and lack long-range cohesion. |
| Approach: | They propose a plot-generation language model and a set of rescoring models that implement an aspect of good story-writing as detailed in Aristotle's Poetics. |
| Outcome: | The proposed system improves the quality of the narrative generated from the proposed model and improves its relevance to a given prompt and quality of stories written with our principled plot structure. |
Plot-guided Adversarial Example Construction for Evaluating Open-domain Story Generation (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to generate implausible stories using plots are unnatural and oversimplify the characteristics of implusible machine-generated stories. |
| Approach: | They propose to generate a more comprehensive set of implausible stories using plots . plots are structured representations of controllable factors used to generate stories . |
| Outcome: | The proposed model improves the quality of generated implausible stories using plots . it shows that the evaluation metrics trained on the generated data correlate better with human judgments compared to baselines. |
Learning to Generalize for Sequential Decision Making (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances show strong evidence of generalization in spatiotemporal modalities such as robotic manipulation. |
| Approach: | They propose a method for converting a reinforcement learning model into a natural language understanding model by a teacher-student imitation learning method. |
| Outcome: | The proposed model outperforms teacher performance on held-out decision problems by 7% and 24% on out-of-domain problems. |
Understanding Multimodal Procedural Knowledge by Sequencing Multimodal Instructional Manuals (2022.acl-long)
Copied to clipboard
| Challenge: | Current machine learning methods are incapable of efficiently utilizing multimodal information. |
| Approach: | They propose to use text-and-image alignment to improve machine learning's performance on multimodal event sequencing. |
| Outcome: | The proposed models perform significantly worse than humans on multimodal event sequencing than humans. |
When ACE met KBP: End-to-End Evaluation of Knowledge Base Population with Component-level Annotation (L18-1)
Copied to clipboard
| Challenge: | Automating constructing a Knowledge Base from unstructured text is a goal of natural language processing. |
| Approach: | They propose a method to evaluate a Knowledge Base population from unstructured text . they propose bootstrap resampling to provide statistical significance to the results . |
| Outcome: | The proposed method uses component-level annotations to evaluate Cold Start KBP . it also uses bootstrap resampling to provide statistical significance to the results reported . |
Remember what you did so you know what to do next (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown large language models (LLMs) to be poor fit for a simulated robot to achieve 30 classes of goals. |
| Approach: | They use the 6B parameter GPT-J language model to create a plan for a simulated robot to achieve 30 classes of goals in ScienceWorld. |
| Outcome: | The proposed model outperforms the state-of-the-art by a factor of 1.4 when training on as many prior steps as will fit, and the results are 2.3x better than the state of the-art. |