Papers with CommitmentBank
Comparing Test Sets with Item Response Theory (2021.acl-long)
Copied to clipboard
Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, Samuel R. Bowman
| Challenge: | Recent results from large pretrained models show that many datasets are saturated and unlikely to detect further progress. |
| Approach: | They evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. |
| Outcome: | The proposed datasets are saturated and unlikely to detect future improvements. |
Identifying inherent disagreement in natural language inference (2021.naacl-main)
Copied to clipboard
| Challenge: | Natural language inference is the task of determining whether text is entailed, contradicted or unrelated to another piece of text. |
| Approach: | They propose to tease systematic inferences from disagreement items by capturing modes in annotations to simulate uncertainty in the annotation process. |
| Outcome: | The proposed approach performs statistically better than baselines on the CommitmentBank corpus in English. |
Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment (P19-1)
Copied to clipboard
| Challenge: | Existing models for speaker commitment fail to generalize to diverse linguistic constructions, highlighting directions for improvement. |
| Approach: | They evaluate two state-of-the-art speaker commitment models on the CommitmentBank . they analyze linguistic correlates of model error on a naturalistic dataset . |
| Outcome: | The proposed models perform well on some classes but fail to generalize to diverse linguistic constructions. |
Evaluating BERT for natural language inference: A case study on the CommitmentBank (D19-1)
Copied to clipboard
| Challenge: | Natural language inference datasets can identify premise-hypothesis relationship without observing premise . recasting of the CommitmentBank for NLI creates hypotheses that stand in entailment/contradiction/neutral relationship with premise. |
| Approach: | They propose to recast the CommitmentBank for NLI to stand in certain relationships with the premise . hypotheses are complements of clause-embedding verbs in each premise, rethinking the CommittedBank . |
| Outcome: | The proposed model performs well on the CommitmentBank with 85% F1 . however, the model does not capture the full complexity of pragmatic reasoning, authors say . |