ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Contract review is a time-consuming procedure that costs companies millions of dollars each year . linguistic characteristics of contracts, such as negations by exceptions, contribute to the difficulty of this task . |
| Approach: | They propose a document-level natural language inference (NLI) task for contracts . they annotate and release the largest corpus to date consisting of 607 annotated contracts a linguistically rich system is proposed . |
| Outcome: | The proposed system is based on a contract review task that includes 607 annotated contracts. |
Similar Papers
DocNLI: A Large-scale Dataset for Document-level Natural Language Inference (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on sentence-level inference, which limits its application in downstream NLP problems. |
| Approach: | They propose to construct a large-scale dataset for document-level NLI that can be used to study NLP problems. |
| Outcome: | The proposed model performs well on popular sentence-level benchmarks and generalizes well to out-of-domain NLP tasks that rely on inference at document granularity. |
DocInfer: Document-level Natural Language Inference using Optimal Evidence Selection (2022.emnlp-main)
Copied to clipboard
| Challenge: | Documentlevel NLI is an important problem for many tasks including verification of factual correctness of documents. |
| Approach: | They propose a document-level natural language inference model that builds a hierarchical document graph enriched through inter-sentence relations and performs paragraph pruning using the novel SubGraph Pooling layer. |
| Outcome: | The proposed model performs on a legal judicial reasoning task with a dataset enriched with document graphs and a proposed evidence selection algorithm. |
Contract Discovery: Dataset and a Few-Shot Semantic Retrieval Challenge with Competitive Baselines (2020.findings-emnlp)
Copied to clipboard
Łukasz Borchmann, Dawid Wisniewski, Andrzej Gretkowski, Izabela Kosmala, Dawid Jurkiewicz, Łukasz Szałkiewicz, Gabriela Pałka, Karol Kaczmarek, Agnieszka Kaliska, Filip Graliński
| Challenge: | Existing methods for detecting text fragments are not suitable for contract discovery, since it requires manual definition of a few examples, followed by conventional information. |
| Approach: | They propose a task where legal clauses are extracted from documents, given a few examples of similar clauses from other legal acts. |
| Outcome: | The proposed task differs substantially from conventional NLI and shared tasks on legal information extraction. |
Validity Assessment of Legal Will Statements as Natural Language Inference (2022.findings-emnlp)
Copied to clipboard
| Challenge: | This study introduces a dataset that focuses on the validity of statements in legal wills. |
| Approach: | They propose a dataset that focuses on the validity of statements in legal wills. |
| Outcome: | The proposed model achieves 80% macro F1 and accuracy, but group accuracy is in mid 80s at best, suggesting that the models’ understanding of the task remains superficial. |
Evaluating BERT for natural language inference: A case study on the CommitmentBank (D19-1)
Copied to clipboard
| Challenge: | Natural language inference datasets can identify premise-hypothesis relationship without observing premise . recasting of the CommitmentBank for NLI creates hypotheses that stand in entailment/contradiction/neutral relationship with premise. |
| Approach: | They propose to recast the CommitmentBank for NLI to stand in certain relationships with the premise . hypotheses are complements of clause-embedding verbs in each premise, rethinking the CommittedBank . |
| Outcome: | The proposed model performs well on the CommitmentBank with 85% F1 . however, the model does not capture the full complexity of pragmatic reasoning, authors say . |
PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing systems for Natural Language Inference (NLI) only recognize textual entailment relations on sentence-level . however, even a simple sentence often contains multiple propositions, i.e. distinct units of meaning conveyed by the sentence . |
| Approach: | They propose a system to recognize whether one text is textually entailed by another . they use a corpus of over 45K propositions annotated by human raters to study the textual entailment relation of each proposition in a sentence individually. |
| Outcome: | The proposed dataset can be used to understand the compositionality of NLI labels. |
A Contract Corpus for Recognizing Rights and Obligations (2020.lrec-1)
Copied to clipboard
| Challenge: | Understanding the content of a contract is often difficult and costly, especially if the contract is long and complex. |
| Approach: | They describe how they built an annotated corpus of contract documents that can be used to recognize rights and obligations. |
| Outcome: | The proposed system can recognize parties' rights and obligations based on 46 English contracts and 25 Japanese contracts drafted by lawyers. |
Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in modeling and datasets demonstrate promising performance for NLI. |
| Approach: | They explore the direct zero-shot applicability of NLI models to real applications . they analyze the robustness of models to longer and out-of-domain inputs . |
| Outcome: | The proposed models are robust to longer and out-of-domain inputs and can perform on full documents. |
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (N18-1)
Copied to clipboard
| Challenge: | et al., 1996, show that many of the most actively studied problems in NLP depend in large part on natural language understanding (NLU). |
| Approach: | They propose a dataset for machine learning that uses ten different genres of English to evaluate sentences for their meanings. |
| Outcome: | The multi-genre natural language inference corpus is one of the largest available for natural language understanding. |
SciNLI: A Corpus for Natural Language Inference on Scientific Text (2022.acl-long)
Copied to clipboard
| Challenge: | Existing Natural Language Inference (NLI) datasets are not related to scientific text. |
| Approach: | They propose a large dataset for NLI that captures the formality in scientific text and contains 107,412 sentence pairs extracted from scholarly papers on NLP and computational linguistics. |
| Outcome: | The proposed model achieves a Macro F1 score of only 78.18% and an accuracy of 78.23%. |