Papers with NoCha
One Thousand and One Pairs: A “novel” challenge for long-context language models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing long-context evaluation methods measure surface-level retrieval capabilities, but do not assess performance on the more challenging task of synthesizing distant and underlying information. |
| Approach: | They propose a dataset of 1,001 minimally different pairs of true and false claims about 67 recently-published English fictional books. |
| Outcome: | The proposed model performs better on pairs that require only sentence-level retrieval vs. global reasoning . the proposed model also performs worse on speculative fiction books with extensive world-building . |