Papers by Jan-Micha Bodensohn
Document Structure in Long Document Transformers (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing long-document Transformers do not learn representations of document structure during pretraining. |
| Approach: | They propose to use long-document Transformers to acquire an internal representation of document structure during pre-training and evaluate the effects of structure infusion on QASPER and Evidence Inference. |
| Outcome: | The proposed models acquire implicit understanding of document structure during pre-training, which can be enhanced by structure infusion, leading to improved end-task performance. |