Papers by Marcelo Mendoza
Augmenting BERT-style Models with Predictive Coding to Improve Discourse-level Representations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models do not produce suitable representations at the discourse level. |
| Approach: | They propose to augment BERT-style language models with a mechanism that allows them to learn suitable discourse-level representations by incorporating top-down connections that operate at the intermediate layers of the network. |
| Outcome: | The proposed approach improves in 6 out of 11 tasks by detecting discourse relationship detection. |
Evaluation Benchmarks for Spanish Sentence Representations (2022.lrec-1)
Copied to clipboard
Vladimir Araujo, Andrés Carvallo, Souvik Kundu, José Cañete, Marcelo Mendoza, Robert E. Mercer, Felipe Bravo-Marquez, Marie-Francine Moens, Alvaro Soto
| Challenge: | Existing and newly constructed datasets address different tasks from various domains. |
| Approach: | They propose to use Spanish SentEval and Spanish DiscoEval to evaluate stand-alone and discourse-aware sentence representations. |
| Outcome: | The proposed benchmarks evaluate the capabilities of stand-alone and discourse-aware sentence representations in Spanish and show that they are more robust and comparable than previous benchmarks. |
Inspecting the concept knowledge graph encoded by modern language models (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models are used to solve tasks such as summarization and information retrieval. |
| Approach: | They propose to use word embeddings, text generators, context encoders to extract underlying knowledge graphs of nine influential language models. |
| Outcome: | The proposed model is able to encode word embeddings, text generators, and context encoders, but suffers from several inaccuracies. |