| Challenge: | a corpus of 1.49 million parallel segments is available in the public domain . the corpus is the largest publicly available English-Hindi parallel corpus . |
| Approach: | They present the IIT Bombay English-Hindi Parallel Corpus . they present a compilation of public and private parallel corpora . |
| Outcome: | The corpus contains 1.49 million parallel segments, of which 694k were not previously available in the public domain. |
Similar Papers
A Multilingual Parallel Corpora Collection Effort for Indian Languages (2020.lrec-1)
Copied to clipboard
| Challenge: | Currently, neural network based approaches for machine translation are data hungry and sentence-level aligned parallel pairs are the currency. |
| Approach: | They propose to build sentence aligned parallel corpora across 10 Indian languages using online sources which have content shared across languages. |
| Outcome: | The proposed corpora significantly extends existing resources that are either not large enough or are restricted to a specific domain (such as health). |
The LTRC Hindi-Telugu Parallel Corpus (2022.lrec-1)
Copied to clipboard
| Challenge: | a qualitative corpus of 700K parallel sentences was created using multiple methods such as extract, align and review of Hindi-Telugu corpora. |
| Approach: | They propose to create a Hindi-Telugu parallel corpus of different technical domains using different methods including extract, align and review. |
| Outcome: | The proposed corpus is the largest, publicly available domain parallel corpus for Hindi-Telugu. |
Leveraging the Cross-Domain & Cross-Linguistic Corpus for Low Resource NMT: A Case Study On Bhili-Hindi-English Parallel Corpus (2025.findings-emnlp)
Copied to clipboard
| Challenge: | linguistic diversity of India poses significant machine translation challenges, authors say . underrepresented tribal languages like Bhili lack high-quality linguistic resources . |
| Approach: | They introduce a Bhili-Hindi-English Parallel Corpus, the first and largest parallel corpus worldwide . they evaluated a wide range of proprietary and open-source MLLMs on bidirectional translation tasks . |
| Outcome: | The proposed corpus spans critical domains such as education, administration, and news. |
Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages (2022.tacl-1)
Copied to clipboard
Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, Mitesh Shantadevi Khapra
| Challenge: | We present Samanantar, the largest publicly available parallel corpora collection for Indic languages . based on existing corporative, there has been limited benefit for resource-poor languages despite the lack of parallel corporals and monolingual corporata. |
| Approach: | They compile 12.4 million sentence pairs from existing corpora and mine 37.4 million from the Web. |
| Outcome: | The proposed model outperforms existing models and benchmarks on public datasets. |
IndicSpeech: Text-to-Speech Corpus for Indian Languages (2020.lrec-1)
Copied to clipboard
| Challenge: | India has 22 languages, each of them being spoken by over a million people . the current state of the art text-to-speech systems for Indian languages are lacking in the multimedia domain . |
| Approach: | They propose to train a state-of-the-art TTS system for Hindi, Malayalam and Bengali and publish the results. |
| Outcome: | The proposed system trains neural text-to-speech systems for Hindi, Malayalam and Bengali and makes them publicly available. |
The EuroPat Corpus: A Parallel Corpus of European Patent Data (2022.lrec-1)
Copied to clipboard
| Challenge: | a new corpus of patent-specific parallel data is available for 6 official European languages paired with English: German, Spanish, French, Croatian, Norwegian, and Polish. |
| Approach: | They present a patent-specific corpus of parallel data for 6 official European languages paired with English: German, Spanish, French, Croatian, Norwegian, and Polish. |
| Outcome: | The filtered corpus ranges in size from 51 million sentences (Spanish-English) to 154k sentences (Croatian-English), with the unfiltered (raw) corpus being up to 2 times larger. |
PMIndiaSum: Multilingual and Cross-lingual Headline Summarization for Languages in India (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets for Indian languages are limited in terms of coverage and size. |
| Approach: | They propose a multilingual and massively parallel summarization corpus focused on languages in India that provides a training and testing ground for four language families, 14 languages, and the largest to date with 196 language pairs. |
| Outcome: | The proposed dataset provides a training and testing ground for four language families, 14 languages, and the largest to date with 196 language pairs. |
Extended Parallel Corpus for Amharic-English Machine Translation (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing approaches to automate the complex task of translation are tedious and expensive. |
| Approach: | They describe acquisition, preprocessing, segmentation, and alignment of an Amharic-English parallel corpus. |
| Outcome: | The proposed corpus outperforms statistical machine translation models by six to seven BLEU points . the results show that the subword models outperformed word-based models by three to four BLUE points compared with the word-base models . |
A Multilingual Dataset for Evaluating Parallel Sentence Extraction from Comparable Corpora (L18-1)
Copied to clipboard
| Challenge: | BUCC Shared Task aims to extract parallel sentences from comparable corporad . resulting corpus contains about 3.5 million distinct sentences in english, french, german, Russian, and Chinese . |
| Approach: | They present challenges faced to build a parallel sentences dataset from comparable corporad . they emphasize issues faced to include Chinese as one of the languages . |
| Outcome: | The 2017 BUCC Shared Task was a first for this task . the dataset contains 3.5 million sentences in English, French, German, Russian, and Chinese . |
JParaCrawl v3.0: A Large-scale English-Japanese Parallel Corpus (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing parallel corpora for English-Japanese are limited, limiting the accuracy of machine translation models. |
| Approach: | They propose a web-based English-Japanese parallel corpus with 21 million unique sentence pairs . this is more than twice as many as the previous corpus JParaCrawl v2.0 . |
| Outcome: | The proposed corpus boosts the accuracy of machine translation models on various domains. |