HPLTDatasets v1.1

DEPRECATED RELEASE

This version ellaborates on the previous one, version 1.0, which was made of mostly raw plain text data. De-duplication is now applied to both monolingual and bilingual datasets. Further cleaning is applied to the bilingual datasets. At the same time, the texts come with some metadata, which is currently unused but can be employed by the end users to conduct their own filtering.
Please check this report for a full description of the original data release (version 1.0), but note that the corpus statistics are very different due to deduplication and further cleaning.

Monolingual

Data release 1.1 (October 2023)

There are 75 languages in this release (7.6 TB in total size) provided as JSONL files compressed with zstd. For convenience, data is split into multiple shards, a few GB each. The number of shards per language depends on the size of the specific corpus.

The format is JSONL, where each line is a valid JSON value and a full document with metadata. For example:

{"id":1, "document_lang":"en",
"scores":["0.76","0.76","0.76"],
"langs":["en","en","en"],
"text":"this is paragraph1\nthis is paragraph2\nthis is paragraph3",
"url":"url1", "collection":"collection-1"
}
{"id":2, "document_lang":"en",
"scores":["0.65",...],
"langs":["en",...],
"text":"another paragraph\n...",
...

In each document, each paragraph is concatenated using new-line separators. langs and scores are lists containing one entry per paragraph, corresponding to the language identified and monocleaner score of each one.

How to download it?

The simplest way to download the data is to use wget -i with the language-specific mapping files containing full URLs for all shards of this particular language, for example:

wget -i https://data.hplt-project.org/one/monotext/deduplicated/eu_map.txt

Full download

If you want to download all the 7 Terabytes in one click, use

wget -nH -x --cut-dirs=2 -i https://data.hplt-project.org/one/monotext/deduplicated/hplt_monolingual_map_all_1.2.txt

Choose a language

Afrikaans (af)

raw 45.31 GB dedup 13.48 GB

Source: CC/IA

RAW:

Docs: 4.36M

Words: 6.09B

DEDUPLICATED:

Docs: 1.37M

Words: 1.87B

Arabic (ar)

raw 2.19 TB dedup 527.15 GB

Source: CC/IA

RAW:

Docs: 197.49M

Words: 216.32B

DEDUPLICATED:

Docs: 46.64M

Words: 51.53B

Azerbaijani (az)

raw 91.53 GB dedup 25.21 GB

Source: CC/IA

RAW:

Docs: 10.45M

Words: 10.68B

DEDUPLICATED:

Docs: 3.00M

Words: 2.94B

Belarusian (be)

raw 68.12 GB dedup 19.34 GB

Source: CC/IA

RAW:

Docs: 3.91M

Words: 4.71B

DEDUPLICATED:

Docs: 1.26M

Words: 1.39B

Bulgarian (bg)

raw 742.68 GB dedup 173.05 GB

Source: CC/IA

RAW:

Docs: 58.27M

Words: 66.37B

DEDUPLICATED:

Docs: 13.34M

Words: 15.48B

Bangla (bn)

raw 726.29 GB dedup 129.34 GB

Source: CC/IA

RAW:

Docs: 23.35M

Words: 46.18B

DEDUPLICATED:

Docs: 5.97M

Words: 8.23B

Catalan (ca)

raw 160.81 GB dedup 50.95 GB

Source: CC/IA

RAW:

Docs: 24.17M

Words: 24.53B

DEDUPLICATED:

Docs: 7.79M

Words: 7.88B

Czech (cs)

raw 1.37 TB dedup 287.29 GB

Source: CC/IA

RAW:

Docs: 184.09M

Words: 192.48B

DEDUPLICATED:

Docs: 38.56M

Words: 41.04B

Welsh (cy)

raw 4.28 GB dedup 1.64 GB

Source: CC/IA

RAW:

Docs: 725.32K

Words: 623.99M

DEDUPLICATED:

Docs: 285.05K

Words: 243.81M

Danish (da)

raw 944.15 GB dedup 156.17 GB

Source: CC/IA

RAW:

Docs: 686.47M

Words: 139.54B

DEDUPLICATED:

Docs: 23.58M

Words: 22.10B

German (de)

raw 6.83 TB dedup 1.46 TB

Source: CC/IA

RAW:

Docs: 1.02B

Words: 899.14B

DEDUPLICATED:

Docs: 226.47M

Words: 193.11B

Greek (el)

raw 2.57 TB dedup 546.83 GB

Source: CC/IA

RAW:

Docs: 134.47M

Words: 248.08B

DEDUPLICATED:

Docs: 30.63M

Words: 52.24B

English (en)

raw 68.22 TB dedup 20.34 TB

Source: CC/IA

RAW:

Docs: 12.68B

Words: 10.27T

DEDUPLICATED:

Docs: 1.78B

Words: 2.86T

Esperanto (eo)

raw 2.07 GB dedup 991.32 MB

Source: CC/IA

RAW:

Docs: 400.65K

Words: 295.87M

DEDUPLICATED:

Docs: 177.13K

Words: 147.06M

Spanish (es)

raw 5.51 TB dedup 1.60 TB

Source: CC/IA

RAW:

Docs: 671.98M

Words: 869.38B

DEDUPLICATED:

Docs: 201.06M

Words: 250.08B

Estonian (et)

raw 170.57 GB dedup 46.04 GB

Source: CC/IA

RAW:

Docs: 21.18M

Words: 21.75B

DEDUPLICATED:

Docs: 5.84M

Words: 5.85B

Basque (eu)

raw 11.44 GB dedup 5.30 GB

Source: CC/IA

RAW:

Docs: 2.29M

Words: 1.55B

DEDUPLICATED:

Docs: 1.01M

Words: 717.14M

Persian (fa)

raw 2.93 TB dedup 645.39 GB

Source: CC/IA

RAW:

Docs: 189.04M

Words: 318.13B

DEDUPLICATED:

Docs: 42.28M

Words: 70.38B

Finnish (fi)

raw 673.58 GB dedup 142.16 GB

Source: CC/IA

RAW:

Docs: 89.06M

Words: 80.59B

DEDUPLICATED:

Docs: 19.51M

Words: 16.85B

French (fr)

raw 5.25 TB dedup 1.41 TB

Source: CC/IA

RAW:

Docs: 660.32M

Words: 791.82B

DEDUPLICATED:

Docs: 175.76M

Words: 213.84B

Irish (ga)

raw 9.97 GB dedup 3.61 GB

Source: CC/IA

RAW:

Docs: 2.71M

Words: 1.43B

DEDUPLICATED:

Docs: 931.55K

Words: 520.10M

Galician (gl)

raw 33.97 GB dedup 8.56 GB

Source: CC/IA

RAW:

Docs: 4.60M

Words: 5.02B

DEDUPLICATED:

Docs: 1.79M

Words: 1.29B

Gujarati (gu)

raw 13.24 GB dedup 6.10 GB

Source: CC/IA

RAW:

Docs: 915.29K

Words: 1.06B

DEDUPLICATED:

Docs: 454.73K

Words: 473.46M

Serbo-Croatian (hbs)

raw 518.89 GB dedup 145.11 GB

Source: CC/IA

RAW:

Docs: 60.64M

Words: 69.45B

DEDUPLICATED:

Docs: 17.84M

Words: 19.40B

Hebrew (he)

raw 603.15 GB dedup 151.60 GB

Source: CC/IA

RAW:

Docs: 46.22M

Words: 61.85B

DEDUPLICATED:

Docs: 11.24M

Words: 15.45B

Hindi (hi)

raw 488.64 GB dedup 162.28 GB

Source: CC/IA

RAW:

Docs: 33.67M

Words: 41.21B

DEDUPLICATED:

Docs: 11.42M

Words: 13.65B

Hungarian (hu)

raw 1.12 TB dedup 216.64 GB

Source: CC/IA

RAW:

Docs: 137.33M

Words: 137.25B

DEDUPLICATED:

Docs: 28.49M

Words: 26.54B

Armenian (hy)

raw 47.53 GB dedup 15.40 GB

Source: CC/IA

RAW:

Docs: 3.99M

Words: 4.06B

DEDUPLICATED:

Docs: 1.36M

Words: 1.31B

Indonesian (id)

raw 1.45 TB dedup 483.75 GB

Source: CC/IA

RAW:

Docs: 125.73M

Words: 208.16B

DEDUPLICATED:

Docs: 45.77M

Words: 69.22B

Icelandic (is)

raw 26.09 GB dedup 9.99 GB

Source: CC/IA

RAW:

Docs: 3.86M

Words: 3.48B

DEDUPLICATED:

Docs: 1.44M

Words: 1.34B

Italian (it)

raw 2.79 TB dedup 777.14 GB

Source: CC/IA

RAW:

Docs: 337.44M

Words: 405.66B

DEDUPLICATED:

Docs: 96.50M

Words: 114.44B

Japanese (ja)

raw 5.97 TB dedup 1.98 TB

Source: CC/IA

RAW:

Docs: 679.79M

Words: 305.03B

DEDUPLICATED:

Docs: 218.85M

Words: 91.41B

Georgian (ka)

raw 105.10 GB dedup 25.07 GB

Source: CC/IA

RAW:

Docs: 6.46M

Words: 7.19B

DEDUPLICATED:

Docs: 1.67M

Words: 1.68B

Kazakh (kk)

raw 33.77 GB dedup 13.65 GB

Source: CC/IA

RAW:

Docs: 3.46M

Words: 2.53B

DEDUPLICATED:

Docs: 1.43M

Words: 1.01B

Kannada (kn)

raw 29.45 GB dedup 8.71 GB

Source: CC/IA

RAW:

Docs: 1.93M

Words: 2.10B

DEDUPLICATED:

Docs: 557.82K

Words: 577.95M

Korean (ko)

raw 1.56 TB dedup 279.76 GB

Source: CC/IA

RAW:

Docs: 248.06M

Words: 161.42B

DEDUPLICATED:

Docs: 44.46M

Words: 29.00B

Kyrgyz (ky)

raw 3.48 GB dedup 1.96 GB

Source: CC/IA

RAW:

Docs: 333.58K

Words: 263.05M

DEDUPLICATED:

Docs: 188.17K

Words: 146.44M

Latin (la)

raw 104.45 GB dedup 24.20 GB

Source: CC/IA

RAW:

Docs: 20.54M

Words: 14.38B

DEDUPLICATED:

Docs: 4.81M

Words: 3.32B

Lithuanian (lt)

raw 252.84 GB dedup 56.73 GB

Source: CC/IA

RAW:

Docs: 32.35M

Words: 33.12B

DEDUPLICATED:

Docs: 7.40M

Words: 7.78B

Latvian (lv)

raw 220.92 GB dedup 42.31 GB

Source: CC/IA

RAW:

Docs: 21.15M

Words: 27.37B

DEDUPLICATED:

Docs: 5.12M

Words: 5.39B

Macedonian (mk)

raw 34.73 GB dedup 11.99 GB

Source: CC/IA

RAW:

Docs: 3.41M

Words: 3.24B

DEDUPLICATED:

Docs: 1.25M

Words: 1.11B

Malayalam (ml)

raw 43.21 GB dedup 19.39 GB

Source: CC/IA

RAW:

Docs: 2.19M

Words: 2.67B

DEDUPLICATED:

Docs: 1.13M

Words: 1.05B

Mongolian (mn)

raw 29.23 GB dedup 12.70 GB

Source: CC/IA

RAW:

Docs: 2.50M

Words: 2.49B

DEDUPLICATED:

Docs: 1.06M

Words: 1.09B

Marathi (mr)

raw 27.08 GB dedup 13.12 GB

Source: CC/IA

RAW:

Docs: 1.64M

Words: 1.91B

DEDUPLICATED:

Docs: 857.21K

Words: 911.64M

Malay (ms)

raw 351.87 GB dedup 102.48 GB

Source: CC/IA

RAW:

Docs: 28.32M

Words: 56.68B

DEDUPLICATED:

Docs: 8.36M

Words: 16.50B

Maltese (mt)

raw 10.08 GB dedup 6.01 GB

Source: CC/IA

RAW:

Docs: 926.22K

Words: 1.39B

DEDUPLICATED:

Docs: 484.42K

Words: 818.61M

Burmese (my)

raw 61.30 GB dedup 19.70 GB

Source: CC/IA

RAW:

Docs: 2.46M

Words: 4.41B

DEDUPLICATED:

Docs: 826.11K

Words: 1.25B

Norwegian Bokmål (nb)

raw 379.45 GB dedup 94.06 GB

Source: CC/IA

RAW:

Docs: 61.09M

Words: 55.51B

DEDUPLICATED:

Docs: 14.58M

Words: 13.90B

Nepali (ne)

raw 24.34 GB dedup 15.58 GB

Source: CC/IA

RAW:

Docs: 2.11M

Words: 1.68B

DEDUPLICATED:

Docs: 1.36M

Words: 1.06B

Dutch (nl)

raw 1.83 TB dedup 523.45 GB

Source: CC/IA

RAW:

Docs: 234.44M

Words: 250.08B

DEDUPLICATED:

Docs: 66.62M

Words: 71.75B

Norwegian Nynorsk (nn)

raw 11.66 GB dedup 4.46 GB

Source: CC/IA

RAW:

Docs: 1.85M

Words: 1.64B

DEDUPLICATED:

Docs: 752.53K

Words: 640.11M

Punjabi (pa)

raw 16.04 GB dedup 6.41 GB

Source: CC/IA

RAW:

Docs: 2.40M

Words: 1.34B

DEDUPLICATED:

Docs: 888.47K

Words: 532.71M

Polish (pl)

raw 2.65 TB dedup 617.29 GB

Source: CC/IA

RAW:

Docs: 346.33M

Words: 366.18B

DEDUPLICATED:

Docs: 82.92M

Words: 85.41B

Pashto (ps)

raw 2.81 GB dedup 1.32 GB

Source: CC/IA

RAW:

Docs: 313.15K

Words: 356.65M

DEDUPLICATED:

Docs: 142.66K

Words: 167.53M

Portuguese (pt)

raw 3.91 TB dedup 852.98 GB

Source: CC/IA

RAW:

Docs: 448.20M

Words: 607.05B

DEDUPLICATED:

Docs: 103.82M

Words: 132.09B

Romanian (ro)

raw 1.12 TB dedup 246.55 GB

Source: CC/IA

RAW:

Docs: 103.69M

Words: 144.82B

DEDUPLICATED:

Docs: 24.94M

Words: 32.80B

Russian (ru)

raw 21.76 TB dedup 6.02 TB

Source: CC/IA

RAW:

Docs: 1.53B

Words: 1.79T

DEDUPLICATED:

Docs: 397.27M

Words: 492.53B

Sinhala (si)

raw 30.04 GB dedup 11.88 GB

Source: CC/IA

RAW:

Docs: 1.37M

Words: 2.35B

DEDUPLICATED:

Docs: 563.99K

Words: 917.69M

Slovak (sk)

raw 693.56 GB dedup 101.85 GB

Source: CC/IA

RAW:

Docs: 90.43M

Words: 94.65B

DEDUPLICATED:

Docs: 13.99M

Words: 13.95B

Slovenian (sl)

raw 160.99 GB dedup 49.00 GB

Source: CC/IA

RAW:

Docs: 19.17M

Words: 22.85B

DEDUPLICATED:

Docs: 5.82M

Words: 7.04B

Somali (so)

raw 3.64 GB dedup 2.01 GB

Source: CC/IA

RAW:

Docs: 676.04K

Words: 544.50M

DEDUPLICATED:

Docs: 374.71K

Words: 299.68M

Albanian (sq)

raw 80.95 GB dedup 26.55 GB

Source: CC/IA

RAW:

Docs: 9.18M

Words: 11.50B

DEDUPLICATED:

Docs: 3.22M

Words: 3.78B

Swedish (sv)

raw 663.34 GB dedup 195.34 GB

Source: CC/IA

RAW:

Docs: 96.13M

Words: 95.99B

DEDUPLICATED:

Docs: 29.96M

Words: 28.26B

Swahili (sw)

raw 13.64 GB dedup 5.94 GB

Source: CC/IA

RAW:

Docs: 2.17M

Words: 2.10B

DEDUPLICATED:

Docs: 983.51K

Words: 910.98M

Tamil (ta)

raw 157.27 GB dedup 65.83 GB

Source: CC/IA

RAW:

Docs: 5.46M

Words: 9.30B

DEDUPLICATED:

Docs: 2.47M

Words: 3.87B

Telugu (te)

raw 35.86 GB dedup 15.94 GB

Source: CC/IA

RAW:

Docs: 3.51M

Words: 2.43B

DEDUPLICATED:

Docs: 1.61M

Words: 1.06B

Thai (th)

raw 1.39 TB dedup 405.36 GB

Source: CC/IA

RAW:

Docs: 95.27M

Words: 78.63B

DEDUPLICATED:

Docs: 29.48M

Words: 21.99B

Filipino (tl)

raw 47.36 GB dedup 11.56 GB

Source: CC/IA

RAW:

Docs: 4.97M

Words: 7.07B

DEDUPLICATED:

Docs: 1.20M

Words: 1.63B

Turkish (tr)

raw 1.81 TB dedup 492.82 GB

Source: CC/IA

RAW:

Docs: 215.38M

Words: 238.30B

DEDUPLICATED:

Docs: 59.43M

Words: 64.92B

Tatar (tt)

raw 2.97 GB dedup 1.63 GB

Source: CC/IA

RAW:

Docs: 368.35K

Words: 261.38M

DEDUPLICATED:

Docs: 172.47K

Words: 134.18M

Ukrainian (uk)

raw 655.65 GB dedup 230.81 GB

Source: CC/IA

RAW:

Docs: 47.12M

Words: 52.95B

DEDUPLICATED:

Docs: 17.86M

Words: 18.19B

Urdu (ur)

raw 50.52 GB dedup 16.13 GB

Source: CC/IA

RAW:

Docs: 6.09M

Words: 6.59B

DEDUPLICATED:

Docs: 2.23M

Words: 2.02B

Uzbek (uz)

raw 12.04 GB dedup 6.12 GB

Source: CC/IA

RAW:

Docs: 1.37M

Words: 1.11B

DEDUPLICATED:

Docs: 633.22K

Words: 556.45M

Vietnamese (vi)

raw 1.73 TB dedup 391.80 GB

Source: CC/IA

RAW:

Docs: 174.17M

Words: 287.20B

DEDUPLICATED:

Docs: 40.10M

Words: 64.97B

Chinese (zh)

raw 39.97 TB dedup 7.51 TB

Source: CC/IA

RAW:

Docs: 6.91B

Words: 1.79T

DEDUPLICATED:

Docs: 1.20B

Words: 334.58B

Bilingual

Data release 1.1 (October 2023)

There are 18 language pairs in this release. The parallel corpus contains over 96 million clean and unique sentence pairs and covers over 1.4 billion English tokens. The corpora are provided in TMX and TXT compressed formats derived from the raw files published in the previous release. These corpora have been highly curated, de-duplicated and filtered using the full Bitextor pipeline.

Choose a language pair

Arabic (ar) - English (en)

moses 1.51 GB tmx 7.40 GB

Source: CC/IA

Stats:

Src lang tokens: 29.29M

Translation units: 14.65M

Bosnian (bs) - English (en)

moses 14.98 MB tmx 15.77 MB

Source: CC/IA

Stats:

Src lang tokens: 2.64M

Translation units: 240.01K

Catalan (ca) - English (en)

moses 825.38 MB tmx 840.28 MB

Source: CC/IA

Stats:

Src lang tokens: 149.86M

Translation units: 8.91M

English (en) - Estonian (et)

moses 541.46 MB tmx 562.77 MB

Source: CC/IA

Stats:

Src lang tokens: 95.94M

Translation units: 6.09M

English (en) - Basque (eu)

moses 57.91 MB tmx 176.89 MB

Source: CC/IA

Stats:

Src lang tokens: 1.22M

Translation units: 610.69K

English (en) - Finnish (fi)

moses 2.08 GB tmx 1.90 GB

Source: CC/IA

Stats:

Src lang tokens: 50.35M

Translation units: 25.18M

English (en) - Irish (ga)

moses 87.00 MB tmx 93.54 MB

Source: CC/IA

Stats:

Src lang tokens: 16.33M

Translation units: 994.75K

English (en) - Galician (gl)

moses 84.24 MB tmx 279.09 MB

Source: CC/IA

Stats:

Src lang tokens: 2.13M

Translation units: 1.06M

English (en) - Hindi (hi)

moses 1.17 GB tmx 1.22 GB

Source: CC/IA

Stats:

Src lang tokens: 165.14M

Translation units: 12.04M

English (en) - Croatian (hr)

moses 833.38 MB tmx 872.52 MB

Source: CC/IA

Stats:

Src lang tokens: 138.36M

Translation units: 9.31M

English (en) - Icelandic (is)

moses 146.87 MB tmx 849.71 MB

Source: CC/IA

Stats:

Src lang tokens: 4.30M

Translation units: 2.15M

English (en) - Macedonian (mk)

moses 122.18 MB tmx 643.29 MB

Source: CC/IA

Stats:

Src lang tokens: 2.28M

Translation units: 1.14M

English (en) - Maltese (mt)

moses 103.87 MB tmx 382.54 MB

Source: CC/IA

Stats:

Src lang tokens: 1.71M

Translation units: 854.82K

English (en) - Norwegian Nynorsk (nn)

moses 11.85 MB tmx 32.53 MB

Source: CC/IA

Stats:

Src lang tokens: 265.08K

Translation units: 132.54K

English (en) - Albanian (sq)

moses 151.32 MB tmx 720.56 MB

Source: CC/IA

Stats:

Src lang tokens: 3.31M

Translation units: 1.66M

English (en) - Serbian (sr)

moses 319.21 MB tmx 333.89 MB

Source: CC/IA

Stats:

Src lang tokens: 55.12M

Translation units: 3.90M

English (en) - Swahili (sw)

moses 99.89 MB tmx 95.13 MB

Source: CC/IA

Stats:

Src lang tokens: 20.04M

Translation units: 1.71M

English (en) - Traditional Chinese (zh-hant)

moses 508.09 MB tmx 539.63 MB

Source: CC/IA

Stats:

Src lang tokens: 84.11M

Translation units: 5.31M

License

These data are released under this licensing scheme:

We do not own any of the text from which these text data has been extracted.*
We license the actual packaging of these text data under the Creative Commons CC0 license ("no rights reserved") .

Notice and take down policy

Notice: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please:

Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted.
Clearly identify the copyrighted work claimed to be infringed.
Clearly identify the material that is claimed to be infringing and information reasonably sufficient to allow us to locate the material.
You can reach us at hplt-datasets@ufal.mff.cuni.cz

Take down: We will comply to legitimate requests by removing the affected sources from the next release of the corpora.

*It is your resposibility that any use of the data complies with any applicable legal framework, such as, among others, the EU Copyright Directive 2019/790 and the General Data Protection Regulation 2018, as amended.