| --- |
| pretty_name: mC4 |
| annotations_creators: |
| - no-annotation |
| language_creators: |
| - found |
| languages: |
| - af |
| - am |
| - ar |
| - az |
| - be |
| - bg |
| - bg-Latn |
| - bn |
| - ca |
| - ceb |
| - co |
| - cs |
| - cy |
| - da |
| - de |
| - el |
| - el-Latn |
| - en |
| - eo |
| - es |
| - et |
| - eu |
| - fa |
| - fi |
| - fil |
| - fr |
| - fy |
| - ga |
| - gd |
| - gl |
| - gu |
| - ha |
| - haw |
| - hi |
| - hi-Latn |
| - hmn |
| - ht |
| - hu |
| - hy |
| - id |
| - ig |
| - is |
| - it |
| - iw |
| - ja |
| - ja-Latn |
| - jv |
| - ka |
| - kk |
| - km |
| - kn |
| - ko |
| - ku |
| - ky |
| - la |
| - lb |
| - lo |
| - lt |
| - lv |
| - mg |
| - mi |
| - mk |
| - ml |
| - mn |
| - mr |
| - ms |
| - mt |
| - my |
| - ne |
| - nl |
| - "no" |
| - ny |
| - pa |
| - pl |
| - ps |
| - pt |
| - ro |
| - ru |
| - ru-Latn |
| - sd |
| - si |
| - sk |
| - sl |
| - sm |
| - sn |
| - so |
| - sq |
| - sr |
| - st |
| - su |
| - sv |
| - sw |
| - ta |
| - te |
| - tg |
| - th |
| - tr |
| - uk |
| - und |
| - ur |
| - uz |
| - vi |
| - xh |
| - yi |
| - yo |
| - zh |
| - zh-Latn |
| - zu |
| licenses: |
| - odc-by-1.0 |
| multilinguality: |
| - multilingual |
| size_categories: |
| - n<1K |
| - 1K<n<10K |
| - 10K<n<100K |
| - 100K<n<1M |
| - 1M<n<10M |
| - 10M<n<100M |
| - 100M<n<1B |
| - 1B<n<10B |
| source_datasets: |
| - original |
| task_categories: |
| - sequence-modeling |
| task_ids: |
| - language-modeling |
| paperswithcode_id: mc4 |
| --- |
| |
| # Dataset Card for mC4 |
|
|
| ## Table of Contents |
|
|
| - [Dataset Card for mC4](#dataset-card-for-mc4) |
| - [Table of Contents](#table-of-contents) |
| - [Dataset Description](#dataset-description) |
| - [Dataset Summary](#dataset-summary) |
| - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards) |
| - [Languages](#languages) |
| - [Dataset Structure](#dataset-structure) |
| - [Data Instances](#data-instances) |
| - [Data Fields](#data-fields) |
| - [Data Splits](#data-splits) |
| - [Dataset Creation](#dataset-creation) |
| - [Curation Rationale](#curation-rationale) |
| - [Source Data](#source-data) |
| - [Initial Data Collection and Normalization](#initial-data-collection-and-normalization) |
| - [Who are the source language producers?](#who-are-the-source-language-producers) |
| - [Annotations](#annotations) |
| - [Annotation process](#annotation-process) |
| - [Who are the annotators?](#who-are-the-annotators) |
| - [Personal and Sensitive Information](#personal-and-sensitive-information) |
| - [Considerations for Using the Data](#considerations-for-using-the-data) |
| - [Social Impact of Dataset](#social-impact-of-dataset) |
| - [Discussion of Biases](#discussion-of-biases) |
| - [Other Known Limitations](#other-known-limitations) |
| - [Additional Information](#additional-information) |
| - [Dataset Curators](#dataset-curators) |
| - [Licensing Information](#licensing-information) |
| - [Citation Information](#citation-information) |
| - [Contributions](#contributions) |
|
|
| ## Dataset Description |
|
|
| - **Homepage:** https://huggingface.co/datasets/allenai/c4 |
| - **Paper:** https://arxiv.org/abs/1910.10683 |
|
|
| ### Dataset Summary |
|
|
| A multilingual colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset: "https://commoncrawl.org". |
|
|
| This is the version prepared by AllenAI, hosted at this address: https://huggingface.co/datasets/allenai/c4 |
|
|
| 108 languages are available and are reported in the table below. |
|
|
| Note that the languages that end with "-Latn" are simply romanized variants, i.e. written using the Latin script. |
|
|
| | language code | language name | |
| |:----------------|:---------------------| |
| | af | Afrikaans | |
| | am | Amharic | |
| | ar | Arabic | |
| | az | Azerbaijani | |
| | be | Belarusian | |
| | bg | Bulgarian | |
| | bg-Latn | Bulgarian (Latin) | |
| | bn | Bangla | |
| | ca | Catalan | |
| | ceb | Cebuano | |
| | co | Corsican | |
| | cs | Czech | |
| | cy | Welsh | |
| | da | Danish | |
| | de | German | |
| | el | Greek | |
| | el-Latn | Greek (Latin) | |
| | en | English | |
| | eo | Esperanto | |
| | es | Spanish | |
| | et | Estonian | |
| | eu | Basque | |
| | fa | Persian | |
| | fi | Finnish | |
| | fil | Filipino | |
| | fr | French | |
| | fy | Western Frisian | |
| | ga | Irish | |
| | gd | Scottish Gaelic | |
| | gl | Galician | |
| | gu | Gujarati | |
| | ha | Hausa | |
| | haw | Hawaiian | |
| | hi | Hindi | |
| | hi-Latn | Hindi (Latin script) | |
| | hmn | Hmong, Mong | |
| | ht | Haitian | |
| | hu | Hungarian | |
| | hy | Armenian | |
| | id | Indonesian | |
| | ig | Igbo | |
| | is | Icelandic | |
| | it | Italian | |
| | iw | former Hebrew | |
| | ja | Japanese | |
| | ja-Latn | Japanese (Latin) | |
| | jv | Javanese | |
| | ka | Georgian | |
| | kk | Kazakh | |
| | km | Khmer | |
| | kn | Kannada | |
| | ko | Korean | |
| | ku | Kurdish | |
| | ky | Kyrgyz | |
| | la | Latin | |
| | lb | Luxembourgish | |
| | lo | Lao | |
| | lt | Lithuanian | |
| | lv | Latvian | |
| | mg | Malagasy | |
| | mi | Maori | |
| | mk | Macedonian | |
| | ml | Malayalam | |
| | mn | Mongolian | |
| | mr | Marathi | |
| | ms | Malay | |
| | mt | Maltese | |
| | my | Burmese | |
| | ne | Nepali | |
| | nl | Dutch | |
| | no | Norwegian | |
| | ny | Nyanja | |
| | pa | Punjabi | |
| | pl | Polish | |
| | ps | Pashto | |
| | pt | Portuguese | |
| | ro | Romanian | |
| | ru | Russian | |
| | ru-Latn | Russian (Latin) | |
| | sd | Sindhi | |
| | si | Sinhala | |
| | sk | Slovak | |
| | sl | Slovenian | |
| | sm | San Marino | |
| | sn | Shona | |
| | so | Somali | |
| | sq | Albanian | |
| | sr | Serbian | |
| | st | Southern Sotho | |
| | su | Sundanese | |
| | sv | Swedish | |
| | sw | Swahili | |
| | ta | Tamil | |
| | te | Telugu | |
| | tg | Tajik | |
| | th | Thai | |
| | tr | Turkish | |
| | uk | Ukrainian | |
| | und | Unknown language | |
| | ur | Urdu | |
| | uz | Uzbek | |
| | vi | Vietnamese | |
| | xh | Xhosa | |
| | yi | Yiddish | |
| | yo | Yoruba | |
| | zh | Chinese | |
| | zh-Latn | Chinese (Latin) | |
| | zu | Zulu | |
|
|
| You can load the mC4 subset of any language like this: |
|
|
| ```python |
| from datasets import load_dataset |
| |
| en_mc4 = load_dataset("mc4", "en") |
| ``` |
|
|
| And if you can even specify a list of languages: |
|
|
| ```python |
| from datasets import load_dataset |
| |
| mc4_subset_with_five_languages = load_dataset("mc4", languages=["en", "fr", "es", "de", "zh"]) |
| ``` |
|
|
| ### Supported Tasks and Leaderboards |
|
|
| mC4 is mainly intended to pretrain language models and word representations. |
|
|
| ### Languages |
|
|
| The dataset supports 108 languages. |
|
|
| ## Dataset Structure |
|
|
| ### Data Instances |
|
|
| An example form the `en` config is: |
|
|
| ``` |
| {'timestamp': '2018-06-24T01:32:39Z', |
| 'text': 'Farm Resources in Plumas County\nShow Beginning Farmer Organizations & Professionals (304)\nThere are 304 resources serving Plumas County in the following categories:\nMap of Beginning Farmer Organizations & Professionals serving Plumas County\nVictoria Fisher - Office Manager - Loyalton, CA\nAmy Lynn Rasband - UCCE Plumas-Sierra Administrative Assistant II - Quincy , CA\nShow Farm Income Opportunities Organizations & Professionals (353)\nThere are 353 resources serving Plumas County in the following categories:\nFarm Ranch And Forest Retailers (18)\nMap of Farm Income Opportunities Organizations & Professionals serving Plumas County\nWarner Valley Wildlife Area - Plumas County\nShow Farm Resources Organizations & Professionals (297)\nThere are 297 resources serving Plumas County in the following categories:\nMap of Farm Resources Organizations & Professionals serving Plumas County\nThere are 57 resources serving Plumas County in the following categories:\nMap of Organic Certification Organizations & Professionals serving Plumas County', |
| 'url': 'http://www.californialandcan.org/Plumas/Farm-Resources/'} |
| ``` |
|
|
| ### Data Fields |
|
|
| The data have several fields: |
|
|
| - `url`: url of the source as a string |
| - `text`: text content as a string |
| - `timestamp`: timestamp as a string |
|
|
| ### Data Splits |
|
|
| To build mC4, the authors used [CLD3](https://github.com/google/cld3) to identify over 100 languages. The resulting mC4 subsets for each language are reported in this table: |
|
|
| | config | train | validation | |
| |:---------|:--------|:-------------| |
| | af | ? | ? | |
| | am | ? | ? | |
| | ar | ? | ? | |
| | az | ? | ? | |
| | be | ? | ? | |
| | bg | ? | ? | |
| | bg-Latn | ? | ? | |
| | bn | ? | ? | |
| | ca | ? | ? | |
| | ceb | ? | ? | |
| | co | ? | ? | |
| | cs | ? | ? | |
| | cy | ? | ? | |
| | da | ? | ? | |
| | de | ? | ? | |
| | el | ? | ? | |
| | el-Latn | ? | ? | |
| | en | ? | ? | |
| | eo | ? | ? | |
| | es | ? | ? | |
| | et | ? | ? | |
| | eu | ? | ? | |
| | fa | ? | ? | |
| | fi | ? | ? | |
| | fil | ? | ? | |
| | fr | ? | ? | |
| | fy | ? | ? | |
| | ga | ? | ? | |
| | gd | ? | ? | |
| | gl | ? | ? | |
| | gu | ? | ? | |
| | ha | ? | ? | |
| | haw | ? | ? | |
| | hi | ? | ? | |
| | hi-Latn | ? | ? | |
| | hmn | ? | ? | |
| | ht | ? | ? | |
| | hu | ? | ? | |
| | hy | ? | ? | |
| | id | ? | ? | |
| | ig | ? | ? | |
| | is | ? | ? | |
| | it | ? | ? | |
| | iw | ? | ? | |
| | ja | ? | ? | |
| | ja-Latn | ? | ? | |
| | jv | ? | ? | |
| | ka | ? | ? | |
| | kk | ? | ? | |
| | km | ? | ? | |
| | kn | ? | ? | |
| | ko | ? | ? | |
| | ku | ? | ? | |
| | ky | ? | ? | |
| | la | ? | ? | |
| | lb | ? | ? | |
| | lo | ? | ? | |
| | lt | ? | ? | |
| | lv | ? | ? | |
| | mg | ? | ? | |
| | mi | ? | ? | |
| | mk | ? | ? | |
| | ml | ? | ? | |
| | mn | ? | ? | |
| | mr | ? | ? | |
| | ms | ? | ? | |
| | mt | ? | ? | |
| | my | ? | ? | |
| | ne | ? | ? | |
| | nl | ? | ? | |
| | no | ? | ? | |
| | ny | ? | ? | |
| | pa | ? | ? | |
| | pl | ? | ? | |
| | ps | ? | ? | |
| | pt | ? | ? | |
| | ro | ? | ? | |
| | ru | ? | ? | |
| | ru-Latn | ? | ? | |
| | sd | ? | ? | |
| | si | ? | ? | |
| | sk | ? | ? | |
| | sl | ? | ? | |
| | sm | ? | ? | |
| | sn | ? | ? | |
| | so | ? | ? | |
| | sq | ? | ? | |
| | sr | ? | ? | |
| | st | ? | ? | |
| | su | ? | ? | |
| | sv | ? | ? | |
| | sw | ? | ? | |
| | ta | ? | ? | |
| | te | ? | ? | |
| | tg | ? | ? | |
| | th | ? | ? | |
| | tr | ? | ? | |
| | uk | ? | ? | |
| | und | ? | ? | |
| | ur | ? | ? | |
| | uz | ? | ? | |
| | vi | ? | ? | |
| | xh | ? | ? | |
| | yi | ? | ? | |
| | yo | ? | ? | |
| | zh | ? | ? | |
| | zh-Latn | ? | ? | |
| | zu | ? | ? | |
|
|
| ## Dataset Creation |
|
|
| ### Curation Rationale |
|
|
| [More Information Needed] |
|
|
| ### Source Data |
|
|
| #### Initial Data Collection and Normalization |
|
|
| [More Information Needed] |
|
|
| #### Who are the source language producers? |
|
|
| [More Information Needed] |
|
|
| ### Annotations |
|
|
| #### Annotation process |
|
|
| [More Information Needed] |
|
|
| #### Who are the annotators? |
|
|
| [More Information Needed] |
|
|
| ### Personal and Sensitive Information |
|
|
| [More Information Needed] |
|
|
| ## Considerations for Using the Data |
|
|
| ### Social Impact of Dataset |
|
|
| [More Information Needed] |
|
|
| ### Discussion of Biases |
|
|
| [More Information Needed] |
|
|
| ### Other Known Limitations |
|
|
| [More Information Needed] |
|
|
| ## Additional Information |
|
|
| ### Dataset Curators |
|
|
| [More Information Needed] |
|
|
| ### Licensing Information |
|
|
| AllenAI are releasing this dataset under the terms of ODC-BY. By using this, you are also bound by the Common Crawl terms of use in respect of the content contained in the dataset. |
|
|
| ### Citation Information |
|
|
| ``` |
| @article{2019t5, |
| author = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu}, |
| title = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer}, |
| journal = {arXiv e-prints}, |
| year = {2019}, |
| archivePrefix = {arXiv}, |
| eprint = {1910.10683}, |
| } |
| ``` |
|
|
| ### Contributions |
|
|
| Thanks to [@dirkgr](https://github.com/dirkgr) and [@lhoestq](https://github.com/lhoestq) for adding this dataset. |
|
|
