Contents
If you need a bigger list for any other purpose, please contact the originator of the list. AllenAI are releasing this dataset under the umarkets review terms of ODC-BY. By using this, you are also bound by the Common Crawl terms of use in respect of the content contained in the dataset.
The total size of compressed .json.gz files is roughly halved after the procedure. With more than 151GB of cleaned Dutch text and more than 23B estimated words, this is by far the largest available cleaned corpus for the Dutch language. The second largest dataset available is OSCAR, which is only 39GB in size for its deduplicated variant, and contains vulgarity.
Dataset Creation
To build mC4, the original authors used CLD3 to identify over 100 languages. For Dutch, the whole corpus of scraped text was divided in 1032 jsonl files, 1024 for training following the naming style c4-nl-cleaned.tfrecord-0XXXX-of-01024.json.gz cmc markets review and 4 for validation following the naming style c4-nl-cleaned.tfrecord-0000X-of-00004.json.gz. The full set of pre-processed files takes roughly 208GB of disk space to download with Git LFS.
The Dutch portion of mC4 was cleaned in a similar fashion as the English cleaned C4 version. Please contact the moderators of this subreddit if you have any questions or concerns.
Datasets:
Using this corpus for training language models with adequate computational resources will allow researchers to reach parity with the performances observed for the English language. This can in turn have important repercussions for the development of commercial language technology applications for the Dutch language. Despite the cleaning procedure aimed at removing vulgarity and profanity, it must be considered that model trained motivewave review on this scraped corpus will inevitably reflect biases present in blog articles and comments on the Internet. This makes the corpus especially interesting in the context of studying data biases and how to limit their impacts. This is a word list of 4621 most used Dutch words based on contents of The list has only been cleaned to an extent and it is possible that you might find English entries – as it is based on movie subtitles.
Discussion about this post