CulturaX uonlp
CulturaX Cleaned, Enormous, and Public: The Multilingual Fuel to Democratize Large Language Models for 167 Languages Dataset Summary We present CulturaX, a substantial multilingual dataset with 6.3 trillion tokens in 167 languages, tailored for large language model (LLM) development. Our dataset undergoes meticulous cleaning and deduplication through a rigorous pipeline of multiple stages to accomplish the best quality for model training, including language… See the full description on the dataset page: https://huggingface.co/datasets/uonlp/CulturaX.
- 类型
- dataset
- 语言
- af
- 下载量
- 12,768
- 点赞
- 665
- 访问
- gated
- 文件
- 0