Reports say that Chinese large-model companies are strengthening data governance, and Tencent Hunyuan has rebuilt its training infrastructure

Article

Even trillion parameters cannot save dirty data: domestic large-model developers have begun redoing pretraining

Dongcha Beating AI news flash: Leifeng.com reports that over the past months or more, multiple domestic large-model developers have begun redoing pretraining, with the problems all pointing to data.

One developer once spent nearly 1 years training a trillion-parameter model, but its actual performance was surpassed by a small model with only about 1%parameters. The team eventually discovered that the problem lay in the training data. Web junk, duplicate corpora, and low-quality annotations had entered the training set; no matter how large the model, it could not handle them. This case came from an anonymous source, and the specific company has not been disclosed.

Tencent Hunyuan has publicly disclosed that it redid one round. Starting this year,February the team rebuilt its pretraining and reinforcement-learning infrastructure. Media reports had previously revealed problems in the old Hunyuan, including benchmark data entering the training set and chaotic annotation rules. The new team reset the data standards and cleaned the original corpora, while Hy3 also listed data quality and diversity as major areas for improvement.

Alibaba and Baidu are also stepping up data governance. Qwen3 used the Qwen2.5 series models to clean documents and added large amounts of synthetic mathematics and code data. Wenxin 4.5 then added deduplication, low-quality filtering, a data map, and human review. Neither company has publicly said that it had previously reworked its models because of dirty data, but the new-generation models are both devoting more effort to data processing.

Original link https://m.theblockbeats.info/flash/368576

Key points

Over the past more than months, several domestic large-model companies have begun redoing pretraining, with the problem consistently pointing to data.
One company trained a trillion-parameter model for nearly 1 a year, but its performance was surpassed by a smaller model whose parameter count was approximately 1% of its own.
The team found that web junk, duplicate corpora, and low-quality annotations had entered the training set.
Tencent Hunyuan has, since this year,February begun rebuilding its pre-training and reinforcement learning infrastructure, while redefining data standards and cleaning its existing corpus.
Qwen3 used Qwen2.5 series models to clean documents and supplemented them with synthetic mathematics and code data.
Wenxin 4.5 added deduplication, low-quality-data filtering, a data map, and manual review.

AI-assisted interpretation

The following is analysis, separate from reported facts. Verify important claims independently.

Training data are like textbooks that models use to learn. If the textbooks contain noise, duplicate content, or incorrect labels, training performance may be poor even if the model has many parameters. Therefore, vendors are reorganizing their data and retraining their models.

Why it matters to readers

This indicates that improving model capabilities does not depend solely on parameter scale; the quality and diversity of training data are also receiving attention.

When following progress in large language models, in addition to parameter count, pay attention to data cleaning, deduplication, annotation, and human review.

Risks and unknowns

The key turnaround case comes from an anonymous source; the specific company has not been disclosed.
Alibaba and Baidu have not publicly stated that they ever had to rework data because of dirty data.
The original text does not clearly specify the exact timing and scope of rework by multiple manufacturers.

Reports say that Chinese large-model companies are strengthening data governance, and Tencent Hunyuan has rebuilt its training infrastructure

Article

AI-assisted interpretation

Why it matters to readers

Risks and unknowns

Related Developments

Related concepts

Keep following

Security Center