ByteDance's Seed team discovered DeepSeek-V4 periodic retrieval fluctuations

Article

ByteDance uncovers DeepSeek-V4 a hidden blind spot: the code stays the same, but the answer periodically goes wrong

According to a news flash from Beating AI, ByteDance's Seed team found that DeepSeek-V4 exhibits a systematic weakness in long-text processing. Researchers asked the model to complete a piece of code. The code itself remained unchanged; only a preceding section of irrelevant comments was extended, yet the model's answer switched back and forth between correct and incorrect.V4-Flash The base version repeats a cycle every 4 Token, while V4.1-Flash shows a similar phenomenon, with the cycle becoming 2.

The issue is related to Token and its memory-saving "chunked DeepSeek cache compression." The model merges and stores the information from several consecutive KV, reducing the cost of processing long texts. However, this compression causes information at different positions to be retained to different degrees, making content at certain positions more difficult to accurately retrieve later. ByteDance calls this periodic difference in retrieval "phase sensitivity."

In retrieval tests covering approximately Token, 128000 Token the V4-Flash base version's accuracy at different positions differed by as much as 40.2 percentage points. After post-training, the gap narrowed to 19.1 percentage points;V4.1-Flash narrowed it further to 6.1 percentage points, but periodic fluctuations remained. ByteDance also trained multiple control models from scratch and found that the fluctuation period matched the compression stride. Control models that did not use chunked compression did not exhibit the same pattern.

Original link https://m.theblockbeats.info/flash/370939

Key points

ByteDance's Seed team said that DeepSeek-V4 has a systematic weakness in long-text processing.
When the code remains unchanged and only preceding irrelevant comments are extended, the model's answer switches between correct and incorrect.
V4-Flash The base version repeats every 4 Token, while V4.1-Flash has a cycle of 2.Token。
In approximately 128000 Token tests,V4-Flash the maximum accuracy difference across positions was 40.2 percentage points; after post-training, the gap was 19.1 percentage points, and V4.1-Flash was 6.1 percentage points.
The report links the phenomenon to chunked KV cache compression and says that control models without chunked compression did not exhibit the same pattern.

AI-assisted interpretation

The following is analysis, separate from reported facts. Verify important claims independently.

The model merges and stores the information from several consecutive Token to reduce the cost of long-text processing; the report suggests that this may cause information at different positions to be retained differently, thereby affecting subsequent retrieval.

Why it matters to readers

The same code may produce a different answer after irrelevant content is added before it, indicating a position-related stability issue in long-text retrieval results.

When using long texts AI for code analysis or information retrieval, answers may be affected by the position of irrelevant content.

Risks and unknowns

The original text did not provide the original paper or information from independent reviews.
The original text did not state whether the phenomenon applies to other tasks, context lengths, or models.
V4.1-Flash Periodic fluctuations still exist, but the original text did not state whether subsequent versions will eliminate the phenomenon.

ByteDance's Seed team discovered DeepSeek-V4 periodic retrieval fluctuations

Article

AI-assisted interpretation

Why it matters to readers

Risks and unknowns

Related Developments

Related concepts

Keep following

Security Center