ByteDance Seed Intern Exposes DeepSeek-V4 Flaw: Long-Context Retrieval Hits 'Periodic Blind Spots'

Researchers from ByteDance's Seed team, together with researchers from Princeton University, Stanford University, and UC Berkeley, conducted systematic tests on DeepSeek-V4 and V4.1 and found a periodic technical defect in its long-context compression mechanism. The related paper is titled "Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression."

The clue came from an anomalous code completion result: in an FP8 inference kernel code completion experiment, the standard answer was 8, but the model frequently output 32; the researchers did not change the core code, only the number of decorative equals signs in the preceding comment, and the answer periodically flipped between 8 and 32, repeating every 4 tokens. After replacing four sets of filler text, 60 of 64 comparisons conformed to this periodic pattern; V3.1-Base, which does not use chunked KV compression, showed no similar periodic flipping.

The Seed team named this phenomenon "Phase Sensitivity," meaning that the remainder (phase) of a token's absolute position divided by the compression stride affects whether information can be accurately retrieved.

DeepSeek-V4 alternates between two compressed attention mechanisms in its architecture: CSA uses an overlapping sliding window of length 8 and stride 4, cutting the KV data volume by three quarters, and then uses an Indexer to select key blocks for sparse computation; HCA compresses more aggressively, merging every 128 tokens into one entry with no overlap.

In a native Transformer, RoPE's dot product computes only relative distance, thus preserving translational symmetry. But CSA packs once every 4 tokens forward, giving each token an additional implicit physical attribute, "phase," thereby breaking this symmetry.

The technical essence of this defect is that the information is simply not written into the KV Cache at the physical level: when the model looks back to retrieve it, this memory does not exist in the cache pool, and the model can only piece it together based on autoregressive local statistical probabilities. This also explains why position groups separated by 4 tokens show similar performance across V4 versions, almost exactly matching CSA's compression stride—it is not simply "forgetting" information, but rather that information at certain positions is more easily damaged during the process from compressed storage to retrieval.

In the 128K long-context "needle in a haystack" experiment, the DeepSeek-V4-Flash base model's worst position group achieved only 29.69% retrieval accuracy, while the best position group reached 69.92%, a difference of 40.23 percentage points.

Post-training brought a clear narrowing: for the post-trained V4-Flash-0731, the gap between the best and worst positions shrank to 19.14 percentage points; the V4-Pro base and post-trained versions also had gaps of 34.77 and 14.84 percentage points, respectively.

This set of data shows that post-training can alleviate but cannot eliminate the systematic differences caused by position.

Mainstream long-context evaluations (such as NIAH or RULER) tend to scatter target information randomly; under uniform sampling, full scores in strong slots and zero scores in weak slots cancel each other out, thereby masking the performance paralysis at fixed positions.

A control experiment trained from scratch based on the Qwen3-0.6B architecture confirmed this: full attention had an average accuracy of 61.1%, and 8x compressed attention had 59.4%, a difference of only 1.7 percentage points; but full attention's worst position still had 58.4%, while the compressed model's worst position plunged to 9.9%, with the gap surging to 71.1 percentage points.

Average scores were almost on par, yet worst-case performance differed by nearly 50 percentage points—this is precisely the structural blind spot of "composite average score" evaluation.

DeepSeek had already adjusted the large-stride compression mechanism in V4.1-Flash, released as early as September 10, while the paper in which Seed revealed the defect was not submitted to arXiv until September 28. The source says DeepSeek appears to have recognized the related hidden risk in earlier iterations and begun improving it, but this judgment is an inference based on public information.

V4.1's improvements include reducing the compression stride from 4 to 2 (CSA2) and introducing the Engram module to share the attention computation burden, but the essence of fixed-stride chunked compression remains unchanged.

From the evaluation results, V4.1-Flash greatly narrowed the gap between the best and worst phases from 40% to 6.1 percentage points, but the periodicity did not disappear; instead it degenerated into a period of 2: even positions have about 95% accuracy, and odd positions about 89%–90%. Whatever the stride is, that is the period—halving the stride can alleviate the pain, but as long as it remains fixed-stride chunked compression, the essence of phase sensitivity has not been completely eliminated.

The above experimental data all come from the paper authors' own tests and lack third-party independent replication and verification; the source also does not provide DeepSeek's official confirmation of or response to the defect.

Whether DeepSeek had independently recognized this phase defect before releasing V4.1, or whether the improvement was coincidental, the source only says it is "judged based on public information," with no definitive conclusion. The actual degree of impact of phase sensitivity in real user scenarios remains unclear, and the source provides no online performance data; whether this defect affects models beyond the DeepSeek-V4 series that also use chunked KV compression is also not addressed by the source.

The paper is 65 pages long, with only 10 pages of main text and the remaining 52 pages as appendices, with the core evidence concentrated in the appendices; the ByteDance Seed team mainly follows a closed-source technical route, and whether its testing methods and data are fully public and reproducible is not explained by the source.