ByteDance researchers identify cause of inconsistent long-context retrieval in DeepSeek models · TechNode

ByteDance researchers identify cause of inconsistent long-context retrieval in DeepSeek models · TechNode

Researchers from ByteDance’s Seed team have identified a mechanism behind inconsistent long-context retrieval in DeepSeek models. Their study found that chunked KV-cache compression makes retrieval accuracy sensitive to where information falls within a compression window, producing differences of up to 40 percentage points across positions. The technique reduces memory and attention costs by compressing groups […]

ByteDance Seed Paper Finds a 'Phase Sensitivity' Blind Spot in Chunked KV-Cache Compression

ByteDance Seed researchers have identified a periodic weak spot in language models that compress their key-value (KV) cache in fixed-size chunks, a design DeepSeek-V4 uses to cut the memory and attention cost of long contexts. In a paper titled "Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression," posted to arXiv on September 28, the team reports that the same information can be easy to retrieve at one position and hard at …