SemiAnalysis Says AI Inference Bottlenecks Are Shifting Toward Memory Bandwidth and Scheduling

Article

SemiAnalysis Breaks Down AI Inference: Memory Bandwidth Matters More Than Capacity, Making the Scheduling Layer Critical

Dongcha Beating AI News Flash: Research firm SemiAnalysis released a report providing a systematic breakdown of the underlying architecture of large-model inference services. The report argues that as MoE models become mainstream, AI inference has evolved from a single computational task into a complex pipeline comprising Prefill, Midfill, Decode Attention, and Decode Experts, with clear differences in the demands for compute, memory bandwidth, and networking across stages.

In most inference scenarios, memory bandwidth offers greater economic value than capacity. High-bandwidth memory can improve Token generation efficiency, while idle HBM only adds cost. SemiAnalysis expects that in 2027, a single pipeline stage may require approximately 400 to 500GB of local high-speed memory, but completed KV cache processing should be promptly migrated to CPU DRAM and lower-cost network storage to avoid occupying scarce HBM resources.

The scheduling layer will become a critical component of AI inference infrastructure. Prefill and Midfill processing times are relatively predictable, while Decode durations fluctuate considerably, making task backlogs and “latency feedback oscillations” likely. Therefore, stable buffering mechanisms, scheduling across multiple time scales, and tiered KV cache management will directly affect the throughput efficiency of inference clusters.

In terms of architectural choices, the report compares aggregated and disaggregated approaches. The former consolidates Prefill and Decode on the same device, reducing cross-node KV cache transfers, but requires hardware to provide both high compute density and high memory bandwidth. The latter assigns dedicated nodes to different stages, enabling hardware optimization but increasing data movement and network overhead. SemiAnalysis believes that the trade-off between the two architectures will ultimately depend on whether an “all-purpose” accelerator combining high compute performance with high memory bandwidth emerges in the future.

The report also uses a simulator to predict Kimi K3’s performance on NVIDIA B200, B300, and GB200 systems. The results show that GB200 has an advantage in some low-latency scenarios, while the peak HBM residency of a single GPU in most frontier configurations is below 80GB, further highlighting the importance of memory bandwidth and storage orchestration.

Original link https://m.theblockbeats.info/flash/368378

Key points

SemiAnalysis released a report breaking down the underlying architecture of large-model inference services.
The report says that MoE model inference includes stages such as Prefill, Midfill, Decode Attention, and Decode Experts.
The report argues that memory bandwidth offers greater economic value than capacity in most inference scenarios.
SemiAnalysis expects that in 2027, a single pipeline stage may require approximately 400 to 500GB of local high-speed memory.
The report says that processed KV cache should be migrated to CPU DRAM and lower-cost network storage.
The report compares aggregated and disaggregated architectures and says their trade-off depends on whether an accelerator combining high compute performance with high memory bandwidth can emerge in the future.
Simulation results show that GB200 has an advantage in some low-latency scenarios, while the peak HBM residency of a single GPU in most frontier configurations is below 80GB.

AI-assisted interpretation

The following is analysis, separate from reported facts. Verify important claims independently.

Large-model inference is not a single step but a pipeline made up of multiple stages with different requirements. The report emphasizes assigning high-bandwidth memory to work that requires faster access, moving processed KV cache to lower-cost storage, and reducing task backlogs through buffering, scheduling, and tiered management.

Why it matters to readers

The report argues that memory bandwidth, data movement, and scheduling methods directly affect the throughput and cost of inference clusters, while hardware selection requires trade-offs among compute density, memory bandwidth, and network overhead.

Beginners can think of it this way: running an AI model depends not only on how much memory it has, but also on how quickly data can be read and written and whether different tasks can be queued and moved efficiently.

Risks and unknowns

The estimate of approximately 400 to 500GB in 2027 is a SemiAnalysis forecast, and it cannot yet be confirmed from the original source whether it will materialize.
Kimi K3’s performance on different systems comes from simulator predictions and is not equivalent to results from actual deployment.
The original source does not provide the full report, simulation parameters, or testing methodology, so the relevant conclusions cannot be independently verified.
The specific scope of “most frontier configurations” is not explained in the original source.

SemiAnalysis Says AI Inference Bottlenecks Are Shifting Toward Memory Bandwidth and Scheduling

Article

AI-assisted interpretation

Why it matters to readers

Risks and unknowns

Related Developments

Related concepts

Keep following

Security Center