As AI models grow larger and inference workloads scale, larger models, longer context windows and growing KV caches are driving demands on memory resources. Traditional CPU-centric networked JBOF (Just Bunch of Flash) can’t keep pace with the throughput, latency, and efficiency requirements of these modern AI clusters.

DPU (data processing unit) -based storage is a compelling alternative to traditional storage, acting as the broker between the network and SSD for remote storage. By offloading storage, networking and security processing from the host CPU onto a dedicated DPU, AI infrastructure can move data closer to compute, reduce latency, and free up valuable CPU cycles for AI workloads.

To address this need, Marvell offers the OCTEON DPU family, purpose-built for hyperscale cloud workloads and data center applications, extending its use specifically into network storage acceleration for AI environments.

KV Cache Hitting the Memory Wall

Managing KV cache efficiently is a critical part of AI inference. One of the biggest memory and storage challenges AI infrastructure is facing today is KV cache exponential growth. With every token, KV cache grows, and it hogs scarce HBM. When KV cache faces the memory wall, it can stall GPUs.

During multi-turn agentic AI sessions, especially when these sessions are concurrent and running on bare metal GPUs as multiple instances, KV cache grows faster than the HBM can hold it and eventually may run into overflows of that memory.

Once the HBM overflows with KV cache, there are two options to handle it. The first approach is to throw away the context and then recompute the KV cache later, which is a very heavy process. This involves recomputing and pre-filling from scratch, and this spikes the time to first token (TTFT).

Also compounding this particular issue is the fact that, let's say, a session run starts on a particular instance on GPU in a rack, and if it resumes on a different GPU in a different rack, then all the KV cache that was computed is lost, so it needs to be recomputed. Of course, the scheduler is going to make all the efforts possible to resume the session in the same instance or GPU, but it may not happen due to load issues the way routing works. Recomputing is a problem because whatever was computed may still be in the HBM, or in the origin GPU, but it cannot be used because the resumption happens on a different GPU. The second option is to offload the KV cache to the next tier of memory, from HBM or maybe even to external network storage.

Tackling KV Cache with Memory and Storage Tiering

The industry today revolves around Vera Rubin-based SuperPOD architecture, a tiered system for memory and storage. HBM is tier one, which is local and the fastest but very scarce. CPU DRAM is tier two, and optionally flash memory is tier 3, but, in practice, even these three memory tires put together is not enough.

Source: NVIDIA. Diagram of KV cache memory hierarchy.

Marvell OCTEON Addressing the Need

The way the industry is solving this problem is to add a pooled memory tier, aka tier 3.5, a network flash pool. Without 3.5, one must go to the network cold storage, which is an additional hop outside the pod, or the super pod, with higher performance impact to TTFT.

OCTEON-driven JBOF offers 3.5 memory tier, a pooled memory tier where KV cache can offload and reuse across multiple tracks. The OCTEON-based memory pool shows the benefits of having tier 3.5, the benefits of being able to reuse KV cache across multiple rack and across multiple GPUs.

Marvell OCTEON-based Storage demonstration

Marvell OCTEON based storage demonstrates acceleration of inference use. It shows a reduction in HBM held by KV cache and an overall 5X reduction in TTFT which could save substantial cost per token.

By moving data closer to compute and reducing latency, Marvell is helping customers improve utilization, boost token efficiency, and scale AI infrastructure without compromising performance, power, or cost—positioning DPU-based storage as a foundational layer for the next generation of AI data centers.

# # #

This blog contains forward-looking statements within the meaning of the federal securities laws that involve risks and uncertainties. Forward-looking statements include, without limitation, any statement that may predict, forecast, indicate or imply future events or achievements. Actual events or results may differ materially from those contemplated in this blog. Forward-looking statements are only predictions and are subject to risks, uncertainties and assumptions that are difficult to predict, including those described in the “Risk Factors” section of our Annual Reports on Form 10-K, Quarterly Reports on Form 10-Q and other documents filed by us from time to time with the SEC. Forward-looking statements speak only as of the date they are made. Readers are cautioned not to put undue reliance on forward-looking statements, and no person assumes any obligation to update or revise any such forward-looking statements, whether as a result of new information, future events or otherwise.