SambaNova and Supermicro Split Inference in Two — and Sell the Rack That Does It

Sixteen chips, one rack, twenty kilowatts. The SambaRack SN50 fits a standard air-cooled hall — no liquid loop, no new substation, no three-year queue for a grid connection. That is the pitch SambaNova and Supermicro put on the table on November 3. It is narrower than the headline suggests. The rack does not replace a GPU cluster. It plugs into one.

The division of labour is explicit. Nvidia GPUs read the prompt. SambaNova's SN50 chips write the answer. Prefill is compute-bound. Decode is memory-bound. Run both on the same accelerator and each phase waits for the other.

What actually changed on November 3

Supermicro will integrate, test, deliver and support the systems, initially planning for 10,000 SN50 RDUs. Shipments start with NeoClouds and sovereign clouds, then move to enterprises running AI in their own facilities. SambaNova builds the chip and tunes models for it. Supermicro owns the channel, the service contract and the customer relationship.

The 10,000 figure is a planning number. Neither company disclosed a purchase commitment, a price, or a ship date. At 16 RDUs per rack, it implies roughly 625 racks if Supermicro keeps SambaNova's density.

Why a separate decode chip exists

Decode re-reads the model weights for every token. That makes it a bandwidth problem, not a compute problem. In six frontier-model configurations SambaNova analysed, decode consumed 75% to 97% of model-inference time. The high end was DeepSeek-V3 at an 8K input and 1K output on B300 FP4 with a single active request: 97% of the time went to decode.

SN50 attacks that phase with a three-tier memory hierarchy. Each RDU carries 432 MB of on-chip SRAM, 64 GB of HBM2E and up to 2 TB of DDR5, rated at 1,600 TFLOPS in BF16 and 3,200 TFLOPS in FP8. Ten 800G Ethernet ports give it 2 TB/s of aggregate bandwidth per chip. A separate 400G RoCEv2 NIC moves the KV cache between prefill and decode pools.

The KV handoff is the weak point. Prefill builds the cache; decode needs it before the first token. Move it across racks and the transfer becomes a latency line item. SambaNova's own modelling puts per-user speed on DeepSeek-R1 at 200, 300 and 500 tokens per second as the configuration scales from 64 to 128 to 256 RDUs. Those are modelled operating points, not production measurements.

The clean external data point is smaller. At RAISE Summit 2026, a preview system with four Nvidia H200 GPUs on prefill and 16 SN50 RDUs on decode reached 763 output tokens per second on MiniMax M2.7 with a 10,000-token input, benchmarked by Artificial Analysis. That is one model, one configuration and one measurement window.

The rack math is the real argument

Performance claims are easy to discount. Power numbers are harder. One SN50 node draws 16 kW; two nodes draw 32 kW. SambaNova rates the full SambaRack at 20 kW average, air-cooled. An Nvidia GB200 NVL72 rack, by contrast, is a liquid-cooled system in the 120–140 kW class. The comparison is not like-for-like on compute, but it explains the sales motion: SN50 adds decode capacity to halls that cannot absorb another liquid-cooled megawatt.

Density scales in two directions. A single scale-up domain reaches 256 RDUs across 16 racks. Scale-out Ethernet extends the pool to 32,000 RDUs. That ceiling matters for the target customer — a NeoCloud selling tokens, not servers.

Supermicro's incentive is equally clear. It already resells Nvidia's Vera Rubin NVL72 and a Vera CPU rack for agent sandboxes. Adding an RDU rack lets it quote a mixed configuration instead of losing the deal to a rival integrator.

Three ecosystems converged on the same split

AWS pairs Trainium3 for prefill with Cerebras CS-3 for decode and claims about 5× more high-speed token capacity in the same footprint. Nvidia positions its Groq-derived Groq 3 LPX rack as a decode co-processor for Vera Rubin, with a 35× decode-throughput-per-megawatt claim over HBM GPUs. Intel's Xeon 6 heterogeneous design sits alongside SambaNova's SN50.

An October analysis on arXiv traced the arc: disaggregated, phase-specialised serving moved from OSDI and ISCA prototypes to the default architecture of three vendor ecosystems in about eighteen months. The same paper flags the open question. Per-step reliability compounds across long agent trajectories, and a 2% per-step failure rate can erase a 2× decode advantage.

Agentic AI is the next wave of inference, and it has to run everywhere.— Rodrigo Liang, co-founder and CEO, SambaNova

As we wrote in November, General Compute is already selling inference by the token instead of the GPU. The Supermicro deal pushes the same logic one layer down: the unit of sale becomes a phase-specialised rack.

What a buyer should actually price

Three numbers decide whether disaggregation pays. Tokens per kilowatt-hour under the real agentic workload. KV-transfer P99 latency between pools. Prefix-cache hit rate, because a weak hit rate inflates prefill and starves decode.

The vendor claims are not yet auditable at system level. SambaNova says SN50 delivers 5× the compute and 4× the network bandwidth of SN40, producing 10× throughput when paired with Nvidia B300 GPUs for prefill. Those are component claims. SambaNova has not published combined GPU-plus-RDU throughput, latency or power for the Supermicro systems. Supermicro has not specified GPU models, interconnects, or the GPU-to-RDU ratio per system.

The sizing risk cuts both ways. Provision too little decode and agents stall on token generation. Provision too much and the prefill pool idles between bursts. SambaNova's target customer — a NeoCloud running many tenants — has burstier traffic than a single enterprise, which makes the ratio harder to hold.

The strategic read is simpler than the spec sheet. SambaNova raised a $1 billion Series F at an $11 billion valuation in July. JPMorganChase has selected SN40 and SN50 for on-premises inference. Oracle said on October 27 that it plans to deploy SN50 in its cloud. Foxconn and Intel joined the rack-scale build in June. Supermicro adds the one thing a chip company cannot manufacture quickly: a global integration and service channel.

Will phase-specialised racks become the default by 2028?

Scenarios