What an RDU is

The RDU is the processor at the heart of SambaNova's systems - the hardware our platform runs on. Rather than a fixed instruction pipeline, the chip provides a grid of Pattern Compute Units (PCUs) and Pattern Memory Units (PMUs) that the compiler configures per model: operations are laid out spatially, and tensors stream through them as a pipeline - the dataflow execution model. SambaNova describes the result as data flowing from one AI operation to the next like an assembly line, with data for the next operation fetched while the current one is still running.

The current production chip, the SN40L, pairs that compute fabric with a three-tier memory system: 520 MB of on-chip SRAM for the hottest data, 64 GB of co-packaged HBM that streams model weights, and directly attached DDR memory for prompt caching and holding a catalog of models. Each SN40L socket delivers 638 BF16 teraFLOPS from 1,040 compute units.

Why it exists: the memory wall

SambaNova's engineering team frames the problem bluntly: AI inference is a data movement problem, not a compute problem. During the decode phase of inference, the hardware must move model weights from memory for every generated token - on instruction-based architectures, that memory traffic, not arithmetic, dominates latency. The RDU's dataflow design attacks this directly: operator fusion keeps intermediate results on-chip, and in SambaNova's published SN40L paper, fusing operators into large pipelines reached close to 85% of HBM bandwidth utilization while eliminating per-kernel launch overheads.

The peer-reviewed SN40L paper (MICRO 2024) reports speedups of 2x to 13x over an unfused baseline and 3.7x over a DGX H100 system on composition-of-experts inference workloads. Because the DDR tier sits directly on the chip's memory system, switching between model checkpoints takes milliseconds - SambaNova measured roughly 60-90 ms for hot-swaps that take around 800 ms on GPU-based serving stacks.

RDUs in practice

A SambaRack combines 16 RDUs into a single air-cooled 19-inch rack drawing about 10 kW typical power - standard datacenter infrastructure, no liquid cooling. Each chip can address the memory of all 16 chips in the rack, so one rack runs models far larger than the memory of a single chip. The speed this architecture delivers is measurable: 713 tokens per second on gpt-oss-120b, independently verifiable through our published benchmarks.

Sources

Learn how SambaNova's dataflow architecture changes the economics of inference - and why we built on it.

Explore the Technology