Running Kimi K3 on AMD MI355X: How Open-Source AI Inference Wins on Performance Per Dollar

AMD's MI355X delivers better performance per dollar than NVIDIA's B300 for serving Kimi K3, a 2.8-trillion-parameter MoE model that pushes the limits of single-GPU inference and forces a rethink of GPU economics for open-weight models.

What Kimi K3 Is and Why It Changes the Math

Kimi K3 is a 2.8-trillion parameter Mixture-of-Experts model released by Moonshot AI. It ships with open weights but without training code or datasets, which puts it in the "open weights" camp rather than true open source. The model is designed for long-context reasoning and coding tasks, and it uses a dense routing mechanism that activates only a subset of its 2.8T parameters for each token generated.

What makes Kimi K3 notable for inference engineers is its size. Most LLM serving deployments today run models in the 7B to 70B range. Kimi K3 sits two orders of magnitude larger. Serving it efficiently requires careful attention to memory bandwidth, interconnect latency, and kernel-level optimization.

2.8T Parameters, 1M-Context Serving Reality

Kimi K3 supports a 1-million-token context window. At that length, the KV cache alone consumes hundreds of gigabytes of GPU memory. A single B200 node with 192GB of HBM isn't close to enough capacity for weights plus the KV pool. The model needs distributed tensor parallelism across multiple GPUs, and usually across multiple nodes, to run.

Memory Wall: Why a Single B200 Node Isn't Enough

A single 8-GPU B200 node falls short because the model weights alone exceed the available memory when sharded across the GPUs. The KV cache for a 1M-token context adds another massive footprint. This forces you to span across at least two nodes, which introduces cross-node communication overhead on the critical path for decode.

The GPU Serving Challenge

Running a model like Kimi K3 isn't just about fitting it in memory. It's about how efficiently you can push tokens through the system under load. The bottleneck shifts depending on whether you're looking at single-stream latency or aggregate throughput across many concurrent requests.

Speculative decoding helps here. Kimi K3 ships without built-in draft tensors, so Wafer used RadixArk's external DSpark draft model. This runs a smaller model in parallel to predict tokens, then verifies those predictions in a single forward pass through the large model. The benefit is higher throughput at the same latency for decode-bound workloads.

AMD MI355X vs NVIDIA B300 vs B200: Specs and Pricing

Wafer published benchmarks comparing Kimi K3 served on three configurations: 8x MI355X, 2x8x B200, and B300 with distributed cache parallelism. The pricing assumptions they used are $2.50/hour for MI355X, $4.25/hour for B200, and $6.00/hour for B300, based on GPU rental rates from gpus.io.

Here's how the numbers break down:

Metric	8x MI355X (TP8)	2x8 B200 (TP16)	B300 (TP8+DCP8)
Decode tok/s per stream	118	90	172
Peak aggregate throughput	952	498	1,568
Peak aggregate per GPU	119	31	196
Peak aggregate per $/GPU-hr	48	7	33
Cost per GPU-hr (assumed)	$2.50	$4.25	$6.00

The B300 wins on raw throughput. But the MI355X wins on performance per dollar, delivering 48 tokens per second per dollar compared to 33 for the B300 and 7 for the B200. That gap is dramatic.

Raw Throughput vs Economic Throughput

The B200 comparison is somewhat unfair. It spans two nodes and pays cross-node all-reduce costs on the decode critical path. The MI355X runs on a single node with 8 GPUs, which is a cleaner comparison to the B300 configuration.

Even so, the per-dollar story holds. The MI355X costs less than half what the B300 costs per hour and still delivers usable throughput. For teams running Kimi K3 at scale, that's a significant operating cost difference.

The Optimization Work: Speculative Decode and Prefill Kernel Fixes

Getting Kimi K3 to run efficiently on ROCm required two key fixes. Neither is a large change, but both were necessary to unlock the performance numbers Wafer reported.

Fixing Speculative Decode: The Missing top_k_renorm_prob

The first issue surfaced when running speculative decode with the DSpark draft model. The sglang sampling verifier expected a tensor called top_k_renorm_prob from the CUDA sampling kernel. The ROCm build only aliased a Triton top-p kernel and didn't define that tensor. Requests hitting the dense sampling path would fail with a NameError.

The fix was to add the missing top-k renormalization computation in the ROCm sampling branch. No custom kernel was needed. It was a small plumbing change, but it unblocked speculative decode entirely.

Prefill Bottleneck and the AITER MLA Head-Padding Fix

The second issue was a prefill performance problem. Kimi K3 uses multi-latent attention (MLA), and the AITER MLA fast-path kernel only loads when the head count per rank matches certain shapes: 4, 8, or multiples of 16.

With TP8, each rank had 12 heads. That didn't match the kernel's expected shapes, so the system fell back to a slow generic Triton attention implementation. Cold prefill of 172k tokens took around 4 to 7k tokens per second, which was unacceptably slow for a model with a 1M context.

The fix was to zero-pad the head dimension from 12 to 16 before calling the fast AITER MLA kernel, then extract the real 12-head outputs afterward. This boosted steady-state prefill throughput from roughly 4-7k tok/s to around 13k tok/s without affecting decode speed.

Measurable Gains from the Fixes

Together, these two fixes produced significant gains:

Single-stream decode improved by roughly 2.2x
Per-stream throughput at moderate load improved by roughly 1.7x
Peak aggregate throughput improved by about 18%
The concurrency sweet spot shifted from c24 to c64, meaning the system handles more concurrent requests efficiently

What This Means for the Open-Source AI Ecosystem

The MI355X results signal a shift in open-weight inference economics. AMD is closing the software gap with NVIDIA, and the performance-per-dollar story is becoming more compelling for teams that need to serve large models at scale.

Is the CUDA Moat Weaker Than It Looks?

CUDA has been the default for LLM serving for years. But the fixes required to run Kimi K3 on ROCm were small and localized. They didn't require rewriting kernels or building new infrastructure. The pattern suggests that as more open-weight models push toward the limits of current hardware, the ROCm gap will continue to narrow.

The fixes were framework-level changes in sglang and AITER, not custom kernel work. That's a meaningful distinction. It means the path to parity isn't about reinventing the stack, it's about filling in the gaps that already exist.

Where AMD's Software Gaps Still Show

The ROCm ecosystem still requires engineering effort to make frontier models work. The fixes above were straightforward for a team familiar with sglang and AITER, but they aren't one-click solutions. Teams evaluating AMD for production LLM serving should expect to invest in optimization work.

The missing top-k renorm was a bug. The head-padding fix was a workaround for a shape constraint. These aren't systemic failures, but they are signals that AMD is still playing catch-up on the software layer for models that push the envelope.

How This Affects Model Serving Economics

For teams running Kimi K3 or similar large MoE models, the MI355X offers a compelling alternative to NVIDIA. The per-dollar advantage is real, and it compounds at scale. A deployment running 64 GPUs on MI355X instead of B300 would see a meaningful reduction in hourly cloud costs.

The trade-off is raw throughput. The B300 still delivers about 65% more tokens per second than the MI355X. If latency-sensitive single-stream performance is the priority, the B300 remains the stronger choice. But for batch serving and high-concurrency workloads, the MI355X's throughput-per-dollar wins out.

Open Questions and Counterpoints

The benchmarks are promising, but a few questions deserve scrutiny.

Pricing Skepticism and Real-World GPU Costs

The $2.50/hour MI355X rate comes from a specific rental aggregator. Other providers list rates closer to $2.95 to $3.00 per hour, and spot or on-demand pricing can be higher. The performance-per-dollar advantage shrinks but doesn't disappear at those rates.

The B200 at $4.25/hour also warrants a footnote. Cross-node comparisons penalize the B200 configuration, and some commenters on Hacker News noted that the comparison may not reflect how teams would actually deploy a B200 cluster.

Benchmark Transparency and Reproducibility

Wafer hasn't published the full configuration or code for replication. The benchmark numbers are reported in the blog post, but independent verification isn't possible yet. That's a reasonable expectation gap for a company blog post, but it's worth noting for anyone building a procurement decision on these numbers alone.

Key Takeaways

Kimi K3 is a 2.8T open-weight MoE model that requires multi-node, multi-GPU serving. A single B200 node can't fit the weights and KV cache for 1M-token contexts.
The AMD MI355X delivers 48 tok/s per dollar for Kimi K3 serving, compared to 33 for the B300 and 7 for the B200, based on Wafer's benchmarks and GPU rental rates from gpus.io.
Two ROCm fixes unlocked the performance: adding missing top-k renorm for speculative decode and zero-padding head dimensions from 12 to 16 for the AITER MLA kernel.
The B300 still wins on raw throughput, but the MI355X offers better economics for throughput-sensitive, high-concurrency workloads.
AMD's software gap is narrowing. The fixes required were small and localized, not systemic rewrites. But teams should still plan for optimization effort.

FAQ

Can Kimi K3 run on consumer GPUs?

No. Kimi K3 is a 2.8T parameter model. Even with quantization, it requires the memory bandwidth and capacity of data-center GPUs. The minimum viable configuration runs across multiple GPUs in tensor parallelism. Consumer cards like the RTX 4090 don't have enough VRAM or the interconnect bandwidth to serve this model effectively.

What is speculative decode?

Speculative decode runs a smaller draft model in parallel to predict tokens, then verifies all those predictions in a single forward pass through the large model. This reduces the number of sequential decode steps and increases throughput at the same latency. Kimi K3 doesn't ship with built-in draft tensors, so Wafer used RadixArk's DSpark external draft model.

Is AMD ready for production LLM serving?

AMD is close, but not effortless. The ROCm fixes for Kimi K3 were small and localized, but they required framework-level engineering. Teams with ROCm expertise can replicate the work. Teams without that experience should plan for a non-trivial optimization phase before treating AMD as a drop-in replacement for CUDA deployments.

How does Kimi K3 compare to other open-weight models for inference?

Kimi K3 is on the larger end of open-weight models. Models like DeepSeek V3 and Qwen 2.5 are smaller and run more easily on single-GPU or small multi-GPU setups. Kimi K3's size makes it a stress test for inference infrastructure, which is why the MI355X results are noteworthy.

What is the AITER MLA fast path?

AITER is AMD's optimized kernel library for machine learning workloads. The MLA fast path is a highly optimized implementation of multi-latent attention, the attention mechanism used by Kimi K3. It only loads when the head count per rank matches specific shapes. The head-padding fix worked around that constraint by zero-padding from 12 to 16 heads.