State Space Models Explained: From S4 to Mamba — The Quiet Revolution in Deep Learning

State space models are a class of sequence modeling architectures that process data through a continuous hidden state updated incrementally with each new input, offering linear scaling instead of the quadratic cost of Transformer self-attention.

Key Takeaways:

• S4 introduced structured state matrices and HiPPO-based memory for long-range modeling
• Mamba added input-dependent parameters for selective, hardware-aware sequence processing
• SSMs achieve linear memory and compute growth versus Transformers' quadratic scaling
• Production hybrids like Jamba combine Transformer attention with Mamba state spaces
• RWKV offers a time-decay RNN that trains like GPT but runs like a streaming model

Why Transformers Need Alternatives

The self-attention mechanism in Transformers requires computing pairwise interactions between every token, producing quadratic memory and computation costs O(n²). The Mamba paper highlighted that this quadratic scaling becomes a bottleneck when processing sequences longer than several thousand tokens, limiting practical context windows in real deployments.

Industry pressure to support longer contexts has grown substantially. The Sequence reported that demand for tens-of-thousands-token context has pushed teams to seek alternatives maintaining performance without the quadratic memory wall that attention creates.

What Is an SSM?

A state space model maintains a continuous hidden state that updates with each input token through a discretized recurrence relation derived from a linear differential equation. The model learns structured state matrices capturing long-range dependencies while using constant memory per token, avoiding the quadratic blowup of attention. This frames the system as a dynamical system evolving its internal representation as information streams in.

By constraining state matrices to special algebraic structure—such as being low-rank—the approach remains expressive while keeping parameter counts manageable and enabling parallelizable training through convolutional formulations. The discretization converts the continuous differential equation into a discrete recurrence suitable for efficient training on modern hardware.

S4: The Long-Range Foundation

S4 introduced HiPPO-based state initialization, projecting past inputs onto orthogonal polynomial bases to capture long-range dependencies. The method maintains a compressed summary of earlier tokens that can be queried later, allowing the model to remember information across very long sequences without explicit attention over all past tokens.

S4 employed DPLR (Diagonal Plus Low-Rank) parameterization of the state matrix, reducing learnable parameters while preserving expressive power. This structured approach combined with a convolutional recurrence enabled parallel training through a scanning algorithm, bridging recurrent efficiency with convolutional speed described in the foundational S4 work.

Mamba: Selective State Spaces

Mamba introduced input-dependent parameters to the state space model, making the transformation selective based on the current input token. The model dynamically decides what information to retain, forget, or pass through at each step—a mechanism akin to gating integrated directly into the state transition.

The Mamba paper stated that selective state spaces achieved approximately 5× inference speed improvements compared to equivalent Transformer models on certain sequence tasks, thanks to the linear memory and compute profile. The hardware-aware parallel scan implementation allowed Mamba to match or exceed Transformer training throughput while maintaining efficient autoregressive inference without caching large key-value states.

Mamba-2 and the Future

Mamba-2 refined the architecture through State Space Duality (SSD), simplifying the selective mechanism and enabling more efficient computation. The reformulation made it easier to implement chunked computations and faster variants retaining the core benefits of selective state spaces.

Subsequent research has focused on practical engineering improvements: more efficient data loading, better GPU kernel fusion, and integration with deep learning frameworks. The Mamba-2 line demonstrates that selective SSMs are a stable direction with ongoing optimization work from multiple teams.

RWKV: A Transformer-Friendly RNN

RWKV presents a time-decay recurrent model training with standard Transformer objectives but running like a pure RNN during inference. The model assigns exponentially decaying weights to past states, maintaining long-range memory without attention or KV cache storage.

The RWKV paper documents that the model uses the same language modeling objectives as Transformers while benefiting from mature tokenizer choices, pretraining techniques, and evaluation benchmarks. At inference time, it streams tokens without a growing key-value cache, making it memory-efficient for long-sequence generation tasks.

SSMs vs Transformers

Training throughput favors SSMs on long sequences because their linear scaling allows more tokens per second than Transformers when sequence length dominates memory and compute budgets. Inference similarly benefits: SSMs do not require a KV cache that grows linearly with sequence length, so peak memory usage remains bounded during long output generation.

On certain long-range reasoning benchmarks, SSMs have demonstrated competitive performance with Transformers while using substantially less memory. However, Transformers maintain advantages on many standard language modeling tasks where self-attention induction biases are favorable. The tradeoff depends on the specific problem, model size, and sequence length regime as discussed in comparative analyses of hybrid and pure architectures.

Where SSMs Are Used Practically

The Jamba paper describes a production hybrid model combining attention and Mamba blocks, retaining strengths of attention on local dependencies while using SSMs for long-range context. This mixed architecture claims higher throughput than pure Transformer baselines at scale while supporting longer context windows.

The open-weight ecosystem around SSMs includes public repositories and pretrained checkpoints allowing experimentation without building from scratch. Framework integration efforts have made it possible to plug SSM layers into existing model zoos, lowering adoption barriers for teams testing SSM-based components.

A 2026 survey found that multiple production systems began evaluating SSMs for real-time text generation and code completion where latency and memory footprint are critical constraints. While adoption remains selective, practical interest has moved beyond academic curiosity into engineering consideration.

When to Use SSMs

Developers should consider SSMs when sequences exceed several thousand tokens and memory-constrained inference is a priority. The linear memory profile suits document-level generation, long-context chat, and time-series forecasting where input length varies widely.

However, The Sequence noted that SSMs remain weaker than strong Transformer baselines on few-shot and in-context learning tasks as of the 2024-2025 research period. Inference speed gains vary by sequence length, hardware, implementation quality, and quantization level, so concrete benchmarks should be run rather than relying on headline claims. SSMs are a valuable tool in the architecture toolbox but not a universal replacement for attention.

For teams choosing a sequence modeling architecture, prototype both Transformer and SSM variants on your specific workload. Your sequence length characteristics, inference memory budget, and the inductive biases your task needs will determine whether the linear scalability of state spaces gives enough advantage to outweigh attention's strengths on shorter, pattern-rich sequences.

Conclusion

State space models have emerged as an alternative to Transformer attention, offering linear scaling and strong performance on long-sequence tasks. From S4's HiPPO foundations through Mamba's selective mechanisms to hybrid production models like Jamba, the field is maturing with a practical focus on real-world deployment constraints.

Frequently Asked Questions

Q: How do SSMs compare to attention in terms of training speed?
A: SSMs achieve faster training throughput on long sequences because their linear scaling avoids the quadratic attention matrix computation. The parallel scan algorithm in S4 and Mamba enables efficient GPU training, though Transformers may still be competitive on shorter sequences due to optimized attention kernels.

Q: Do SSMs require a KV cache during inference?
A: No, pure SSMs do not require a KV cache because the hidden state is updated incrementally and stored as a fixed-size vector. Inference memory usage remains constant regardless of sequence length, a significant advantage over Transformers during long output generation.

Q: Can SSMs handle in-context learning like Transformers?
A: SSMs show promising in-context learning capabilities, particularly with selective mechanisms like Mamba's input-dependent parameters. However, they generally lag behind strong Transformers on complex few-shot reasoning tasks, making hybrid architectures combining both approaches an active research area.

References

• Mamba: Linear-time sequence modeling with selective state spaces—introduces selective state spaces and hardware-aware parallel scan.
• Jamba: A Transformer-State Space Hybrid—describes a production hybrid model combining attention and Mamba blocks.
• RWKV: An RNN with GPT-level Performance—presents the time-decay RNN that trains like a Transformer.
• A 2026 survey of Transformer alternatives covering Mamba, RWKV, and practical tradeoffs—provides overview context for the SSM landscape.
• The Sequence: How state space models are rethinking sequence modeling—offers framing on long-range reasoning from S4 through later developments.