Spiking neural networks compute with events in time. A neuron stays silent until its state crosses a threshold, emits a spike, then resets or adapts according to the neuron model. That event-driven shape is the reason SNNs keep attracting attention: sparse spikes map well to low-power neuromorphic hardware, and spike timing carries temporal information directly (Surrogate-gradient review). The same shape also explains why SNNs remain hard to train and hard to deploy with mainstream deep-learning tools.

Spike Computation

A conventional neural network passes continuous activations through dense matrix operations. An SNN passes discrete spike events through time. The network state evolves over timesteps, and information can live in spike rate, spike timing, or the order of events (Surrogate-gradient review).

That makes SNNs natural for streaming sensors. If nothing changes, little work happens. If an event camera pixel fires only when brightness changes, an SNN downstream can process the event instead of recomputing a whole image frame. Computation follows change.

The Training Problem

The hard part is the spike itself. A threshold spike is non-differentiable, so ordinary backpropagation has no clean derivative to follow. Competitive SNN training often uses surrogate gradients: the forward pass keeps discrete spikes, while the backward pass substitutes a smooth approximate derivative so gradient descent can update the weights (Surrogate-gradient review).

That workaround is powerful, but it undercuts the simple story that SNNs learn without backpropagation. Local learning rules such as spike-timing-dependent plasticity are part of the neuromorphic toolkit; the classic result is timing-specific, with presynaptic-before-postsynaptic spikes strengthening a synapse and the reverse order weakening it (Bi and Poo 1998). Many competitive supervised SNN results still rely on gradient-based training. The field has to choose between biological elegance, task performance, and engineering tractability.

Neuromorphic Hardware

SNNs make the most sense when the hardware also thinks in events. IBM's TrueNorth is the historical anchor: the chip integrated 1 million programmable neurons and 256 million synapses, and the paper reports low-power real-time operation at about 65 mW (TrueNorth paper). Intel's Loihi added programmable synaptic learning rules, dendritic compartments, and axonal delays, making on-chip learning and timing dynamics more flexible (Loihi paper).

SpiNNaker2 attacks the same problem from a many-core systems angle, with processing elements designed for event-driven neural simulation and adaptive power management (SpiNNaker2 PE paper). Intel's Loihi 2 brief claims up to 10x improvement over Loihi 1 in selected capabilities and points to Lava as the software stack for neuromorphic development (Loihi 2 technology brief).

Platform	Useful fact	Architectural lesson
TrueNorth	1M neurons, 256M synapses, about 65 mW reported	Event-driven inference can be power-efficient
Loihi	Programmable learning rules, compartments, delays	SNN hardware needs timing and plasticity features, not just matrix units
SpiNNaker2	Many-core event-driven neural simulation with adaptive power features	Neuromorphic scaling is a systems problem
Loihi 2	Improved Loihi family hardware plus Lava software stack	Tooling matters as much as chip design

Why GPUs Are Awkward

Modern deep learning hardware is optimized for dense linear algebra. Transformers and CNNs map well to GPUs because the workload is dominated by large matrix multiplications. SNNs want sparse, asynchronous event handling. Running an SNN on a GPU often means simulating timestep dynamics in a way that leaves the hardware doing work it was not designed to do (Loihi paper, Surrogate-gradient review).

SNNs face a co-design problem. Algorithms, chips, and software have to mature together. Transformers won because the math, hardware, libraries, and benchmarks aligned. SNNs still have gaps in every part of that chain.

Where They Fit

SNNs make the strongest case for always-on sensing, ultra-low-power edge devices, event-camera perception, audio or radar streams, and applications where timing is part of the signal. In those settings, dense frame-by-frame processing wastes energy on non-events. An event-driven model can spend compute only when the world changes.

They are a weaker fit for general language modeling, large-scale vision pretraining, and tasks where dense GPUs already deliver huge throughput with mature software. If you need best-in-class accuracy on mainstream benchmarks, a transformer or convolutional model is the safer engineering choice today.

Failure Modes

SNN projects fail when teams import expectations from standard deep learning. Training can be less stable because the temporal dynamics add extra state. Hyperparameters tied to thresholds, leak constants, timesteps, and surrogate-gradient shape can dominate results. Tooling is fragmented compared with PyTorch and the transformer ecosystem. Hardware access is limited, so many experiments run on devices that do not expose the energy benefits SNNs were supposed to deliver.

The result is a frustrating loop: SNNs look best on hardware that few teams can use, while the hardware needs stronger workloads and software to justify broader adoption.

My Take

SNNs are not failed transformers. They optimize for a different constraint: energy and time-local event processing. That makes them less compelling as a general replacement for dense deep learning and more compelling as the first stage of a sensing stack. The near-term win is not an SNN chatbot. It is a sensor front end that filters events, detects anomalies, and wakes a larger model when something worth reasoning about happens.

Takeaways

Spiking neural networks compute with sparse temporal events, which gives them a real architectural advantage for low-power streaming perception. Their bottleneck is training and ecosystem maturity. Surrogate gradients make SNNs trainable, but they also show that practical SNNs still borrow heavily from backpropagation. Neuromorphic chips such as TrueNorth, Loihi, SpiNNaker2, and Loihi 2 show why the hardware story matters. The best use cases are event-driven and power-limited; the weakest use cases ask SNNs to compete with dense models on dense hardware.

References

• Neftci et al., "Surrogate Gradient Learning in Spiking Neural Networks"
• Davies et al., "Loihi: A Neuromorphic Manycore Processor with On-Chip Learning"
• Akopyan et al., "TrueNorth: A 65 mW 1 Million Neuron Programmable Neurosynaptic Chip"
• Mayr et al., "SpiNNaker2: A 10 Million Core Processor System for Brain Simulation and Machine Learning"
• Intel, "Loihi 2 Technology Brief"
• Bi and Poo, "Synaptic Modifications in Cultured Hippocampal Neurons"
• Eshraghian et al., "Training Spiking Neural Networks Using Lessons From Deep Learning"
• Wikimedia Commons, "The process of how action potential passes through a neuron.svg"

author: Ope tag: #ai links: [[World Model V-JEPA 2]], [[Multi-Token Prediction]], [[Small LLMs — Use Cases and Limits]], [[Full-Duplex Speech Models]]