Spikingformer: A Key Foundation Model for Spiking Neural Networks
Learn how Spikingformer redesigns residual connections to stay fully spike‑driven and what performance gains it shows.
Current spiking backbones like Spikformer and SEW ResNet still perform integer‑float multiplications inside their residual links, breaking the pure spike‑driven model and raising power draw on neuromorphic chips. The authors set out to build a transformer‑style SNN that removes those non‑spike calculations entirely.
They start by dissecting how residual connections inject non‑spike arithmetic: the addition of a floating‑point shortcut to a spike‑based activation forces a mixed‑precision path. To fix this, the design replaces the classic addition with an MS Residual scheme that merges the shortcut as a spike‑compatible mask, preserving binary spike events throughout the block.
The MS Residual is then fused with the self‑attention mechanism. Instead of computing attention scores with real‑valued dot products, the attention operates on spike counts and binary masks, keeping every intermediate value in spike form. This biologically plausible wiring lets the network retain the global modeling power of transformers without leaving the spike domain.
Benchmarks on 13 datasets, including ImageNet‑like static images, neuromorphic event streams, and language modeling tasks, show Spikingformer matching or exceeding prior SNN backbones while staying fully spike‑driven. The results demonstrate a universal backbone that works across vision and language without the extra energy cost of mixed‑precision residuals.
TakeawayRedesign residual paths to stay spike‑only and you keep transformer power without extra energy overhead.