---
_id: '8253'
abstract:
- lang: eng
text: Brains process information in spiking neural networks. Their intricate connections
shape the diverse functions these networks perform. In comparison, the functional
capabilities of models of spiking networks are still rudimentary. This shortcoming
is mainly due to the lack of insight and practical algorithms to construct the
necessary connectivity. Any such algorithm typically attempts to build networks
by iteratively reducing the error compared to a desired output. But assigning
credit to hidden units in multi-layered spiking networks has remained challenging
due to the non-differentiable nonlinearity of spikes. To avoid this issue, one
can employ surrogate gradients to discover the required connectivity in spiking
network models. However, the choice of a surrogate is not unique, raising the
question of how its implementation influences the effectiveness of the method.
Here, we use numerical simulations to systematically study how essential design
parameters of surrogate gradients impact learning performance on a range of classification
problems. We show that surrogate gradient learning is robust to different shapes
of underlying surrogate derivatives, but the choice of the derivative’s scale
can substantially affect learning performance. When we combine surrogate gradients
with a suitable activity regularization technique, robust information processing
can be achieved in spiking networks even at the sparse activity limit. Our study
provides a systematic account of the remarkable robustness of surrogate gradient
learning and serves as a practical guide to model functional spiking neural networks.
acknowledgement: F.Z. was supported by the Wellcome Trust (110124/Z/15/Z) and the
Novartis Research Foundation. T.P.V. was supported by a Wellcome Trust Sir Henry
Dale Research fellowship (WT100000), a Wellcome Trust Senior Research Fellowship
(214316/Z/18/Z), and an ERC Consolidator Grant SYNAPSEEK.
article_processing_charge: No
article_type: original
author:
- first_name: Friedemann
full_name: Zenke, Friedemann
last_name: Zenke
orcid: 0000-0003-1883-644X
- first_name: Tim P
full_name: Vogels, Tim P
id: CB6FF8D2-008F-11EA-8E08-2637E6697425
last_name: Vogels
orcid: 0000-0003-3295-6181
citation:
ama: Zenke F, Vogels TP. The remarkable robustness of surrogate gradient learning
for instilling complex function in spiking neural networks. Neural Computation.
2021;33(4):899-925. doi:10.1162/neco_a_01367
apa: Zenke, F., & Vogels, T. P. (2021). The remarkable robustness of surrogate
gradient learning for instilling complex function in spiking neural networks.
Neural Computation. MIT Press. https://doi.org/10.1162/neco_a_01367
chicago: Zenke, Friedemann, and Tim P Vogels. “The Remarkable Robustness of Surrogate
Gradient Learning for Instilling Complex Function in Spiking Neural Networks.”
Neural Computation. MIT Press, 2021. https://doi.org/10.1162/neco_a_01367.
ieee: F. Zenke and T. P. Vogels, “The remarkable robustness of surrogate gradient
learning for instilling complex function in spiking neural networks,” Neural
Computation, vol. 33, no. 4. MIT Press, pp. 899–925, 2021.
ista: Zenke F, Vogels TP. 2021. The remarkable robustness of surrogate gradient
learning for instilling complex function in spiking neural networks. Neural Computation.
33(4), 899–925.
mla: Zenke, Friedemann, and Tim P. Vogels. “The Remarkable Robustness of Surrogate
Gradient Learning for Instilling Complex Function in Spiking Neural Networks.”
Neural Computation, vol. 33, no. 4, MIT Press, 2021, pp. 899–925, doi:10.1162/neco_a_01367.
short: F. Zenke, T.P. Vogels, Neural Computation 33 (2021) 899–925.
corr_author: '1'
date_created: 2020-08-12T12:08:24Z
date_published: 2021-03-01T00:00:00Z
date_updated: 2025-04-14T09:44:14Z
day: '01'
ddc:
- '000'
- '570'
department:
- _id: TiVo
doi: 10.1162/neco_a_01367
ec_funded: 1
external_id:
isi:
- '000663433900003'
pmid:
- '33513328'
file:
- access_level: open_access
checksum: eac5a51c24c8989ae7cf9ae32ec3bc95
content_type: application/pdf
creator: dernst
date_created: 2022-04-08T06:05:39Z
date_updated: 2022-04-08T06:05:39Z
file_id: '11131'
file_name: 2021_NeuralComputation_Zenke.pdf
file_size: 1611614
relation: main_file
success: 1
file_date_updated: 2022-04-08T06:05:39Z
has_accepted_license: '1'
intvolume: ' 33'
isi: 1
issue: '4'
language:
- iso: eng
month: '03'
oa: 1
oa_version: Published Version
page: 899-925
pmid: 1
project:
- _id: 0aacfa84-070f-11eb-9043-d7eb2c709234
call_identifier: H2020
grant_number: '819603'
name: Learning the shape of synaptic plasticity rules for neuronal architectures
and function through machine learning.
- _id: c084a126-5a5b-11eb-8a69-d75314a70a87
grant_number: 214316/Z/18/Z
name: "Whatâ\x80\x99s in a memory? Spatiotemporal dynamics in strongly coupled recurrent
neuronal networks."
publication: Neural Computation
publication_identifier:
eissn:
- 1530-888X
issn:
- 0899-7667
publication_status: published
publisher: MIT Press
quality_controlled: '1'
scopus_import: '1'
status: public
title: The remarkable robustness of surrogate gradient learning for instilling complex
function in spiking neural networks
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: journal_article
user_id: 4359f0d1-fa6c-11eb-b949-802e58b17ae8
volume: 33
year: '2021'
...