{"project":[{"call_identifier":"H2020","grant_number":"819603","_id":"0aacfa84-070f-11eb-9043-d7eb2c709234","name":"Learning the shape of synaptic plasticity rules for neuronal architectures and function through machine learning."},{"grant_number":"214316/Z/18/Z","_id":"c084a126-5a5b-11eb-8a69-d75314a70a87","name":"What’s in a memory? Spatiotemporal dynamics in strongly coupled recurrent neuronal networks."}],"intvolume":"        33","page":"899-925","issue":"4","day":"01","article_processing_charge":"No","title":"The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks","file_date_updated":"2022-04-08T06:05:39Z","publication_identifier":{"eissn":["1530-888X"],"issn":["0899-7667"]},"date_created":"2020-08-12T12:08:24Z","language":[{"iso":"eng"}],"oa":1,"abstract":[{"lang":"eng","text":"Brains process information in spiking neural networks. Their intricate connections shape the diverse functions these networks perform. In comparison, the functional capabilities of models of spiking networks are still rudimentary. This shortcoming is mainly due to the lack of insight and practical algorithms to construct the necessary connectivity. Any such algorithm typically attempts to build networks by iteratively reducing the error compared to a desired output. But assigning credit to hidden units in multi-layered spiking networks has remained challenging due to the non-differentiable nonlinearity of spikes. To avoid this issue, one can employ surrogate gradients to discover the required connectivity in spiking network models. However, the choice of a surrogate is not unique, raising the question of how its implementation influences the effectiveness of the method. Here, we use numerical simulations to systematically study how essential design parameters of surrogate gradients impact learning performance on a range of classification problems. We show that surrogate gradient learning is robust to different shapes of underlying surrogate derivatives, but the choice of the derivative’s scale can substantially affect learning performance. When we combine surrogate gradients with a suitable activity regularization technique, robust information processing can be achieved in spiking networks even at the sparse activity limit. Our study provides a systematic account of the remarkable robustness of surrogate gradient learning and serves as a practical guide to model functional spiking neural networks."}],"pmid":1,"ec_funded":1,"date_published":"2021-03-01T00:00:00Z","oa_version":"Published Version","department":[{"_id":"TiVo"}],"file":[{"file_id":"11131","access_level":"open_access","success":1,"checksum":"eac5a51c24c8989ae7cf9ae32ec3bc95","creator":"dernst","file_size":1611614,"date_created":"2022-04-08T06:05:39Z","date_updated":"2022-04-08T06:05:39Z","content_type":"application/pdf","relation":"main_file","file_name":"2021_NeuralComputation_Zenke.pdf"}],"article_type":"original","publisher":"MIT Press","tmp":{"legal_code_url":"https://creativecommons.org/licenses/by/4.0/legalcode","image":"/images/cc_by.png","short":"CC BY (4.0)","name":"Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)"},"type":"journal_article","quality_controlled":"1","doi":"10.1162/neco_a_01367","year":"2021","_id":"8253","user_id":"4359f0d1-fa6c-11eb-b949-802e58b17ae8","external_id":{"pmid":["33513328"],"isi":["000663433900003"]},"author":[{"full_name":"Zenke, Friedemann","first_name":"Friedemann","last_name":"Zenke","orcid":"0000-0003-1883-644X"},{"orcid":"0000-0003-3295-6181","last_name":"Vogels","id":"CB6FF8D2-008F-11EA-8E08-2637E6697425","first_name":"Tim P","full_name":"Vogels, Tim P"}],"has_accepted_license":"1","date_updated":"2023-08-04T10:53:14Z","status":"public","acknowledgement":"F.Z. was supported by the Wellcome Trust (110124/Z/15/Z) and the Novartis Research Foundation. T.P.V. was supported by a Wellcome Trust Sir Henry Dale Research fellowship (WT100000), a Wellcome Trust Senior Research Fellowship (214316/Z/18/Z), and an ERC Consolidator Grant SYNAPSEEK.","ddc":["000","570"],"volume":33,"scopus_import":"1","month":"03","publication":"Neural Computation","citation":{"mla":"Zenke, Friedemann, and Tim P. Vogels. “The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks.” <i>Neural Computation</i>, vol. 33, no. 4, MIT Press, 2021, pp. 899–925, doi:<a href=\"https://doi.org/10.1162/neco_a_01367\">10.1162/neco_a_01367</a>.","ista":"Zenke F, Vogels TP. 2021. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation. 33(4), 899–925.","ieee":"F. Zenke and T. P. Vogels, “The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks,” <i>Neural Computation</i>, vol. 33, no. 4. MIT Press, pp. 899–925, 2021.","short":"F. Zenke, T.P. Vogels, Neural Computation 33 (2021) 899–925.","apa":"Zenke, F., &#38; Vogels, T. P. (2021). The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. <i>Neural Computation</i>. MIT Press. <a href=\"https://doi.org/10.1162/neco_a_01367\">https://doi.org/10.1162/neco_a_01367</a>","chicago":"Zenke, Friedemann, and Tim P Vogels. “The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks.” <i>Neural Computation</i>. MIT Press, 2021. <a href=\"https://doi.org/10.1162/neco_a_01367\">https://doi.org/10.1162/neco_a_01367</a>.","ama":"Zenke F, Vogels TP. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. <i>Neural Computation</i>. 2021;33(4):899-925. doi:<a href=\"https://doi.org/10.1162/neco_a_01367\">10.1162/neco_a_01367</a>"},"isi":1,"publication_status":"published"}