{"volume":33,"ec_funded":1,"oa":1,"citation":{"short":"S.P. Singh, D.-A. Alistarh, in:, Advances in Neural Information Processing Systems, Curran Associates, 2020, pp. 18098–18109.","ieee":"S. P. Singh and D.-A. Alistarh, “WoodFisher: Efficient second-order approximation for neural network compression,” in <i>Advances in Neural Information Processing Systems</i>, Vancouver, Canada, 2020, vol. 33, pp. 18098–18109.","ama":"Singh SP, Alistarh D-A. WoodFisher: Efficient second-order approximation for neural network compression. In: <i>Advances in Neural Information Processing Systems</i>. Vol 33. Curran Associates; 2020:18098-18109.","apa":"Singh, S. P., &#38; Alistarh, D.-A. (2020). WoodFisher: Efficient second-order approximation for neural network compression. In <i>Advances in Neural Information Processing Systems</i> (Vol. 33, pp. 18098–18109). Vancouver, Canada: Curran Associates.","chicago":"Singh, Sidak Pal, and Dan-Adrian Alistarh. “WoodFisher: Efficient Second-Order Approximation for Neural Network Compression.” In <i>Advances in Neural Information Processing Systems</i>, 33:18098–109. Curran Associates, 2020.","mla":"Singh, Sidak Pal, and Dan-Adrian Alistarh. “WoodFisher: Efficient Second-Order Approximation for Neural Network Compression.” <i>Advances in Neural Information Processing Systems</i>, vol. 33, Curran Associates, 2020, pp. 18098–109.","ista":"Singh SP, Alistarh D-A. 2020. WoodFisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems. NeurIPS: Conference on Neural Information Processing Systems vol. 33, 18098–18109."},"day":"06","acknowledgement":"This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 805223 ScaleML). Also, we would like to thank Alexander Shevchenko, Alexandra Peste, and other members of the group for fruitful discussions.","quality_controlled":"1","date_updated":"2023-02-23T14:03:06Z","department":[{"_id":"DaAl"},{"_id":"ToHe"}],"author":[{"first_name":"Sidak Pal","last_name":"Singh","full_name":"Singh, Sidak Pal","id":"DD138E24-D89D-11E9-9DC0-DEF6E5697425"},{"orcid":"0000-0003-3650-940X","last_name":"Alistarh","full_name":"Alistarh, Dan-Adrian","first_name":"Dan-Adrian","id":"4A899BFC-F248-11E8-B48F-1D18A9856A87"}],"publication_identifier":{"isbn":["9781713829546"],"issn":["10495258"]},"year":"2020","_id":"9632","main_file_link":[{"open_access":"1","url":"https://proceedings.neurips.cc/paper/2020/hash/d1ff1ec86b62cd5f3903ff19c3a326b2-Abstract.html"}],"scopus_import":"1","project":[{"call_identifier":"H2020","_id":"268A44D6-B435-11E9-9278-68D0E5697425","name":"Elastic Coordination for Scalable Machine Learning","grant_number":"805223"}],"oa_version":"Published Version","date_created":"2021-07-04T22:01:26Z","language":[{"iso":"eng"}],"publication":"Advances in Neural Information Processing Systems","external_id":{"arxiv":["2004.14340"]},"conference":{"name":"NeurIPS: Conference on Neural Information Processing Systems","start_date":"2020-12-06","end_date":"2020-12-12","location":"Vancouver, Canada"},"user_id":"6785fbc1-c503-11eb-8a32-93094b40e1cf","month":"12","title":"WoodFisher: Efficient second-order approximation for neural network compression","status":"public","article_processing_charge":"No","abstract":[{"text":"Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems. Recently, there has been significant interest in utilizing this information in the context of deep\r\nneural networks; however, relatively little is known about the quality of existing approximations in this context. Our work examines this question, identifies issues with existing approaches, and proposes a method called WoodFisher to compute a faithful and efficient estimate of the inverse Hessian. Our main application is to neural network compression, where we build on the classic Optimal Brain Damage/Surgeon framework. We demonstrate that WoodFisher significantly outperforms popular state-of-the-art methods for oneshot pruning. Further, even when iterative, gradual pruning is allowed, our method results in a gain in test accuracy over the state-of-the-art approaches, for standard image classification datasets such as ImageNet ILSVRC. We examine how our method can be extended to take into account first-order information, as well as\r\nillustrate its ability to automatically set layer-wise pruning thresholds and perform compression in the limited-data regime. The code is available at the following link, https://github.com/IST-DASLab/WoodFisher.","lang":"eng"}],"intvolume":"        33","type":"conference","date_published":"2020-12-06T00:00:00Z","publisher":"Curran Associates","publication_status":"published","page":"18098-18109"}