{"acknowledgement":"MS has received funding from the European Union’s Horizon 2020 research and innovation programme\r\nunder the Marie Skłodowska-Curie grant agreement No 101034413.","ec_funded":1,"status":"public","abstract":[{"text":"Knowledge distillation is a popular approach for enhancing the performance of \"student\" models, with lower representational capacity, by taking advantage of more powerful \"teacher\" models. Despite its apparent simplicity, the underlying mechanics behind knowledge distillation (KD) are not yet fully understood. In this work, we shed new light on the inner workings of this method, by examining it from an optimization perspective. Specifically, we show that, in the context of linear and deep linear models, KD can be interpreted as a novel type of stochastic variance reduction mechanism. We provide a detailed convergence analysis of the resulting dynamics, which hold under standard assumptions for both strongly-convex and non-convex losses, showing that KD acts as a form of \\emph{partial variance reduction}, which can reduce the stochastic gradient noise, but may not eliminate it completely, depending on the properties of the teacher'' model. Our analysis puts further emphasis on the need for careful parametrization of KD, in particular w.r.t. the weighting of the distillation loss, and is validated empirically on both linear models and deep neural networks.","lang":"eng"}],"date_updated":"2024-05-22T08:10:38Z","tmp":{"name":"Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)","short":"CC BY (4.0)","legal_code_url":"https://creativecommons.org/licenses/by/4.0/legalcode","image":"/images/cc_by.png"},"intvolume":"        36","quality_controlled":"1","day":"15","external_id":{"arxiv":["2305.17581"]},"has_accepted_license":"1","project":[{"_id":"fc2ed2f7-9c52-11eb-aca3-c01059dda49c","name":"IST-BRIDGE: International postdoctoral program","grant_number":"101034413","call_identifier":"H2020"}],"volume":36,"author":[{"first_name":"Mher","full_name":"Safaryan, Mher","id":"dd546b39-0804-11ed-9c55-ef075c39778d","last_name":"Safaryan"},{"first_name":"Elena-Alexandra","full_name":"Peste, Elena-Alexandra","id":"32D78294-F248-11E8-B48F-1D18A9856A87","last_name":"Peste"},{"orcid":"0000-0003-3650-940X","first_name":"Dan-Adrian","full_name":"Alistarh, Dan-Adrian","last_name":"Alistarh","id":"4A899BFC-F248-11E8-B48F-1D18A9856A87"}],"language":[{"iso":"eng"}],"file_date_updated":"2024-05-22T08:08:08Z","department":[{"_id":"DaAl"}],"user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","date_created":"2024-05-05T22:01:04Z","alternative_title":["NeurIPS"],"publication":"36th Conference on Neural Information Processing Systems","date_published":"2023-12-15T00:00:00Z","article_processing_charge":"Yes","file":[{"creator":"dernst","checksum":"288c5148a85abf24ad5e22a6b1183655","success":1,"access_level":"open_access","file_id":"15417","relation":"main_file","file_name":"2023_Neurips_Safaryan.pdf","date_created":"2024-05-22T08:08:08Z","file_size":672571,"date_updated":"2024-05-22T08:08:08Z","content_type":"application/pdf"}],"year":"2023","ddc":["000"],"title":"Knowledge distillation performs partial variance reduction","month":"12","publication_identifier":{"issn":["1049-5258"]},"_id":"15363","citation":{"ama":"Safaryan M, Krumes A, Alistarh D-A. Knowledge distillation performs partial variance reduction. In: <i>36th Conference on Neural Information Processing Systems</i>. Vol 36. ; 2023.","ista":"Safaryan M, Krumes A, Alistarh D-A. 2023. Knowledge distillation performs partial variance reduction. 36th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, NeurIPS, vol. 36.","chicago":"Safaryan, Mher, Alexandra Krumes, and Dan-Adrian Alistarh. “Knowledge Distillation Performs Partial Variance Reduction.” In <i>36th Conference on Neural Information Processing Systems</i>, Vol. 36, 2023.","mla":"Safaryan, Mher, et al. “Knowledge Distillation Performs Partial Variance Reduction.” <i>36th Conference on Neural Information Processing Systems</i>, vol. 36, 2023.","ieee":"M. Safaryan, A. Krumes, and D.-A. Alistarh, “Knowledge distillation performs partial variance reduction,” in <i>36th Conference on Neural Information Processing Systems</i>, New Orleans, LA, United States, 2023, vol. 36.","short":"M. Safaryan, A. Krumes, D.-A. Alistarh, in:, 36th Conference on Neural Information Processing Systems, 2023.","apa":"Safaryan, M., Krumes, A., &#38; Alistarh, D.-A. (2023). Knowledge distillation performs partial variance reduction. In <i>36th Conference on Neural Information Processing Systems</i> (Vol. 36). New Orleans, LA, United States."},"type":"conference","oa_version":"Published Version","oa":1,"scopus_import":"1","publication_status":"published","conference":{"start_date":"2023-12-10","location":"New Orleans, LA, United States","end_date":"2023-12-16","name":"NeurIPS: Neural Information Processing Systems"}}