{"publication":"13th International Conference on Learning Representations","file_date_updated":"2025-08-04T08:32:38Z","date_updated":"2025-08-04T08:33:58Z","date_created":"2025-07-20T22:02:02Z","day":"01","type":"conference","date_published":"2025-04-01T00:00:00Z","ddc":["000"],"publication_identifier":{"isbn":["9798331320850"]},"page":"2967-3006","has_accepted_license":"1","publisher":"ICLR","title":"High-dimensional analysis of knowledge distillation: Weak-to-Strong generalization and scaling laws","publication_status":"published","oa_version":"Published Version","scopus_import":"1","conference":{"location":"Singapore, Singapore","end_date":"2025-04-28","start_date":"2025-04-24","name":"ICLR: International Conference on Learning Representations"},"quality_controlled":"1","_id":"20033","acknowledgement":"M.E.I., H.A.G., E.O.T., S.O. are supported by the NSF grants CCF-2046816, CCF-2403075, the Office of Naval Research grant N000142412289, an OpenAI Agentic AI Systems grant, and gifts by Open Philanthropy and Google Research. M. M. is funded by the European Union (ERC, INF2, project number 101161364). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.","arxiv":1,"month":"04","year":"2025","file":[{"file_name":"2025_ICLR_Ildiz.pdf","access_level":"open_access","date_created":"2025-08-04T08:32:38Z","date_updated":"2025-08-04T08:32:38Z","relation":"main_file","file_id":"20112","checksum":"5a38b093ebb4ee4eb662ea142621a5ca","file_size":528171,"success":1,"content_type":"application/pdf","creator":"dernst"}],"user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","OA_place":"publisher","language":[{"iso":"eng"}],"abstract":[{"lang":"eng","text":"A growing number of machine learning scenarios rely on knowledge distillation where one uses the output of a surrogate model as labels to supervise the training of a target model. In this work, we provide a sharp characterization of this process for ridgeless, high-dimensional regression, under two settings: (i) model shift, where the surrogate model is arbitrary, and (ii) distribution shift, where the surrogate model is the solution of empirical risk minimization with out-of-distribution data. In both cases, we characterize the precise risk of the target model through non-asymptotic bounds in terms of sample size and data distribution under mild conditions. As a consequence, we identify the form of the optimal surrogate model, which reveals the benefits and limitations of discarding weak features in a data-dependent fashion. In the context of weak-to-strong (W2S) generalization, this has the interpretation that (i) W2S training, with the surrogate as the weak model, can provably outperform training with strong labels under the same data budget, but (ii) it is unable to improve the data scaling law. We validate our results on numerical experiments both on ridgeless regression and on neural network architectures."}],"project":[{"grant_number":"101161364","_id":"911e6d1f-16d5-11f0-9cad-c5c68c6a1cdf","name":"Inference in High Dimensions: Light-speed Algorithms and Information Limits"}],"status":"public","tmp":{"short":"CC BY (4.0)","legal_code_url":"https://creativecommons.org/licenses/by/4.0/legalcode","name":"Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)","image":"/images/cc_by.png"},"department":[{"_id":"MaMo"}],"OA_type":"diamond","author":[{"first_name":"M.","full_name":"Emrullah Ildiz, M.","last_name":"Emrullah Ildiz"},{"full_name":"Gozeten, Halil Alperen","last_name":"Gozeten","first_name":"Halil Alperen"},{"full_name":"Taga, Ege Onur","last_name":"Taga","first_name":"Ege Onur"},{"last_name":"Mondelli","full_name":"Mondelli, Marco","orcid":"0000-0002-3242-7020","first_name":"Marco","id":"27EB676C-8706-11E9-9510-7717E6697425"},{"full_name":"Oymak, Samet","last_name":"Oymak","first_name":"Samet"}],"external_id":{"arxiv":["2410.18837"]},"citation":{"short":"M. Emrullah Ildiz, H.A. Gozeten, E.O. Taga, M. Mondelli, S. Oymak, in:, 13th International Conference on Learning Representations, ICLR, 2025, pp. 2967–3006.","chicago":"Emrullah Ildiz, M., Halil Alperen Gozeten, Ege Onur Taga, Marco Mondelli, and Samet Oymak. “High-Dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws.” In <i>13th International Conference on Learning Representations</i>, 2967–3006. ICLR, 2025.","apa":"Emrullah Ildiz, M., Gozeten, H. A., Taga, E. O., Mondelli, M., &#38; Oymak, S. (2025). High-dimensional analysis of knowledge distillation: Weak-to-Strong generalization and scaling laws. In <i>13th International Conference on Learning Representations</i> (pp. 2967–3006). Singapore, Singapore: ICLR.","ama":"Emrullah Ildiz M, Gozeten HA, Taga EO, Mondelli M, Oymak S. High-dimensional analysis of knowledge distillation: Weak-to-Strong generalization and scaling laws. In: <i>13th International Conference on Learning Representations</i>. ICLR; 2025:2967-3006.","mla":"Emrullah Ildiz, M., et al. “High-Dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws.” <i>13th International Conference on Learning Representations</i>, ICLR, 2025, pp. 2967–3006.","ista":"Emrullah Ildiz M, Gozeten HA, Taga EO, Mondelli M, Oymak S. 2025. High-dimensional analysis of knowledge distillation: Weak-to-Strong generalization and scaling laws. 13th International Conference on Learning Representations. ICLR: International Conference on Learning Representations, 2967–3006.","ieee":"M. Emrullah Ildiz, H. A. Gozeten, E. O. Taga, M. Mondelli, and S. Oymak, “High-dimensional analysis of knowledge distillation: Weak-to-Strong generalization and scaling laws,” in <i>13th International Conference on Learning Representations</i>, Singapore, Singapore, 2025, pp. 2967–3006."},"oa":1,"article_processing_charge":"No"}