High-dimensional analysis of knowledge distillation: Weak-to-Strong generalization and scaling laws

conference paper High-dimensional analysis of knowledge distillation: Weak-to-Strong generalization and scaling laws published yes M. Emrullah Ildiz author Halil Alperen Gozeten author Ege Onur Taga author Marco Mondelli author 27EB676C-8706-11E9-9510-7717E66974250000-0002-3242-7020 Samet Oymak author MaMo department ICLR: International Conference on Learning Representations Inference in High Dimensions: Light-speed Algorithms and Information Limits project A growing number of machine learning scenarios rely on knowledge distillation where one uses the output of a surrogate model as labels to supervise the training of a target model. In this work, we provide a sharp characterization of this process for ridgeless, high-dimensional regression, under two settings: (i) model shift, where the surrogate model is arbitrary, and (ii) distribution shift, where the surrogate model is the solution of empirical risk minimization with out-of-distribution data. In both cases, we characterize the precise risk of the target model through non-asymptotic bounds in terms of sample size and data distribution under mild conditions. As a consequence, we identify the form of the optimal surrogate model, which reveals the benefits and limitations of discarding weak features in a data-dependent fashion. In the context of weak-to-strong (W2S) generalization, this has the interpretation that (i) W2S training, with the surrogate as the weak model, can provably outperform training with strong labels under the same data budget, but (ii) it is unable to improve the data scaling law. We validate our results on numerical experiments both on ridgeless regression and on neural network architectures. https://research-explorer.ista.ac.at/download/20033/20112/2025_ICLR_Ildiz.pdf application/pdfno ICLR2025Singapore, Singapore eng 13th International Conference on Learning Representations 9798331320850 2410.18837 2967-3006 M. Emrullah Ildiz, H. A. Gozeten, E. O. Taga, M. Mondelli, and S. Oymak, “High-dimensional analysis of knowledge distillation: Weak-to-Strong generalization and scaling laws,” in 13th International Conference on Learning Representations, Singapore, Singapore, 2025, pp. 2967–3006. Emrullah Ildiz, M., et al. “High-Dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws.” 13th International Conference on Learning Representations, ICLR, 2025, pp. 2967–3006. Emrullah Ildiz, M., Halil Alperen Gozeten, Ege Onur Taga, Marco Mondelli, and Samet Oymak. “High-Dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws.” In 13th International Conference on Learning Representations, 2967–3006. ICLR, 2025. Emrullah Ildiz, M., Gozeten, H. A., Taga, E. O., Mondelli, M., & Oymak, S. (2025). High-dimensional analysis of knowledge distillation: Weak-to-Strong generalization and scaling laws. In 13th International Conference on Learning Representations (pp. 2967–3006). Singapore, Singapore: ICLR. M. Emrullah Ildiz, H.A. Gozeten, E.O. Taga, M. Mondelli, S. Oymak, in:, 13th International Conference on Learning Representations, ICLR, 2025, pp. 2967–3006. Emrullah Ildiz M, Gozeten HA, Taga EO, Mondelli M, Oymak S. High-dimensional analysis of knowledge distillation: Weak-to-Strong generalization and scaling laws. In: 13th International Conference on Learning Representations. ICLR; 2025:2967-3006. Emrullah Ildiz M, Gozeten HA, Taga EO, Mondelli M, Oymak S. 2025. High-dimensional analysis of knowledge distillation: Weak-to-Strong generalization and scaling laws. 13th International Conference on Learning Representations. ICLR: International Conference on Learning Representations, 2967–3006. 200332025-07-20T22:02:02Z2025-08-04T08:33:58Z