---
OA_place: publisher
OA_type: diamond
_id: '20033'
abstract:
- lang: eng
  text: 'A growing number of machine learning scenarios rely on knowledge distillation
    where one uses the output of a surrogate model as labels to supervise the training
    of a target model. In this work, we provide a sharp characterization of this process
    for ridgeless, high-dimensional regression, under two settings: (i) model shift,
    where the surrogate model is arbitrary, and (ii) distribution shift, where the
    surrogate model is the solution of empirical risk minimization with out-of-distribution
    data. In both cases, we characterize the precise risk of the target model through
    non-asymptotic bounds in terms of sample size and data distribution under mild
    conditions. As a consequence, we identify the form of the optimal surrogate model,
    which reveals the benefits and limitations of discarding weak features in a data-dependent
    fashion. In the context of weak-to-strong (W2S) generalization, this has the interpretation
    that (i) W2S training, with the surrogate as the weak model, can provably outperform
    training with strong labels under the same data budget, but (ii) it is unable
    to improve the data scaling law. We validate our results on numerical experiments
    both on ridgeless regression and on neural network architectures.'
acknowledgement: M.E.I., H.A.G., E.O.T., S.O. are supported by the NSF grants CCF-2046816,
  CCF-2403075, the Office of Naval Research grant N000142412289, an OpenAI Agentic
  AI Systems grant, and gifts by Open Philanthropy and Google Research. M. M. is funded
  by the European Union (ERC, INF2, project number 101161364). Views and opinions
  expressed are however those of the author(s) only and do not necessarily reflect
  those of the European Union or the European Research Council Executive Agency. Neither
  the European Union nor the granting authority can be held responsible for them.
article_processing_charge: No
arxiv: 1
author:
- first_name: M.
  full_name: Emrullah Ildiz, M.
  last_name: Emrullah Ildiz
- first_name: Halil Alperen
  full_name: Gozeten, Halil Alperen
  last_name: Gozeten
- first_name: Ege Onur
  full_name: Taga, Ege Onur
  last_name: Taga
- first_name: Marco
  full_name: Mondelli, Marco
  id: 27EB676C-8706-11E9-9510-7717E6697425
  last_name: Mondelli
  orcid: 0000-0002-3242-7020
- first_name: Samet
  full_name: Oymak, Samet
  last_name: Oymak
citation:
  ama: 'Emrullah Ildiz M, Gozeten HA, Taga EO, Mondelli M, Oymak S. High-dimensional
    analysis of knowledge distillation: Weak-to-Strong generalization and scaling
    laws. In: <i>13th International Conference on Learning Representations</i>. ICLR;
    2025:2967-3006.'
  apa: 'Emrullah Ildiz, M., Gozeten, H. A., Taga, E. O., Mondelli, M., &#38; Oymak,
    S. (2025). High-dimensional analysis of knowledge distillation: Weak-to-Strong
    generalization and scaling laws. In <i>13th International Conference on Learning
    Representations</i> (pp. 2967–3006). Singapore, Singapore: ICLR.'
  chicago: 'Emrullah Ildiz, M., Halil Alperen Gozeten, Ege Onur Taga, Marco Mondelli,
    and Samet Oymak. “High-Dimensional Analysis of Knowledge Distillation: Weak-to-Strong
    Generalization and Scaling Laws.” In <i>13th International Conference on Learning
    Representations</i>, 2967–3006. ICLR, 2025.'
  ieee: 'M. Emrullah Ildiz, H. A. Gozeten, E. O. Taga, M. Mondelli, and S. Oymak,
    “High-dimensional analysis of knowledge distillation: Weak-to-Strong generalization
    and scaling laws,” in <i>13th International Conference on Learning Representations</i>,
    Singapore, Singapore, 2025, pp. 2967–3006.'
  ista: 'Emrullah Ildiz M, Gozeten HA, Taga EO, Mondelli M, Oymak S. 2025. High-dimensional
    analysis of knowledge distillation: Weak-to-Strong generalization and scaling
    laws. 13th International Conference on Learning Representations. ICLR: International
    Conference on Learning Representations, 2967–3006.'
  mla: 'Emrullah Ildiz, M., et al. “High-Dimensional Analysis of Knowledge Distillation:
    Weak-to-Strong Generalization and Scaling Laws.” <i>13th International Conference
    on Learning Representations</i>, ICLR, 2025, pp. 2967–3006.'
  short: M. Emrullah Ildiz, H.A. Gozeten, E.O. Taga, M. Mondelli, S. Oymak, in:, 13th
    International Conference on Learning Representations, ICLR, 2025, pp. 2967–3006.
conference:
  end_date: 2025-04-28
  location: Singapore, Singapore
  name: 'ICLR: International Conference on Learning Representations'
  start_date: 2025-04-24
date_created: 2025-07-20T22:02:02Z
date_published: 2025-04-01T00:00:00Z
date_updated: 2025-08-04T08:33:58Z
day: '01'
ddc:
- '000'
department:
- _id: MaMo
external_id:
  arxiv:
  - '2410.18837'
file:
- access_level: open_access
  checksum: 5a38b093ebb4ee4eb662ea142621a5ca
  content_type: application/pdf
  creator: dernst
  date_created: 2025-08-04T08:32:38Z
  date_updated: 2025-08-04T08:32:38Z
  file_id: '20112'
  file_name: 2025_ICLR_Ildiz.pdf
  file_size: 528171
  relation: main_file
  success: 1
file_date_updated: 2025-08-04T08:32:38Z
has_accepted_license: '1'
language:
- iso: eng
month: '04'
oa: 1
oa_version: Published Version
page: 2967-3006
project:
- _id: 911e6d1f-16d5-11f0-9cad-c5c68c6a1cdf
  grant_number: '101161364'
  name: 'Inference in High Dimensions: Light-speed Algorithms and Information Limits'
publication: 13th International Conference on Learning Representations
publication_identifier:
  isbn:
  - '9798331320850'
publication_status: published
publisher: ICLR
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'High-dimensional analysis of knowledge distillation: Weak-to-Strong generalization
  and scaling laws'
tmp:
  image: /images/cc_by.png
  legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
  name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
  short: CC BY (4.0)
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2025'
...