---
_id: '9416'
abstract:
- lang: eng
  text: 'We study the inductive bias of two-layer ReLU networks trained by gradient
    flow. We identify a class of easy-to-learn (`orthogonally separable'') datasets,
    and characterise the solution that ReLU networks trained on such datasets converge
    to. Irrespective of network width, the solution turns out to be a combination
    of two max-margin classifiers: one corresponding to the positive data subset and
    one corresponding to the negative data subset. The proof is based on the recently
    introduced concept of extremal sectors, for which we prove a number of properties
    in the context of orthogonal separability. In particular, we prove stationarity
    of activation patterns from some time  onwards, which enables a reduction of the
    ReLU network to an ensemble of linear subnetworks.'
article_processing_charge: No
author:
- first_name: Phuong
  full_name: Bui Thi Mai, Phuong
  id: 3EC6EE64-F248-11E8-B48F-1D18A9856A87
  last_name: Bui Thi Mai
- first_name: Christoph
  full_name: Lampert, Christoph
  id: 40C20FD2-F248-11E8-B48F-1D18A9856A87
  last_name: Lampert
  orcid: 0000-0001-8622-7887
citation:
  ama: 'Phuong M, Lampert C. The inductive bias of ReLU networks on orthogonally separable
    data. In: <i>9th International Conference on Learning Representations</i>. ; 2021.'
  apa: Phuong, M., &#38; Lampert, C. (2021). The inductive bias of ReLU networks on
    orthogonally separable data. In <i>9th International Conference on Learning Representations</i>.
    Virtual.
  chicago: Phuong, Mary, and Christoph Lampert. “The Inductive Bias of ReLU Networks
    on Orthogonally Separable Data.” In <i>9th International Conference on Learning
    Representations</i>, 2021.
  ieee: M. Phuong and C. Lampert, “The inductive bias of ReLU networks on orthogonally
    separable data,” in <i>9th International Conference on Learning Representations</i>,
    Virtual, 2021.
  ista: 'Phuong M, Lampert C. 2021. The inductive bias of ReLU networks on orthogonally
    separable data. 9th International Conference on Learning Representations. ICLR:
    International Conference on Learning Representations.'
  mla: Phuong, Mary, and Christoph Lampert. “The Inductive Bias of ReLU Networks on
    Orthogonally Separable Data.” <i>9th International Conference on Learning Representations</i>,
    2021.
  short: M. Phuong, C. Lampert, in:, 9th International Conference on Learning Representations,
    2021.
conference:
  end_date: 2021-05-07
  location: Virtual
  name: 'ICLR: International Conference on Learning Representations'
  start_date: 2021-05-03
corr_author: '1'
date_created: 2021-05-24T11:16:46Z
date_published: 2021-05-01T00:00:00Z
date_updated: 2025-06-30T10:07:30Z
day: '01'
ddc:
- '000'
department:
- _id: GradSch
- _id: ChLa
file:
- access_level: open_access
  checksum: f34ff17017527db5ba6927f817bdd125
  content_type: application/pdf
  creator: bphuong
  date_created: 2021-05-24T11:15:57Z
  date_updated: 2021-05-24T11:15:57Z
  file_id: '9417'
  file_name: iclr2021_conference.pdf
  file_size: 502356
  relation: main_file
file_date_updated: 2021-05-24T11:15:57Z
has_accepted_license: '1'
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://openreview.net/pdf?id=krz7T0xU9Z_
month: '05'
oa: 1
oa_version: Published Version
publication: 9th International Conference on Learning Representations
publication_status: published
quality_controlled: '1'
related_material:
  record:
  - id: '9418'
    relation: dissertation_contains
    status: public
scopus_import: '1'
status: public
title: The inductive bias of ReLU networks on orthogonally separable data
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2021'
...