---
_id: '7481'
abstract:
- lang: eng
  text: 'We address the following question:  How redundant is the parameterisation
    of ReLU networks? Specifically, we consider transformations of the weight space
    which leave the function implemented by the network intact.  Two such transformations
    are known for feed-forward architectures:  permutation of neurons within a layer,
    and positive scaling of all incoming weights of a neuron coupled with inverse
    scaling of its outgoing weights. In this work, we show for architectures with
    non-increasing widths that permutation and scaling are in fact the only function-preserving
    weight transformations.  For any eligible architecture we give an explicit construction
    of a neural network such that any other network that implements the same function
    can be obtained from the original one by the application of permutations and rescaling.  The
    proof relies on a geometric understanding of boundaries between linear regions
    of ReLU networks, and we hope the developed mathematical tools are of independent
    interest.'
article_processing_charge: No
author:
- first_name: Phuong
  full_name: Bui Thi Mai, Phuong
  id: 3EC6EE64-F248-11E8-B48F-1D18A9856A87
  last_name: Bui Thi Mai
- first_name: Christoph
  full_name: Lampert, Christoph
  id: 40C20FD2-F248-11E8-B48F-1D18A9856A87
  last_name: Lampert
  orcid: 0000-0001-8622-7887
citation:
  ama: 'Phuong M, Lampert C. Functional vs. parametric equivalence of ReLU networks.
    In: <i>8th International Conference on Learning Representations</i>. ; 2020.'
  apa: Phuong, M., &#38; Lampert, C. (2020). Functional vs. parametric equivalence
    of ReLU networks. In <i>8th International Conference on Learning Representations</i>.
    Online.
  chicago: Phuong, Mary, and Christoph Lampert. “Functional vs. Parametric Equivalence
    of ReLU Networks.” In <i>8th International Conference on Learning Representations</i>,
    2020.
  ieee: M. Phuong and C. Lampert, “Functional vs. parametric equivalence of ReLU networks,”
    in <i>8th International Conference on Learning Representations</i>, Online, 2020.
  ista: 'Phuong M, Lampert C. 2020. Functional vs. parametric equivalence of ReLU
    networks. 8th International Conference on Learning Representations. ICLR: International
    Conference on Learning Representations.'
  mla: Phuong, Mary, and Christoph Lampert. “Functional vs. Parametric Equivalence
    of ReLU Networks.” <i>8th International Conference on Learning Representations</i>,
    2020.
  short: M. Phuong, C. Lampert, in:, 8th International Conference on Learning Representations,
    2020.
conference:
  end_date: 2020-04-30
  location: Online
  name: 'ICLR: International Conference on Learning Representations'
  start_date: 2020-04-27
corr_author: '1'
date_created: 2020-02-11T09:07:37Z
date_published: 2020-04-26T00:00:00Z
date_updated: 2026-04-08T07:01:16Z
day: '26'
ddc:
- '000'
department:
- _id: ChLa
file:
- access_level: open_access
  checksum: 8d372ea5defd8cb8fdc430111ed754a9
  content_type: application/pdf
  creator: bphuong
  date_created: 2020-02-11T09:07:27Z
  date_updated: 2020-07-14T12:47:59Z
  file_id: '7482'
  file_name: main.pdf
  file_size: 405469
  relation: main_file
file_date_updated: 2020-07-14T12:47:59Z
has_accepted_license: '1'
language:
- iso: eng
month: '04'
oa: 1
oa_version: Published Version
publication: 8th International Conference on Learning Representations
publication_status: published
quality_controlled: '1'
related_material:
  link:
  - relation: supplementary_material
    url: https://iclr.cc/virtual_2020/poster_Bylx-TNKvH.html
  record:
  - id: '9418'
    relation: dissertation_contains
    status: public
status: public
title: Functional vs. parametric equivalence of ReLU networks
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2020'
...