---
_id: '12536'
abstract:
- lang: eng
  text: 'We consider the problem of estimating a rank-1 signal corrupted by structured
    rotationally invariant noise, and address the following question: how well do
    inference algorithms perform when the noise statistics is unknown and hence Gaussian
    noise is assumed? While the matched Bayes-optimal setting with unstructured noise
    is well understood, the analysis of this mismatched problem is only at its premises.
    In this paper, we make a step towards understanding the effect of the strong source
    of mismatch which is the noise statistics. Our main technical contribution is
    the rigorous analysis of a Bayes estimator and of an approximate message passing
    (AMP) algorithm, both of which incorrectly assume a Gaussian setup. The first
    result exploits the theory of spherical integrals and of low-rank matrix perturbations;
    the idea behind the second one is to design and analyze an artificial AMP which,
    by taking advantage of the flexibility in the denoisers, is able to "correct"
    the mismatch. Armed with these sharp asymptotic characterizations, we unveil a
    rich and often unexpected phenomenology. For example, despite AMP is in principle
    designed to efficiently compute the Bayes estimator, the former is outperformed
    by the latter in terms of mean-square error. We show that this performance gap
    is due to an incorrect estimation of the signal norm. In fact, when the SNR is
    large enough, the overlaps of the AMP and the Bayes estimator coincide, and they
    even match those of optimal estimators taking into account the structure of the
    noise.'
acknowledgement: "M. Mondelli was partially supported by the 2019 Lopez-Loreta Prize.
  The authors acknowledge\r\ndiscussions with A. Krajenbrink, M. Robinson, A. Depope,
  N. Macris and F. Pourkamali.\r\n"
alternative_title:
- NeurIPS
article_processing_charge: No
arxiv: 1
author:
- first_name: Jean
  full_name: Barbier, Jean
  last_name: Barbier
- first_name: TianQi
  full_name: Hou, TianQi
  last_name: Hou
- first_name: Marco
  full_name: Mondelli, Marco
  id: 27EB676C-8706-11E9-9510-7717E6697425
  last_name: Mondelli
  orcid: 0000-0002-3242-7020
- first_name: Manuel
  full_name: Saenz, Manuel
  last_name: Saenz
citation:
  ama: 'Barbier J, Hou T, Mondelli M, Saenz M. The price of ignorance: How much does
    it cost to forget noise structure in low-rank matrix estimation? In: <i>36th Annual
    Conference on Neural Information Processing Systems</i>. Vol 35. ; 2022.'
  apa: 'Barbier, J., Hou, T., Mondelli, M., &#38; Saenz, M. (2022). The price of ignorance:
    How much does it cost to forget noise structure in low-rank matrix estimation?
    In <i>36th Annual Conference on Neural Information Processing Systems</i> (Vol.
    35). New Orleans, LA, United States.'
  chicago: 'Barbier, Jean, TianQi Hou, Marco Mondelli, and Manuel Saenz. “The Price
    of Ignorance: How Much Does It Cost to Forget Noise Structure in Low-Rank Matrix
    Estimation?” In <i>36th Annual Conference on Neural Information Processing Systems</i>,
    Vol. 35, 2022.'
  ieee: 'J. Barbier, T. Hou, M. Mondelli, and M. Saenz, “The price of ignorance: How
    much does it cost to forget noise structure in low-rank matrix estimation?,” in
    <i>36th Annual Conference on Neural Information Processing Systems</i>, New Orleans,
    LA, United States, 2022, vol. 35.'
  ista: 'Barbier J, Hou T, Mondelli M, Saenz M. 2022. The price of ignorance: How
    much does it cost to forget noise structure in low-rank matrix estimation? 36th
    Annual Conference on Neural Information Processing Systems. NeurIPS: Neural Information
    Processing Systems, NeurIPS, vol. 35.'
  mla: 'Barbier, Jean, et al. “The Price of Ignorance: How Much Does It Cost to Forget
    Noise Structure in Low-Rank Matrix Estimation?” <i>36th Annual Conference on Neural
    Information Processing Systems</i>, vol. 35, 2022.'
  short: J. Barbier, T. Hou, M. Mondelli, M. Saenz, in:, 36th Annual Conference on
    Neural Information Processing Systems, 2022.
conference:
  end_date: 2022-12-09
  location: New Orleans, LA, United States
  name: 'NeurIPS: Neural Information Processing Systems'
  start_date: 2022-11-28
corr_author: '1'
date_created: 2023-02-10T13:45:41Z
date_published: 2022-11-20T00:00:00Z
date_updated: 2024-10-09T21:04:25Z
day: '20'
department:
- _id: MaMo
external_id:
  arxiv:
  - '2205.10009'
intvolume: '        35'
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://arxiv.org/abs/2205.10009
month: '11'
oa: 1
oa_version: Preprint
publication: 36th Annual Conference on Neural Information Processing Systems
publication_identifier:
  isbn:
  - '9781713871088'
publication_status: published
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'The price of ignorance: How much does it cost to forget noise structure in
  low-rank matrix estimation?'
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2022'
...
---
OA_place: repository
OA_type: green
_id: '12537'
abstract:
- lang: eng
  text: 'The Neural Tangent Kernel (NTK) has emerged as a powerful tool to provide
    memorization, optimization and generalization guarantees in deep neural networks.
    A line of work has studied the NTK spectrum for two-layer and deep networks with
    at least a layer with Ω(N) neurons, N being the number of training samples. Furthermore,
    there is increasing evidence suggesting that deep networks with sub-linear layer
    widths are powerful memorizers and optimizers, as long as the number of parameters
    exceeds the number of samples. Thus, a natural open question is whether the NTK
    is well conditioned in such a challenging sub-linear setup. In this paper, we
    answer this question in the affirmative. Our key technical contribution is a lower
    bound on the smallest NTK eigenvalue for deep networks with the minimum possible
    over-parameterization: the number of parameters is roughly Ω(N) and, hence, the
    number of neurons is as little as Ω(N−−√). To showcase the applicability of our
    NTK bounds, we provide two results concerning memorization capacity and optimization
    guarantees for gradient descent training.'
acknowledgement: "The authors were partially supported by the 2019 Lopez-Loreta prize,
  and they would like to thank\r\nQuynh Nguyen, Mahdi Soltanolkotabi and Adel Javanmard
  for helpful discussions.\r\n"
alternative_title:
- Advances in Neural Information Processing Systems
article_processing_charge: No
arxiv: 1
author:
- first_name: Simone
  full_name: Bombari, Simone
  id: ca726dda-de17-11ea-bc14-f9da834f63aa
  last_name: Bombari
- first_name: Mohammad Hossein
  full_name: Amani, Mohammad Hossein
  last_name: Amani
- first_name: Marco
  full_name: Mondelli, Marco
  id: 27EB676C-8706-11E9-9510-7717E6697425
  last_name: Mondelli
  orcid: 0000-0002-3242-7020
citation:
  ama: 'Bombari S, Amani MH, Mondelli M. Memorization and optimization in deep neural
    networks with minimum over-parameterization. In: <i>36th Conference on Neural
    Information Processing Systems</i>. Vol 35. Neural Information Processing Systems
    Foundation; 2022:7628-7640.'
  apa: 'Bombari, S., Amani, M. H., &#38; Mondelli, M. (2022). Memorization and optimization
    in deep neural networks with minimum over-parameterization. In <i>36th Conference
    on Neural Information Processing Systems</i> (Vol. 35, pp. 7628–7640). New Orleans,
    LA, United States: Neural Information Processing Systems Foundation.'
  chicago: Bombari, Simone, Mohammad Hossein Amani, and Marco Mondelli. “Memorization
    and Optimization in Deep Neural Networks with Minimum Over-Parameterization.”
    In <i>36th Conference on Neural Information Processing Systems</i>, 35:7628–40.
    Neural Information Processing Systems Foundation, 2022.
  ieee: S. Bombari, M. H. Amani, and M. Mondelli, “Memorization and optimization in
    deep neural networks with minimum over-parameterization,” in <i>36th Conference
    on Neural Information Processing Systems</i>, New Orleans, LA, United States,
    2022, vol. 35, pp. 7628–7640.
  ista: 'Bombari S, Amani MH, Mondelli M. 2022. Memorization and optimization in deep
    neural networks with minimum over-parameterization. 36th Conference on Neural
    Information Processing Systems. NeurIPS: Neural Information Processing Systems,
    Advances in Neural Information Processing Systems, vol. 35, 7628–7640.'
  mla: Bombari, Simone, et al. “Memorization and Optimization in Deep Neural Networks
    with Minimum Over-Parameterization.” <i>36th Conference on Neural Information
    Processing Systems</i>, vol. 35, Neural Information Processing Systems Foundation,
    2022, pp. 7628–40.
  short: S. Bombari, M.H. Amani, M. Mondelli, in:, 36th Conference on Neural Information
    Processing Systems, Neural Information Processing Systems Foundation, 2022, pp.
    7628–7640.
conference:
  end_date: 2022-12-09
  location: New Orleans, LA, United States
  name: 'NeurIPS: Neural Information Processing Systems'
  start_date: 2022-11-28
corr_author: '1'
date_created: 2023-02-10T13:46:37Z
date_published: 2022-07-24T00:00:00Z
date_updated: 2025-05-14T11:28:22Z
day: '24'
department:
- _id: MaMo
external_id:
  arxiv:
  - '2205.10217'
intvolume: '        35'
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: ' https://doi.org/10.48550/arXiv.2205.10217'
month: '07'
oa: 1
oa_version: Preprint
page: 7628-7640
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
  name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: 36th Conference on Neural Information Processing Systems
publication_identifier:
  eissn:
  - 1049-5258
  isbn:
  - '9781713871088'
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
status: public
title: Memorization and optimization in deep neural networks with minimum over-parameterization
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2022'
...
---
_id: '14106'
abstract:
- lang: eng
  text: "We show that deep networks trained to satisfy demographic parity often do
    so\r\nthrough a form of race or gender awareness, and that the more we force a
    network\r\nto be fair, the more accurately we can recover race or gender from
    the internal state\r\nof the network. Based on this observation, we investigate
    an alternative fairness\r\napproach: we add a second classification head to the
    network to explicitly predict\r\nthe protected attribute (such as race or gender)
    alongside the original task. After\r\ntraining the two-headed network, we enforce
    demographic parity by merging the\r\ntwo heads, creating a network with the same
    architecture as the original network.\r\nWe establish a close relationship between
    existing approaches and our approach\r\nby showing (1) that the decisions of a
    fair classifier are well-approximated by our\r\napproach, and (2) that an unfair
    and optimally accurate classifier can be recovered\r\nfrom a fair classifier and
    our second head predicting the protected attribute. We use\r\nour explicit formulation
    to argue that the existing fairness approaches, just as ours,\r\ndemonstrate disparate
    treatment and that they are likely to be unlawful in a wide\r\nrange of scenarios
    under US law."
alternative_title:
- Advances in Neural Information Processing Systems
article_processing_charge: No
arxiv: 1
author:
- first_name: Michael
  full_name: Lohaus, Michael
  last_name: Lohaus
- first_name: Matthäus
  full_name: Kleindessner, Matthäus
  last_name: Kleindessner
- first_name: Krishnaram
  full_name: Kenthapadi, Krishnaram
  last_name: Kenthapadi
- first_name: Francesco
  full_name: Locatello, Francesco
  id: 26cfd52f-2483-11ee-8040-88983bcc06d4
  last_name: Locatello
  orcid: 0000-0002-4850-0683
- first_name: Chris
  full_name: Russell, Chris
  last_name: Russell
citation:
  ama: 'Lohaus M, Kleindessner M, Kenthapadi K, Locatello F, Russell C. Are two heads
    the same as one? Identifying disparate treatment in fair neural networks. In:
    <i>36th Conference on Neural Information Processing Systems</i>. Vol 35. Neural
    Information Processing Systems Foundation; 2022:16548-16562.'
  apa: 'Lohaus, M., Kleindessner, M., Kenthapadi, K., Locatello, F., &#38; Russell,
    C. (2022). Are two heads the same as one? Identifying disparate treatment in fair
    neural networks. In <i>36th Conference on Neural Information Processing Systems</i>
    (Vol. 35, pp. 16548–16562). New Orleans, LA, United States: Neural Information
    Processing Systems Foundation.'
  chicago: Lohaus, Michael, Matthäus Kleindessner, Krishnaram Kenthapadi, Francesco
    Locatello, and Chris Russell. “Are Two Heads the Same as One? Identifying Disparate
    Treatment in Fair Neural Networks.” In <i>36th Conference on Neural Information
    Processing Systems</i>, 35:16548–62. Neural Information Processing Systems Foundation,
    2022.
  ieee: M. Lohaus, M. Kleindessner, K. Kenthapadi, F. Locatello, and C. Russell, “Are
    two heads the same as one? Identifying disparate treatment in fair neural networks,”
    in <i>36th Conference on Neural Information Processing Systems</i>, New Orleans,
    LA, United States, 2022, vol. 35, pp. 16548–16562.
  ista: 'Lohaus M, Kleindessner M, Kenthapadi K, Locatello F, Russell C. 2022. Are
    two heads the same as one? Identifying disparate treatment in fair neural networks.
    36th Conference on Neural Information Processing Systems. NeurIPS: Neural Information
    Processing Systems, Advances in Neural Information Processing Systems, vol. 35,
    16548–16562.'
  mla: Lohaus, Michael, et al. “Are Two Heads the Same as One? Identifying Disparate
    Treatment in Fair Neural Networks.” <i>36th Conference on Neural Information Processing
    Systems</i>, vol. 35, Neural Information Processing Systems Foundation, 2022,
    pp. 16548–62.
  short: M. Lohaus, M. Kleindessner, K. Kenthapadi, F. Locatello, C. Russell, in:,
    36th Conference on Neural Information Processing Systems, Neural Information Processing
    Systems Foundation, 2022, pp. 16548–16562.
conference:
  end_date: 2022-12-09
  location: New Orleans, LA, United States
  name: 'NeurIPS: Neural Information Processing Systems'
  start_date: 2022-11-28
date_created: 2023-08-21T12:12:42Z
date_published: 2022-12-15T00:00:00Z
date_updated: 2024-10-14T12:27:01Z
day: '15'
department:
- _id: FrLo
extern: '1'
external_id:
  arxiv:
  - '2204.04440'
intvolume: '        35'
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://arxiv.org/abs/2204.04440
month: '12'
oa: 1
oa_version: Preprint
page: 16548-16562
publication: 36th Conference on Neural Information Processing Systems
publication_identifier:
  isbn:
  - '9781713871088'
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
scopus_import: '1'
status: public
title: Are two heads the same as one? Identifying disparate treatment in fair neural
  networks
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2022'
...
---
_id: '14173'
abstract:
- lang: eng
  text: "Since out-of-distribution generalization is a generally ill-posed problem,
    various proxy targets (e.g., calibration, adversarial robustness, algorithmic
    corruptions, invariance across shifts) were studied across different research
    programs resulting in different recommendations. While sharing the same aspirational
    goal, these approaches have never been tested under the same\r\nexperimental conditions
    on real data. In this paper, we take a unified view of previous work, highlighting
    message discrepancies that we address empirically, and providing recommendations
    on how to measure the robustness of a model and how to improve it. To this end,
    we collect 172 publicly available dataset pairs for training and out-of-distribution
    evaluation of accuracy, calibration error, adversarial attacks, environment invariance,
    and synthetic corruptions. We fine-tune over 31k networks, from nine different
    architectures in the many- and\r\nfew-shot setting. Our findings confirm that
    in- and out-of-distribution accuracies tend to increase jointly, but show that
    their relation is largely dataset-dependent, and in general more nuanced and more
    complex than posited by previous, smaller scale studies."
alternative_title:
- Advances in Neural Information Processing Systems
article_processing_charge: No
arxiv: 1
author:
- first_name: Florian
  full_name: Wenzel, Florian
  last_name: Wenzel
- first_name: Andrea
  full_name: Dittadi, Andrea
  last_name: Dittadi
- first_name: Peter Vincent
  full_name: Gehler, Peter Vincent
  last_name: Gehler
- first_name: Carl-Johann Simon-Gabriel
  full_name: Carl-Johann Simon-Gabriel, Carl-Johann Simon-Gabriel
  last_name: Carl-Johann Simon-Gabriel
- first_name: Max
  full_name: Horn, Max
  last_name: Horn
- first_name: Dominik
  full_name: Zietlow, Dominik
  last_name: Zietlow
- first_name: David
  full_name: Kernert, David
  last_name: Kernert
- first_name: Chris
  full_name: Russell, Chris
  last_name: Russell
- first_name: Thomas
  full_name: Brox, Thomas
  last_name: Brox
- first_name: Bernt
  full_name: Schiele, Bernt
  last_name: Schiele
- first_name: Bernhard
  full_name: Schölkopf, Bernhard
  last_name: Schölkopf
- first_name: Francesco
  full_name: Locatello, Francesco
  id: 26cfd52f-2483-11ee-8040-88983bcc06d4
  last_name: Locatello
  orcid: 0000-0002-4850-0683
citation:
  ama: 'Wenzel F, Dittadi A, Gehler PV, et al. Assaying out-of-distribution generalization
    in transfer learning. In: <i>36th Conference on Neural Information Processing
    Systems</i>. Vol 35. Neural Information Processing Systems Foundation; 2022:7181-7198.'
  apa: 'Wenzel, F., Dittadi, A., Gehler, P. V., Carl-Johann Simon-Gabriel, C.-J. S.-G.,
    Horn, M., Zietlow, D., … Locatello, F. (2022). Assaying out-of-distribution generalization
    in transfer learning. In <i>36th Conference on Neural Information Processing Systems</i>
    (Vol. 35, pp. 7181–7198). New Orleans, LA, United States: Neural Information Processing
    Systems Foundation.'
  chicago: Wenzel, Florian, Andrea Dittadi, Peter Vincent Gehler, Carl-Johann Simon-Gabriel
    Carl-Johann Simon-Gabriel, Max Horn, Dominik Zietlow, David Kernert, et al. “Assaying
    Out-of-Distribution Generalization in Transfer Learning.” In <i>36th Conference
    on Neural Information Processing Systems</i>, 35:7181–98. Neural Information Processing
    Systems Foundation, 2022.
  ieee: F. Wenzel <i>et al.</i>, “Assaying out-of-distribution generalization in transfer
    learning,” in <i>36th Conference on Neural Information Processing Systems</i>,
    New Orleans, LA, United States, 2022, vol. 35, pp. 7181–7198.
  ista: 'Wenzel F, Dittadi A, Gehler PV, Carl-Johann Simon-Gabriel C-JS-G, Horn M,
    Zietlow D, Kernert D, Russell C, Brox T, Schiele B, Schölkopf B, Locatello F.
    2022. Assaying out-of-distribution generalization in transfer learning. 36th Conference
    on Neural Information Processing Systems. NeurIPS: Neural Information Processing
    Systems, Advances in Neural Information Processing Systems, vol. 35, 7181–7198.'
  mla: Wenzel, Florian, et al. “Assaying Out-of-Distribution Generalization in Transfer
    Learning.” <i>36th Conference on Neural Information Processing Systems</i>, vol.
    35, Neural Information Processing Systems Foundation, 2022, pp. 7181–98.
  short: F. Wenzel, A. Dittadi, P.V. Gehler, C.-J.S.-G. Carl-Johann Simon-Gabriel,
    M. Horn, D. Zietlow, D. Kernert, C. Russell, T. Brox, B. Schiele, B. Schölkopf,
    F. Locatello, in:, 36th Conference on Neural Information Processing Systems, Neural
    Information Processing Systems Foundation, 2022, pp. 7181–7198.
conference:
  end_date: 2022-12-09
  location: New Orleans, LA, United States
  name: 'NeurIPS: Neural Information Processing Systems'
  start_date: 2022-11-28
date_created: 2023-08-22T14:01:13Z
date_published: 2022-12-15T00:00:00Z
date_updated: 2023-09-06T10:34:43Z
day: '15'
department:
- _id: FrLo
extern: '1'
external_id:
  arxiv:
  - '2207.09239'
intvolume: '        35'
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://arxiv.org/abs/2207.09239
month: '12'
oa: 1
oa_version: Preprint
page: 7181-7198
publication: 36th Conference on Neural Information Processing Systems
publication_identifier:
  isbn:
  - '9781713871088'
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
scopus_import: '1'
status: public
title: Assaying out-of-distribution generalization in transfer learning
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2022'
...
---
_id: '17086'
abstract:
- lang: eng
  text: 'We consider a high-dimensional mean estimation problem over a binary hidden
    Markov model, which illuminates the interplay between memory in data, sample size,
    dimension, and signal strength in statistical inference. In this model, an estimator
    observes n samples of a d-dimensional parameter vector θ∗∈Rd, multiplied by a
    random sign Si (1≤i≤n), and corrupted by isotropic standard Gaussian noise. The
    sequence of signs {Si}i∈[n]∈{−1,1}n is drawn from a stationary homogeneous Markov
    chain with flip probability δ∈[0,1/2]. As δ varies, this model smoothly interpolates
    two well-studied models: the Gaussian Location Model for which δ=0 and the Gaussian
    Mixture Model for which δ=1/2. Assuming that the estimator knows δ, we establish
    a nearly minimax optimal (up to logarithmic factors) estimation error rate, as
    a function of ∥θ∗∥,δ,d,n. We then provide an upper bound to the case of estimating
    δ, assuming a (possibly inaccurate) knowledge of θ∗. The bound is proved to be
    tight when θ∗ is an accurately known constant. These results are then combined
    to an algorithm which estimates θ∗ with δ unknown a priori, and theoretical guarantees
    on its error are stated.'
acknowledgement: "Part of this work was done when YZ was a postdoc at Technion where
  he received funding from\r\nthe European Union’s Horizon 2020 research and innovation
  programme under grant agreement No 682203-ERC-[Inf-Speed-Tradeoff]. The work of
  of NW was supported in part by the Israel Science Foundation (ISF) under Grant 1782/22.
  NW is grateful to Guy Bresler for introducing him to this problem, for the initial
  ideas that led to this research, and for many helpful discussions on the topic."
alternative_title:
- NeurIPS
article_processing_charge: No
arxiv: 1
author:
- first_name: Yihan
  full_name: Zhang, Yihan
  id: 2ce5da42-b2ea-11eb-bba5-9f264e9d002c
  last_name: Zhang
  orcid: 0000-0002-6465-6258
- first_name: Nir
  full_name: Weinberger, Nir
  last_name: Weinberger
citation:
  ama: 'Zhang Y, Weinberger N. Mean estimation in high-dimensional binary Markov Gaussian
    mixture models. In: <i>36th Conference on Neural Information Processing Systems</i>.
    Vol 35. ML Research Press; 2022.'
  apa: 'Zhang, Y., &#38; Weinberger, N. (2022). Mean estimation in high-dimensional
    binary Markov Gaussian mixture models. In <i>36th Conference on Neural Information
    Processing Systems</i> (Vol. 35). New Orleans, LA, United States: ML Research
    Press.'
  chicago: Zhang, Yihan, and Nir Weinberger. “Mean Estimation in High-Dimensional
    Binary Markov Gaussian Mixture Models.” In <i>36th Conference on Neural Information
    Processing Systems</i>, Vol. 35. ML Research Press, 2022.
  ieee: Y. Zhang and N. Weinberger, “Mean estimation in high-dimensional binary Markov
    Gaussian mixture models,” in <i>36th Conference on Neural Information Processing
    Systems</i>, New Orleans, LA, United States, 2022, vol. 35.
  ista: 'Zhang Y, Weinberger N. 2022. Mean estimation in high-dimensional binary Markov
    Gaussian mixture models. 36th Conference on Neural Information Processing Systems.
    NeurIPS: Neural Information Processing Systems, NeurIPS, vol. 35.'
  mla: Zhang, Yihan, and Nir Weinberger. “Mean Estimation in High-Dimensional Binary
    Markov Gaussian Mixture Models.” <i>36th Conference on Neural Information Processing
    Systems</i>, vol. 35, ML Research Press, 2022.
  short: Y. Zhang, N. Weinberger, in:, 36th Conference on Neural Information Processing
    Systems, ML Research Press, 2022.
conference:
  end_date: 2022-12-09
  location: New Orleans, LA, United States
  name: 'NeurIPS: Neural Information Processing Systems'
  start_date: 2022-11-28
corr_author: '1'
date_created: 2024-05-29T06:37:16Z
date_published: 2022-12-01T00:00:00Z
date_updated: 2024-08-05T09:48:58Z
day: '01'
ddc:
- '000'
department:
- _id: MaMo
external_id:
  arxiv:
  - '2206.02455'
file:
- access_level: open_access
  checksum: 05f6f9f8fc34e224e0cad045b9489030
  content_type: application/pdf
  creator: dernst
  date_created: 2024-08-05T09:44:49Z
  date_updated: 2024-08-05T09:44:49Z
  file_id: '17392'
  file_name: 2022_NeurIPS_Zhang.pdf
  file_size: 476307
  relation: main_file
  success: 1
file_date_updated: 2024-08-05T09:44:49Z
has_accepted_license: '1'
intvolume: '        35'
language:
- iso: eng
month: '12'
oa: 1
oa_version: Published Version
publication: 36th Conference on Neural Information Processing Systems
publication_identifier:
  isbn:
  - '9781713871088'
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
scopus_import: '1'
status: public
title: Mean estimation in high-dimensional binary Markov Gaussian mixture models
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2022'
...
---
_id: '17087'
abstract:
- lang: eng
  text: We consider the problem of model compression for deep neural networks (DNNs)
    in the challenging one-shot/post-training setting, in which we are given an accurate
    trained model, and must compress it without any retraining, based only on a small
    amount of calibration input data. This problem has become popular in view of the
    emerging software and hardware support for executing models compressed via pruning
    and/or quantization with speedup, and well-performing solutions have been proposed
    independently for both compression approaches.In this paper, we introduce a new
    compression framework which covers both weight pruning and quantization in a unified
    setting, is time- and space-efficient, and considerably improves upon the practical
    performance of existing post-training methods. At the technical level, our approach
    is based on an exact and efficient realization of the classical Optimal Brain
    Surgeon (OBS) framework of [LeCun, Denker, and Solla, 1990] extended to also cover
    weight quantization at the scale of modern DNNs. From the practical perspective,
    our experimental results show that it can improve significantly upon the compression-accuracy
    trade-offs of existing post-training methods, and that it can enable the accurate
    compound application of both pruning and quantization in a post-training setting.
acknowledgement: 'We gratefully acknowledge funding from the European Research Council
  (ERC) under the European Union’s Horizon 2020 programme (grant agreement No 805223
  ScaleML), as well as computational support from AWS EC2. We thank Eldar Kurtic for
  providing us BERT code and pretrained models, and the Neural Magic Team, notably
  Michael Goin and Mark Kurtz, for support with their software. '
alternative_title:
- NeurIPS
article_processing_charge: No
arxiv: 1
author:
- first_name: Elias
  full_name: Frantar, Elias
  id: 09a8f98d-ec99-11ea-ae11-c063a7b7fe5f
  last_name: Frantar
- first_name: Sidak Pal
  full_name: Singh, Sidak Pal
  id: DD138E24-D89D-11E9-9DC0-DEF6E5697425
  last_name: Singh
- first_name: Dan-Adrian
  full_name: Alistarh, Dan-Adrian
  id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
  last_name: Alistarh
  orcid: 0000-0003-3650-940X
citation:
  ama: 'Frantar E, Singh SP, Alistarh D-A. Optimal brain compression: A framework
    for accurate post-training quantization and pruning. In: <i>36th Conference on
    Neural Information Processing Systems</i>. Vol 35. ML Research Press; 2022.'
  apa: 'Frantar, E., Singh, S. P., &#38; Alistarh, D.-A. (2022). Optimal brain compression:
    A framework for accurate post-training quantization and pruning. In <i>36th Conference
    on Neural Information Processing Systems</i> (Vol. 35). New Orleans, LA, United
    States: ML Research Press.'
  chicago: 'Frantar, Elias, Sidak Pal Singh, and Dan-Adrian Alistarh. “Optimal Brain
    Compression: A Framework for Accurate Post-Training Quantization and Pruning.”
    In <i>36th Conference on Neural Information Processing Systems</i>, Vol. 35. ML
    Research Press, 2022.'
  ieee: 'E. Frantar, S. P. Singh, and D.-A. Alistarh, “Optimal brain compression:
    A framework for accurate post-training quantization and pruning,” in <i>36th Conference
    on Neural Information Processing Systems</i>, New Orleans, LA, United States,
    2022, vol. 35.'
  ista: 'Frantar E, Singh SP, Alistarh D-A. 2022. Optimal brain compression: A framework
    for accurate post-training quantization and pruning. 36th Conference on Neural
    Information Processing Systems. NeurIPS: Neural Information Processing Systems,
    NeurIPS, vol. 35.'
  mla: 'Frantar, Elias, et al. “Optimal Brain Compression: A Framework for Accurate
    Post-Training Quantization and Pruning.” <i>36th Conference on Neural Information
    Processing Systems</i>, vol. 35, ML Research Press, 2022.'
  short: E. Frantar, S.P. Singh, D.-A. Alistarh, in:, 36th Conference on Neural Information
    Processing Systems, ML Research Press, 2022.
conference:
  end_date: 2022-12-09
  location: New Orleans, LA, United States
  name: 'NeurIPS: Neural Information Processing Systems'
  start_date: 2022-11-28
corr_author: '1'
date_created: 2024-05-29T06:38:26Z
date_published: 2022-12-01T00:00:00Z
date_updated: 2026-04-07T12:43:03Z
day: '01'
ddc:
- '000'
department:
- _id: DaAl
ec_funded: 1
external_id:
  arxiv:
  - '2208.11580'
file:
- access_level: open_access
  checksum: 38e7d75f578e8d2e207c81895e09f211
  content_type: application/pdf
  creator: dernst
  date_created: 2024-08-05T09:25:39Z
  date_updated: 2024-08-05T09:25:39Z
  file_id: '17391'
  file_name: 2022_NeurIPS_Frantar.pdf
  file_size: 491843
  relation: main_file
  success: 1
file_date_updated: 2024-08-05T09:25:39Z
has_accepted_license: '1'
intvolume: '        35'
language:
- iso: eng
month: '12'
oa: 1
oa_version: Submitted Version
project:
- _id: 268A44D6-B435-11E9-9278-68D0E5697425
  call_identifier: H2020
  grant_number: '805223'
  name: Elastic Coordination for Scalable Machine Learning
publication: 36th Conference on Neural Information Processing Systems
publication_identifier:
  isbn:
  - '9781713871088'
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
related_material:
  record:
  - id: '17485'
    relation: dissertation_contains
    status: public
scopus_import: '1'
status: public
title: 'Optimal brain compression: A framework for accurate post-training quantization
  and pruning'
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2022'
...