---
_id: '14924'
abstract:
- lang: eng
text: "The stochastic heavy ball method (SHB), also known as stochastic gradient
descent (SGD) with Polyak's momentum, is widely used in training neural networks.
However, despite the remarkable success of such algorithm in practice, its theoretical
characterization remains limited. In this paper, we focus on neural networks with
two and three layers and provide a rigorous understanding of the properties of
the solutions found by SHB: \\emph{(i)} stability after dropping out part of the
neurons, \\emph{(ii)} connectivity along a low-loss path, and \\emph{(iii)} convergence
to the global optimum.\r\nTo achieve this goal, we take a mean-field view and
relate the SHB dynamics to a certain partial differential equation in the limit
of large network widths. This mean-field perspective has inspired a recent line
of work focusing on SGD while, in contrast, our paper considers an algorithm with
momentum. More specifically, after proving existence and uniqueness of the limit
differential equations, we show convergence to the global optimum and give a quantitative
bound between the mean-field limit and the SHB dynamics of a finite-width network.
Armed with this last bound, we are able to establish the dropout-stability and
connectivity of SHB solutions."
acknowledgement: D. Wu and M. Mondelli are partially supported by the 2019 Lopez-Loreta
Prize. V. Kungurtsev was supported by the OP VVV project CZ.02.1.01/0.0/0.0/16_019/0000765
"Research Center for Informatics".
alternative_title:
- TMLR
article_processing_charge: No
arxiv: 1
author:
- first_name: Diyuan
full_name: Wu, Diyuan
id: 1a5914c2-896a-11ed-bdf8-fb80621a0635
last_name: Wu
- first_name: Vyacheslav
full_name: Kungurtsev, Vyacheslav
last_name: Kungurtsev
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
citation:
ama: 'Wu D, Kungurtsev V, Mondelli M. Mean-field analysis for heavy ball methods:
Dropout-stability, connectivity, and global convergence. In: Transactions on
Machine Learning Research. ML Research Press; 2023.'
apa: 'Wu, D., Kungurtsev, V., & Mondelli, M. (2023). Mean-field analysis for
heavy ball methods: Dropout-stability, connectivity, and global convergence. In
Transactions on Machine Learning Research. ML Research Press.'
chicago: 'Wu, Diyuan, Vyacheslav Kungurtsev, and Marco Mondelli. “Mean-Field Analysis
for Heavy Ball Methods: Dropout-Stability, Connectivity, and Global Convergence.”
In Transactions on Machine Learning Research. ML Research Press, 2023.'
ieee: 'D. Wu, V. Kungurtsev, and M. Mondelli, “Mean-field analysis for heavy ball
methods: Dropout-stability, connectivity, and global convergence,” in Transactions
on Machine Learning Research, 2023.'
ista: 'Wu D, Kungurtsev V, Mondelli M. 2023. Mean-field analysis for heavy ball
methods: Dropout-stability, connectivity, and global convergence. Transactions
on Machine Learning Research. , TMLR, .'
mla: 'Wu, Diyuan, et al. “Mean-Field Analysis for Heavy Ball Methods: Dropout-Stability,
Connectivity, and Global Convergence.” Transactions on Machine Learning Research,
ML Research Press, 2023.'
short: D. Wu, V. Kungurtsev, M. Mondelli, in:, Transactions on Machine Learning
Research, ML Research Press, 2023.
corr_author: '1'
date_created: 2024-02-02T11:21:56Z
date_published: 2023-02-28T00:00:00Z
date_updated: 2026-06-18T17:41:36Z
day: '28'
ddc:
- '000'
department:
- _id: MaMo
external_id:
arxiv:
- '2210.06819'
has_accepted_license: '1'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.48550/arXiv.2210.06819
month: '02'
oa: 1
oa_version: Published Version
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: Transactions on Machine Learning Research
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
status: public
title: 'Mean-field analysis for heavy ball methods: Dropout-stability, connectivity,
and global convergence'
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2023'
...
---
_id: '14921'
abstract:
- lang: eng
text: Neural collapse (NC) refers to the surprising structure of the last layer
of deep neural networks in the terminal phase of gradient descent training. Recently,
an increasing amount of experimental evidence has pointed to the propagation of
NC to earlier layers of neural networks. However, while the NC in the last layer
is well studied theoretically, much less is known about its multi-layered counterpart
- deep neural collapse (DNC). In particular, existing work focuses either on linear
layers or only on the last two layers at the price of an extra assumption. Our
paper fills this gap by generalizing the established analytical framework for
NC - the unconstrained features model - to multiple non-linear layers. Our key
technical contribution is to show that, in a deep unconstrained features model,
the unique global optimum for binary classification exhibits all the properties
typical of DNC. This explains the existing experimental evidence of DNC. We also
empirically show that (i) by optimizing deep unconstrained features models via
gradient descent, the resulting solution agrees well with our theory, and (ii)
trained networks recover the unconstrained features suitable for the occurrence
of DNC, thus supporting the validity of this modeling principle.
acknowledgement: M. M. is partially supported by the 2019 Lopez-Loreta Prize. The
authors would like to thank Eugenia Iofinova, Bernd Prach and Simone Bombari for
valuable feedback on the manuscript.
alternative_title:
- Advances in Neural Information Processing Systems
article_processing_charge: No
arxiv: 1
author:
- first_name: Peter
full_name: Súkeník, Peter
id: d64d6a8d-eb8e-11eb-b029-96fd216dec3c
last_name: Súkeník
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: Christoph
full_name: Lampert, Christoph
id: 40C20FD2-F248-11E8-B48F-1D18A9856A87
last_name: Lampert
orcid: 0000-0001-8622-7887
citation:
ama: 'Súkeník P, Mondelli M, Lampert C. Deep neural collapse is provably optimal
for the deep unconstrained features model. In: 37th Conference on Neural Information
Processing Systems. Neural Information Processing Systems Foundation; 2023.'
apa: 'Súkeník, P., Mondelli, M., & Lampert, C. (2023). Deep neural collapse
is provably optimal for the deep unconstrained features model. In 37th Conference
on Neural Information Processing Systems. New Orleans, LA, United States:
Neural Information Processing Systems Foundation.'
chicago: Súkeník, Peter, Marco Mondelli, and Christoph Lampert. “Deep Neural Collapse
Is Provably Optimal for the Deep Unconstrained Features Model.” In 37th Conference
on Neural Information Processing Systems. Neural Information Processing Systems
Foundation, 2023.
ieee: P. Súkeník, M. Mondelli, and C. Lampert, “Deep neural collapse is provably
optimal for the deep unconstrained features model,” in 37th Conference on Neural
Information Processing Systems, New Orleans, LA, United States, 2023.
ista: 'Súkeník P, Mondelli M, Lampert C. 2023. Deep neural collapse is provably
optimal for the deep unconstrained features model. 37th Conference on Neural Information
Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in
Neural Information Processing Systems, .'
mla: Súkeník, Peter, et al. “Deep Neural Collapse Is Provably Optimal for the Deep
Unconstrained Features Model.” 37th Conference on Neural Information Processing
Systems, Neural Information Processing Systems Foundation, 2023.
short: P. Súkeník, M. Mondelli, C. Lampert, in:, 37th Conference on Neural Information
Processing Systems, Neural Information Processing Systems Foundation, 2023.
conference:
end_date: 2023-12-16
location: New Orleans, LA, United States
name: 'NeurIPS: Neural Information Processing Systems'
start_date: 2023-12-10
corr_author: '1'
das_tickbox: '1'
date_created: 2024-02-02T11:17:41Z
date_published: 2023-12-15T00:00:00Z
date_updated: 2026-07-07T06:35:57Z
day: '15'
department:
- _id: MaMo
- _id: ChLa
external_id:
arxiv:
- '2305.13165'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: ' https://doi.org/10.48550/arXiv.2305.13165'
month: '12'
oa: 1
oa_version: Preprint
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: 37th Conference on Neural Information Processing Systems
publication_identifier:
eissn:
- 1049-5258
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
status: public
title: Deep neural collapse is provably optimal for the deep unconstrained features
model
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2023'
...
---
_id: '13321'
abstract:
- lang: eng
text: We consider the problem of reconstructing the signal and the hidden variables
from observations coming from a multi-layer network with rotationally invariant
weight matrices. The multi-layer structure models inference from deep generative
priors, and the rotational invariance imposed on the weights generalizes the i.i.d.
Gaussian assumption by allowing for a complex correlation structure, which is
typical in applications. In this work, we present a new class of approximate message
passing (AMP) algorithms and give a state evolution recursion which precisely
characterizes their performance in the large system limit. In contrast with the
existing multi-layer VAMP (ML-VAMP) approach, our proposed AMP – dubbed multilayer
rotationally invariant generalized AMP (ML-RI-GAMP) – provides a natural generalization
beyond Gaussian designs, in the sense that it recovers the existing Gaussian AMP
as a special case. Furthermore, ML-RI-GAMP exhibits a significantly lower complexity
than ML-VAMP, as the computationally intensive singular value decomposition is
replaced by an estimation of the moments of the design matrices. Finally, our
numerical results show that this complexity gain comes at little to no cost in
the performance of the algorithm.
acknowledgement: Marco Mondelli was partially supported by the 2019 Lopez-Loreta prize.
article_processing_charge: No
arxiv: 1
author:
- first_name: Yizhou
full_name: Xu, Yizhou
last_name: Xu
- first_name: Tian Qi
full_name: Hou, Tian Qi
last_name: Hou
- first_name: Shan Suo
full_name: Liang, Shan Suo
last_name: Liang
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
citation:
ama: 'Xu Y, Hou TQ, Liang SS, Mondelli M. Approximate message passing for multi-layer
estimation in rotationally invariant models. In: 2023 IEEE Information Theory
Workshop. IEEE; 2023:294-298. doi:10.1109/ITW55543.2023.10160238'
apa: 'Xu, Y., Hou, T. Q., Liang, S. S., & Mondelli, M. (2023). Approximate message
passing for multi-layer estimation in rotationally invariant models. In 2023
IEEE Information Theory Workshop (pp. 294–298). Saint-Malo, France: IEEE.
https://doi.org/10.1109/ITW55543.2023.10160238'
chicago: Xu, Yizhou, Tian Qi Hou, Shan Suo Liang, and Marco Mondelli. “Approximate
Message Passing for Multi-Layer Estimation in Rotationally Invariant Models.”
In 2023 IEEE Information Theory Workshop, 294–98. IEEE, 2023. https://doi.org/10.1109/ITW55543.2023.10160238.
ieee: Y. Xu, T. Q. Hou, S. S. Liang, and M. Mondelli, “Approximate message passing
for multi-layer estimation in rotationally invariant models,” in 2023 IEEE
Information Theory Workshop, Saint-Malo, France, 2023, pp. 294–298.
ista: 'Xu Y, Hou TQ, Liang SS, Mondelli M. 2023. Approximate message passing for
multi-layer estimation in rotationally invariant models. 2023 IEEE Information
Theory Workshop. ITW: Information Theory Workshop, 294–298.'
mla: Xu, Yizhou, et al. “Approximate Message Passing for Multi-Layer Estimation
in Rotationally Invariant Models.” 2023 IEEE Information Theory Workshop,
IEEE, 2023, pp. 294–98, doi:10.1109/ITW55543.2023.10160238.
short: Y. Xu, T.Q. Hou, S.S. Liang, M. Mondelli, in:, 2023 IEEE Information Theory
Workshop, IEEE, 2023, pp. 294–298.
conference:
end_date: 2023-04-28
location: Saint-Malo, France
name: 'ITW: Information Theory Workshop'
start_date: 2023-04-23
corr_author: '1'
date_created: 2023-07-30T22:01:04Z
date_published: 2023-05-01T00:00:00Z
date_updated: 2026-08-12T06:38:51Z
day: '01'
department:
- _id: MaMo
doi: 10.1109/ITW55543.2023.10160238
external_id:
arxiv:
- '2212.01572'
isi:
- '001031733100053'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.48550/arXiv.2212.01572
month: '05'
oa: 1
oa_version: Preprint
page: 294-298
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: 2023 IEEE Information Theory Workshop
publication_identifier:
eissn:
- 2475-4218
isbn:
- '9798350301496'
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: Approximate message passing for multi-layer estimation in rotationally invariant
models
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2023'
...
---
_id: '14751'
abstract:
- lang: eng
text: 'We consider zero-error communication over a two-transmitter deterministic
adversarial multiple access channel (MAC) governed by an adversary who has access
to the transmissions of both senders (hence called omniscient ) and aims to maliciously
corrupt the communication. None of the encoders, jammer and decoder is allowed
to randomize using private or public randomness. This enforces a combinatorial
nature of the problem. Our model covers a large family of channels studied in
the literature, including all deterministic discrete memoryless noisy or noiseless
MACs. In this work, given an arbitrary two-transmitter deterministic omniscient
adversarial MAC, we characterize when the capacity region: 1) has nonempty interior
(in particular, is two-dimensional); 2) consists of two line segments (in particular,
has empty interior); 3) consists of one line segment (in particular, is one-dimensional);
4) or only contains (0,0) (in particular, is zero-dimensional). This extends a
recent result by Wang et al. (201 9) from the point-to-point setting to the multiple
access setting. Indeed, our converse arguments build upon their generalized Plotkin
bound and involve delicate case analysis. One of the technical challenges is to
take care of both “joint confusability” and “marginal confusability”. In particular,
the treatment of marginal confusability does not follow from the point-to-point
results by Wang et al. Our achievability results follow from random coding with
expurgation.'
acknowledgement: "The author would like to thank Amitalok J. Budkuley and Sidharth
Jaggi for many helpful discussions at the early stage of this work. He would also
like to thank Nir Ailon, Qi Cao, and Chandra Nair for discussions on a related problem
regarding zero-error binary adder MACs.\r\nThe work of Yihan Zhang was supported
by the European Union’s Horizon 2020 Research and Innovation Programme under Grant
682203-ERC-[Inf-Speed-Tradeoff]"
article_processing_charge: No
article_type: original
arxiv: 1
author:
- first_name: Yihan
full_name: Zhang, Yihan
id: 2ce5da42-b2ea-11eb-bba5-9f264e9d002c
last_name: Zhang
orcid: 0000-0002-6465-6258
citation:
ama: Zhang Y. Zero-error communication over adversarial MACs. IEEE Transactions
on Information Theory. 2023;69(7):4093-4127. doi:10.1109/tit.2023.3257239
apa: Zhang, Y. (2023). Zero-error communication over adversarial MACs. IEEE Transactions
on Information Theory. IEEE. https://doi.org/10.1109/tit.2023.3257239
chicago: Zhang, Yihan. “Zero-Error Communication over Adversarial MACs.” IEEE
Transactions on Information Theory. IEEE, 2023. https://doi.org/10.1109/tit.2023.3257239.
ieee: Y. Zhang, “Zero-error communication over adversarial MACs,” IEEE Transactions
on Information Theory, vol. 69, no. 7. IEEE, pp. 4093–4127, 2023.
ista: Zhang Y. 2023. Zero-error communication over adversarial MACs. IEEE Transactions
on Information Theory. 69(7), 4093–4127.
mla: Zhang, Yihan. “Zero-Error Communication over Adversarial MACs.” IEEE Transactions
on Information Theory, vol. 69, no. 7, IEEE, 2023, pp. 4093–127, doi:10.1109/tit.2023.3257239.
short: Y. Zhang, IEEE Transactions on Information Theory 69 (2023) 4093–4127.
corr_author: '1'
date_created: 2024-01-08T13:04:54Z
date_published: 2023-07-01T00:00:00Z
date_updated: 2026-08-12T06:41:37Z
day: '01'
department:
- _id: MaMo
doi: 10.1109/tit.2023.3257239
external_id:
arxiv:
- '2101.12426'
isi:
- '001017307000001'
intvolume: ' 69'
isi: 1
issue: '7'
keyword:
- Computer Science Applications
- Information Systems
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.48550/arXiv.2101.12426
month: '07'
oa: 1
oa_version: Preprint
page: 4093-4127
publication: IEEE Transactions on Information Theory
publication_identifier:
eissn:
- 1557-9654
issn:
- 0018-9448
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: Zero-error communication over adversarial MACs
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 69
year: '2023'
...
---
_id: '13269'
abstract:
- lang: eng
text: This paper is a collection of results on combinatorial properties of codes
for the Z-channel . A Z-channel with error fraction τ takes as input a length-
n binary codeword and injects in an adversarial manner up to n τ asymmetric errors,
i.e., errors that only zero out bits but do not flip 0’s to 1’s. It is known that
the largest ( L - 1)-list-decodable code for the Z-channel with error fraction
τ has exponential size (in n ) if τ is less than a critical value that we call
the ( L - 1)- list-decoding Plotkin point and has constant size if τ is larger
than the threshold. The ( L -1)-list-decoding Plotkin point is known to be L -1/L-1
– L -L/ L-1 , which equals 1/4 for unique-decoding with L -1 = 1. In this paper,
we derive various results for the size of the largest codes above and below the
list-decoding Plotkin point. In particular, we show that the largest ( L -1)-list-decodable
code ε-above the Plotkin point, for any given sufficiently small positive constant
ε > 0, has size Θ L (ε -3/2 ) for any L - 1 ≥ 1. We also devise upper and lower
bounds on the exponential size of codes below the list-decoding Plotkin point.
acknowledgement: "Nikita Polyanskii’s research was conducted in part during October
2020 - December 2021 with the Technical University of Munich and the Skolkovo Institute
of Science and Technology. His work was supported by the German Research Foundation
(Deutsche Forschungsgemeinschaft, DFG) under Grant No. WA3907/1-1 and the Russian
Foundation for Basic Research (RFBR)\r\nunder Grant No. 20-01-00559.\r\nYihan Zhang
is supported by funding from the European Union’s Horizon 2020 research and innovation
programme under grant agreement No 682203-ERC-[Inf-Speed-Tradeoff]."
article_processing_charge: No
article_type: original
arxiv: 1
author:
- first_name: Nikita
full_name: Polyanskii, Nikita
last_name: Polyanskii
- first_name: Yihan
full_name: Zhang, Yihan
id: 2ce5da42-b2ea-11eb-bba5-9f264e9d002c
last_name: Zhang
orcid: 0000-0002-6465-6258
citation:
ama: Polyanskii N, Zhang Y. Codes for the Z-channel. IEEE Transactions on Information
Theory. 2023;69(10):6340-6357. doi:10.1109/TIT.2023.3292219
apa: Polyanskii, N., & Zhang, Y. (2023). Codes for the Z-channel. IEEE Transactions
on Information Theory. IEEE. https://doi.org/10.1109/TIT.2023.3292219
chicago: Polyanskii, Nikita, and Yihan Zhang. “Codes for the Z-Channel.” IEEE
Transactions on Information Theory. IEEE, 2023. https://doi.org/10.1109/TIT.2023.3292219.
ieee: N. Polyanskii and Y. Zhang, “Codes for the Z-channel,” IEEE Transactions
on Information Theory, vol. 69, no. 10. IEEE, pp. 6340–6357, 2023.
ista: Polyanskii N, Zhang Y. 2023. Codes for the Z-channel. IEEE Transactions on
Information Theory. 69(10), 6340–6357.
mla: Polyanskii, Nikita, and Yihan Zhang. “Codes for the Z-Channel.” IEEE Transactions
on Information Theory, vol. 69, no. 10, IEEE, 2023, pp. 6340–57, doi:10.1109/TIT.2023.3292219.
short: N. Polyanskii, Y. Zhang, IEEE Transactions on Information Theory 69 (2023)
6340–6357.
corr_author: '1'
date_created: 2023-07-23T22:01:14Z
date_published: 2023-07-04T00:00:00Z
date_updated: 2026-08-12T06:41:54Z
day: '04'
department:
- _id: MaMo
doi: 10.1109/TIT.2023.3292219
external_id:
arxiv:
- '2105.01427'
isi:
- '001069680100011'
intvolume: ' 69'
isi: 1
issue: '10'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.48550/arXiv.2105.01427
month: '07'
oa: 1
oa_version: Preprint
page: 6340-6357
publication: IEEE Transactions on Information Theory
publication_identifier:
eissn:
- 1557-9654
issn:
- 0018-9448
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: Codes for the Z-channel
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 69
year: '2023'
...
---
_id: '14459'
abstract:
- lang: eng
text: Autoencoders are a popular model in many branches of machine learning and
lossy data compression. However, their fundamental limits, the performance of
gradient methods and the features learnt during optimization remain poorly understood,
even in the two-layer setting. In fact, earlier work has considered either linear
autoencoders or specific training regimes (leading to vanishing or diverging compression
rates). Our paper addresses this gap by focusing on non-linear two-layer autoencoders
trained in the challenging proportional regime in which the input dimension scales
linearly with the size of the representation. Our results characterize the minimizers
of the population risk, and show that such minimizers are achieved by gradient
methods; their structure is also unveiled, thus leading to a concise description
of the features obtained via training. For the special case of a sign activation
function, our analysis establishes the fundamental limits for the lossy compression
of Gaussian sources via (shallow) autoencoders. Finally, while the results are
proved for Gaussian data, numerical simulations on standard datasets display the
universality of the theoretical predictions.
acknowledgement: Aleksandr Shevchenko, Kevin Kogler and Marco Mondelli are supported
by the 2019 Lopez-Loreta Prize. Hamed Hassani acknowledges the support by the NSF
CIF award (1910056) and the NSF Institute for CORE Emerging Methods in Data Science
(EnCORE).
alternative_title:
- PMLR
article_processing_charge: No
arxiv: 1
author:
- first_name: Aleksandr
full_name: Shevchenko, Aleksandr
id: F2B06EC2-C99E-11E9-89F0-752EE6697425
last_name: Shevchenko
- first_name: Kevin
full_name: Kögler, Kevin
id: 94ec913c-dc85-11ea-9058-e5051ab2428b
last_name: Kögler
- first_name: Hamed
full_name: Hassani, Hamed
last_name: Hassani
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
citation:
ama: 'Shevchenko A, Kögler K, Hassani H, Mondelli M. Fundamental limits of two-layer
autoencoders, and achieving them with gradient methods. In: Proceedings of
the 40th International Conference on Machine Learning. Vol 202. ML Research
Press; 2023:31151-31209.'
apa: 'Shevchenko, A., Kögler, K., Hassani, H., & Mondelli, M. (2023). Fundamental
limits of two-layer autoencoders, and achieving them with gradient methods. In
Proceedings of the 40th International Conference on Machine Learning (Vol.
202, pp. 31151–31209). Honolulu, Hawaii, HI, United States: ML Research Press.'
chicago: Shevchenko, Alexander, Kevin Kögler, Hamed Hassani, and Marco Mondelli.
“Fundamental Limits of Two-Layer Autoencoders, and Achieving Them with Gradient
Methods.” In Proceedings of the 40th International Conference on Machine Learning,
202:31151–209. ML Research Press, 2023.
ieee: A. Shevchenko, K. Kögler, H. Hassani, and M. Mondelli, “Fundamental limits
of two-layer autoencoders, and achieving them with gradient methods,” in Proceedings
of the 40th International Conference on Machine Learning, Honolulu, Hawaii,
HI, United States, 2023, vol. 202, pp. 31151–31209.
ista: 'Shevchenko A, Kögler K, Hassani H, Mondelli M. 2023. Fundamental limits of
two-layer autoencoders, and achieving them with gradient methods. Proceedings
of the 40th International Conference on Machine Learning. ICML: International
Conference on Machine Learning, PMLR, vol. 202, 31151–31209.'
mla: Shevchenko, Alexander, et al. “Fundamental Limits of Two-Layer Autoencoders,
and Achieving Them with Gradient Methods.” Proceedings of the 40th International
Conference on Machine Learning, vol. 202, ML Research Press, 2023, pp. 31151–209.
short: A. Shevchenko, K. Kögler, H. Hassani, M. Mondelli, in:, Proceedings of the
40th International Conference on Machine Learning, ML Research Press, 2023, pp.
31151–31209.
conference:
end_date: 2023-07-29
location: Honolulu, Hawaii, HI, United States
name: 'ICML: International Conference on Machine Learning'
start_date: 2023-07-23
corr_author: '1'
date_created: 2023-10-29T23:01:17Z
date_published: 2023-07-30T00:00:00Z
date_updated: 2026-08-26T22:30:15Z
day: '30'
department:
- _id: MaMo
- _id: DaAl
external_id:
arxiv:
- '2212.13468'
intvolume: ' 202'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.48550/arXiv.2212.13468
month: '07'
oa: 1
oa_version: Preprint
page: 31151-31209
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: Proceedings of the 40th International Conference on Machine Learning
publication_identifier:
eissn:
- 2640-3498
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
related_material:
record:
- id: '17465'
relation: dissertation_contains
status: public
scopus_import: '1'
status: public
title: Fundamental limits of two-layer autoencoders, and achieving them with gradient
methods
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 202
year: '2023'
...
---
_id: '11639'
abstract:
- lang: eng
text: We study the list decodability of different ensembles of codes over the real
alphabet under the assumption of an omniscient adversary. It is a well-known result
that when the source and the adversary have power constraints P and N respectively,
the list decoding capacity is equal to 1/2logP/N. Random spherical codes achieve
constant list sizes, and the goal of the present paper is to obtain a better understanding
of the smallest achievable list size as a function of the gap to capacity. We
show a reduction from arbitrary codes to spherical codes, and derive a lower bound
on the list size of typical random spherical codes. We also give an upper bound
on the list size achievable using nested Construction-A lattices and infinite
Construction-A lattices. We then define and study a class of infinite constellations
that generalize Construction-A lattices and prove upper and lower bounds for the
same. Other goodness properties such as packing goodness and AWGN goodness of
infinite constellations are proved along the way. Finally, we consider random
lattices sampled from the Haar distribution and show that if a certain conjecture
that originates in analytic number theory is true, then the list size grows as
a polynomial function of the gap-to-capacity.
acknowledgement: "This work was done when Shashank Vatedka was at the Chinese University
of Hong Kong, where he was supported in part by CUHK Direct Grants 4055039 and 4055077.
He would like to acknowledge funding from a seed grant offered by IIT Hyderabad
and the Start-up Research Grant (SRG/2020/000910) from the Science and Engineering
Board, India. Yihan Zhang has received funding from the European Union’s Horizon
2020 research and innovation programme\r\nunder grant agreement No 682203-ERC-[Inf-Speed-Tradeoff]."
article_processing_charge: No
article_type: original
arxiv: 1
author:
- first_name: Yihan
full_name: Zhang, Yihan
id: 2ce5da42-b2ea-11eb-bba5-9f264e9d002c
last_name: Zhang
orcid: 0000-0002-6465-6258
- first_name: Shashank
full_name: Vatedka, Shashank
last_name: Vatedka
citation:
ama: Zhang Y, Vatedka S. List decoding random Euclidean codes and Infinite constellations.
IEEE Transactions on Information Theory. 2022;68(12):7753-7786. doi:10.1109/TIT.2022.3189542
apa: Zhang, Y., & Vatedka, S. (2022). List decoding random Euclidean codes and
Infinite constellations. IEEE Transactions on Information Theory. IEEE.
https://doi.org/10.1109/TIT.2022.3189542
chicago: Zhang, Yihan, and Shashank Vatedka. “List Decoding Random Euclidean Codes
and Infinite Constellations.” IEEE Transactions on Information Theory.
IEEE, 2022. https://doi.org/10.1109/TIT.2022.3189542.
ieee: Y. Zhang and S. Vatedka, “List decoding random Euclidean codes and Infinite
constellations,” IEEE Transactions on Information Theory, vol. 68, no.
12. IEEE, pp. 7753–7786, 2022.
ista: Zhang Y, Vatedka S. 2022. List decoding random Euclidean codes and Infinite
constellations. IEEE Transactions on Information Theory. 68(12), 7753–7786.
mla: Zhang, Yihan, and Shashank Vatedka. “List Decoding Random Euclidean Codes and
Infinite Constellations.” IEEE Transactions on Information Theory, vol.
68, no. 12, IEEE, 2022, pp. 7753–86, doi:10.1109/TIT.2022.3189542.
short: Y. Zhang, S. Vatedka, IEEE Transactions on Information Theory 68 (2022) 7753–7786.
corr_author: '1'
date_created: 2022-07-24T22:01:42Z
date_published: 2022-12-01T00:00:00Z
date_updated: 2024-10-09T21:02:55Z
day: '01'
department:
- _id: MaMo
doi: 10.1109/TIT.2022.3189542
external_id:
arxiv:
- '1901.03790'
isi:
- '000891796100007'
intvolume: ' 68'
isi: 1
issue: '12'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.48550/arXiv.1901.03790
month: '12'
oa: 1
oa_version: Preprint
page: 7753-7786
publication: IEEE Transactions on Information Theory
publication_identifier:
eissn:
- 1557-9654
issn:
- 0018-9448
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: List decoding random Euclidean codes and Infinite constellations
type: journal_article
user_id: 4359f0d1-fa6c-11eb-b949-802e58b17ae8
volume: 68
year: '2022'
...
---
_id: '12011'
abstract:
- lang: eng
text: We characterize the capacity for the discrete-time arbitrarily varying channel
with discrete inputs, outputs, and states when (a) the encoder and decoder do
not share common randomness, (b) the input and state are subject to cost constraints,
(c) the transition matrix of the channel is deterministic given the state, and
(d) at each time step the adversary can only observe the current and past channel
inputs when choosing the state at that time. The achievable strategy involves
stochastic encoding together with list decoding and a disambiguation step. The
converse uses a two-phase "babble-and-push" strategy where the adversary chooses
the state randomly in the first phase, list decodes the output, and then chooses
state inputs to symmetrize the channel in the second phase. These results generalize
prior work on specific channels models (additive, erasure) to general discrete
alphabets and models.
acknowledgement: The work of ADS and ML was supported in part by the US National Science
Foundation under awards CCF-1909468 and CCF-1909451.
article_processing_charge: No
arxiv: 1
author:
- first_name: Yihan
full_name: Zhang, Yihan
id: 2ce5da42-b2ea-11eb-bba5-9f264e9d002c
last_name: Zhang
orcid: 0000-0002-6465-6258
- first_name: Sidharth
full_name: Jaggi, Sidharth
last_name: Jaggi
- first_name: Michael
full_name: Langberg, Michael
last_name: Langberg
- first_name: Anand D.
full_name: Sarwate, Anand D.
last_name: Sarwate
citation:
ama: 'Zhang Y, Jaggi S, Langberg M, Sarwate AD. The capacity of causal adversarial
channels. In: 2022 IEEE International Symposium on Information Theory.
Vol 2022. IEEE; 2022:2523-2528. doi:10.1109/ISIT50566.2022.9834709'
apa: 'Zhang, Y., Jaggi, S., Langberg, M., & Sarwate, A. D. (2022). The capacity
of causal adversarial channels. In 2022 IEEE International Symposium on Information
Theory (Vol. 2022, pp. 2523–2528). Espoo, Finland: IEEE. https://doi.org/10.1109/ISIT50566.2022.9834709'
chicago: Zhang, Yihan, Sidharth Jaggi, Michael Langberg, and Anand D. Sarwate. “The
Capacity of Causal Adversarial Channels.” In 2022 IEEE International Symposium
on Information Theory, 2022:2523–28. IEEE, 2022. https://doi.org/10.1109/ISIT50566.2022.9834709.
ieee: Y. Zhang, S. Jaggi, M. Langberg, and A. D. Sarwate, “The capacity of causal
adversarial channels,” in 2022 IEEE International Symposium on Information
Theory, Espoo, Finland, 2022, vol. 2022, pp. 2523–2528.
ista: 'Zhang Y, Jaggi S, Langberg M, Sarwate AD. 2022. The capacity of causal adversarial
channels. 2022 IEEE International Symposium on Information Theory. ISIT: International
Symposium on Information Theory vol. 2022, 2523–2528.'
mla: Zhang, Yihan, et al. “The Capacity of Causal Adversarial Channels.” 2022
IEEE International Symposium on Information Theory, vol. 2022, IEEE, 2022,
pp. 2523–28, doi:10.1109/ISIT50566.2022.9834709.
short: Y. Zhang, S. Jaggi, M. Langberg, A.D. Sarwate, in:, 2022 IEEE International
Symposium on Information Theory, IEEE, 2022, pp. 2523–2528.
conference:
end_date: 2022-07-01
location: Espoo, Finland
name: 'ISIT: International Symposium on Information Theory'
start_date: 2022-06-26
date_created: 2022-09-04T22:02:03Z
date_published: 2022-08-03T00:00:00Z
date_updated: 2025-09-10T09:40:42Z
day: '03'
department:
- _id: MaMo
doi: 10.1109/ISIT50566.2022.9834709
external_id:
arxiv:
- '2205.06708'
isi:
- '001254261902114'
intvolume: ' 2022'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: ' https://doi.org/10.48550/arXiv.2205.06708'
month: '08'
oa: 1
oa_version: Preprint
page: 2523-2528
publication: 2022 IEEE International Symposium on Information Theory
publication_identifier:
isbn:
- '9781665421591'
issn:
- 2157-8095
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: The capacity of causal adversarial channels
type: conference
user_id: 317138e5-6ab7-11ef-aa6d-ffef3953e345
volume: 2022
year: '2022'
...
---
_id: '12012'
abstract:
- lang: eng
text: This paper is eligible for the Jack Keil Wolf ISIT Student Paper Award. We
generalize a previous framework for designing utility-optimal differentially private
(DP) mechanisms via graphs, where datasets are vertices in the graph and edges
represent dataset neighborhood. The boundary set contains datasets where an individual’s
response changes the binary-valued query compared to its neighbors. Previous work
was limited to the homogeneous case where the privacy parameter ε across all datasets
was the same and the mechanism at boundary datasets was identical. In our work,
the mechanism can take different distributions at the boundary and the privacy
parameter ε is a function of neighboring datasets, which recovers an earlier definition
of personalized DP as special case. The problem is how to extend the mechanism,
which is only defined at the boundary set, to other datasets in the graph in a
computationally efficient and utility optimal manner. Using the concept of strongest
induced DP condition we solve this problem efficiently in polynomial time (in
the size of the graph).
article_processing_charge: No
arxiv: 1
author:
- first_name: Sahel
full_name: Torkamani, Sahel
id: 0503e7f8-2d05-11ed-aa17-db0640c720fc
last_name: Torkamani
- first_name: Javad B.
full_name: Ebrahimi, Javad B.
last_name: Ebrahimi
- first_name: Parastoo
full_name: Sadeghi, Parastoo
last_name: Sadeghi
- first_name: Rafael G.L.
full_name: D'Oliveira, Rafael G.L.
last_name: D'Oliveira
- first_name: Muriel
full_name: Médard, Muriel
last_name: Médard
citation:
ama: 'Torkamani S, Ebrahimi JB, Sadeghi P, D’Oliveira RGL, Médard M. Heterogeneous
differential privacy via graphs. In: 2022 IEEE International Symposium on Information
Theory. Vol 2022. IEEE; 2022:1623-1628. doi:10.1109/ISIT50566.2022.9834711'
apa: 'Torkamani, S., Ebrahimi, J. B., Sadeghi, P., D’Oliveira, R. G. L., & Médard,
M. (2022). Heterogeneous differential privacy via graphs. In 2022 IEEE International
Symposium on Information Theory (Vol. 2022, pp. 1623–1628). Espoo, Finland:
IEEE. https://doi.org/10.1109/ISIT50566.2022.9834711'
chicago: Torkamani, Sahel, Javad B. Ebrahimi, Parastoo Sadeghi, Rafael G.L. D’Oliveira,
and Muriel Médard. “Heterogeneous Differential Privacy via Graphs.” In 2022
IEEE International Symposium on Information Theory, 2022:1623–28. IEEE, 2022.
https://doi.org/10.1109/ISIT50566.2022.9834711.
ieee: S. Torkamani, J. B. Ebrahimi, P. Sadeghi, R. G. L. D’Oliveira, and M. Médard,
“Heterogeneous differential privacy via graphs,” in 2022 IEEE International
Symposium on Information Theory, Espoo, Finland, 2022, vol. 2022, pp. 1623–1628.
ista: 'Torkamani S, Ebrahimi JB, Sadeghi P, D’Oliveira RGL, Médard M. 2022. Heterogeneous
differential privacy via graphs. 2022 IEEE International Symposium on Information
Theory. ISIT: International Symposium on Information Theory vol. 2022, 1623–1628.'
mla: Torkamani, Sahel, et al. “Heterogeneous Differential Privacy via Graphs.” 2022
IEEE International Symposium on Information Theory, vol. 2022, IEEE, 2022,
pp. 1623–28, doi:10.1109/ISIT50566.2022.9834711.
short: S. Torkamani, J.B. Ebrahimi, P. Sadeghi, R.G.L. D’Oliveira, M. Médard, in:,
2022 IEEE International Symposium on Information Theory, IEEE, 2022, pp. 1623–1628.
conference:
end_date: 2022-07-01
location: Espoo, Finland
name: 'ISIT: International Symposium on Information Theory'
start_date: 2022-06-26
date_created: 2022-09-04T22:02:04Z
date_published: 2022-08-03T00:00:00Z
date_updated: 2025-09-10T09:42:41Z
day: '03'
department:
- _id: MaMo
doi: 10.1109/ISIT50566.2022.9834711
external_id:
arxiv:
- '2203.15429'
isi:
- '001254261901131'
intvolume: ' 2022'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.48550/arXiv.2203.15429
month: '08'
oa: 1
oa_version: Preprint
page: 1623-1628
publication: 2022 IEEE International Symposium on Information Theory
publication_identifier:
isbn:
- '9781665421591'
issn:
- 2157-8095
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: Heterogeneous differential privacy via graphs
type: conference
user_id: 317138e5-6ab7-11ef-aa6d-ffef3953e345
volume: 2022
year: '2022'
...
---
_id: '12013'
abstract:
- lang: eng
text: We consider the problem of communication over adversarial channels with feedback.
Two parties comprising sender Alice and receiver Bob seek to communicate reliably.
An adversary James observes Alice's channel transmission entirely and chooses,
maliciously, its additive channel input or jamming state thereby corrupting Bob's
observation. Bob can communicate over a one-way reverse link with Alice; we assume
that transmissions over this feedback link cannot be corrupted by James. Our goal
in this work is to study the optimum throughput or capacity over such channels
with feedback. We first present results for the quadratically-constrained additive
channel where communication is known to be impossible when the noise-to-signal
(power) ratio (NSR) is at least 1. We present a novel achievability scheme to
establish that positive rate communication is possible even when the NSR is as
high as 8/9. We also present new converse upper bounds on the capacity of this
channel under potentially stochastic encoders and decoders. We also study feedback
communication over the more widely studied q-ary alphabet channel under additive
noise. For the q -ary channel, where q > 2, it is well known that capacity is
positive under full feedback if and only if the adversary can corrupt strictly
less than half the transmitted symbols. We generalize this result and show that
the same threshold holds for positive rate communication when the noiseless feedback
may only be partial; our scheme employs a stochastic decoder. We extend this characterization,
albeit partially, to fully deterministic schemes under partial noiseless feedback.
We also present new converse upper bounds for q-ary channels under full feedback,
where the encoder and/or decoder may privately randomize. Our converse results
bring to the fore an interesting alternate expression for the well known converse
bound for the q—ary channel under full feedback which, when specialized to the
binary channel, also equals its known capacity.
article_processing_charge: No
author:
- first_name: Pranav
full_name: Joshi, Pranav
last_name: Joshi
- first_name: Amritakshya
full_name: Purkayastha, Amritakshya
last_name: Purkayastha
- first_name: Yihan
full_name: Zhang, Yihan
id: 2ce5da42-b2ea-11eb-bba5-9f264e9d002c
last_name: Zhang
orcid: 0000-0002-6465-6258
- first_name: Amitalok J.
full_name: Budkuley, Amitalok J.
last_name: Budkuley
- first_name: Sidharth
full_name: Jaggi, Sidharth
last_name: Jaggi
citation:
ama: 'Joshi P, Purkayastha A, Zhang Y, Budkuley AJ, Jaggi S. On the capacity of
additive AVCs with feedback. In: 2022 IEEE International Symposium on Information
Theory. Vol 2022. IEEE; 2022:504-509. doi:10.1109/ISIT50566.2022.9834850'
apa: 'Joshi, P., Purkayastha, A., Zhang, Y., Budkuley, A. J., & Jaggi, S. (2022).
On the capacity of additive AVCs with feedback. In 2022 IEEE International
Symposium on Information Theory (Vol. 2022, pp. 504–509). Espoo, Finland:
IEEE. https://doi.org/10.1109/ISIT50566.2022.9834850'
chicago: Joshi, Pranav, Amritakshya Purkayastha, Yihan Zhang, Amitalok J. Budkuley,
and Sidharth Jaggi. “On the Capacity of Additive AVCs with Feedback.” In 2022
IEEE International Symposium on Information Theory, 2022:504–9. IEEE, 2022.
https://doi.org/10.1109/ISIT50566.2022.9834850.
ieee: P. Joshi, A. Purkayastha, Y. Zhang, A. J. Budkuley, and S. Jaggi, “On the
capacity of additive AVCs with feedback,” in 2022 IEEE International Symposium
on Information Theory, Espoo, Finland, 2022, vol. 2022, pp. 504–509.
ista: 'Joshi P, Purkayastha A, Zhang Y, Budkuley AJ, Jaggi S. 2022. On the capacity
of additive AVCs with feedback. 2022 IEEE International Symposium on Information
Theory. ISIT: International Symposium on Information Theory vol. 2022, 504–509.'
mla: Joshi, Pranav, et al. “On the Capacity of Additive AVCs with Feedback.” 2022
IEEE International Symposium on Information Theory, vol. 2022, IEEE, 2022,
pp. 504–09, doi:10.1109/ISIT50566.2022.9834850.
short: P. Joshi, A. Purkayastha, Y. Zhang, A.J. Budkuley, S. Jaggi, in:, 2022 IEEE
International Symposium on Information Theory, IEEE, 2022, pp. 504–509.
conference:
end_date: 2022-07-01
location: Espoo, Finland
name: 'ISIT: International Symposium on Information Theory'
start_date: 2022-06-26
date_created: 2022-09-04T22:02:04Z
date_published: 2022-08-03T00:00:00Z
date_updated: 2025-09-10T09:42:04Z
day: '03'
department:
- _id: MaMo
doi: 10.1109/ISIT50566.2022.9834850
external_id:
isi:
- '001254261900085'
intvolume: ' 2022'
isi: 1
language:
- iso: eng
month: '08'
oa_version: None
page: 504-509
publication: 2022 IEEE International Symposium on Information Theory
publication_identifier:
isbn:
- '9781665421591'
issn:
- 2157-8095
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: On the capacity of additive AVCs with feedback
type: conference
user_id: 317138e5-6ab7-11ef-aa6d-ffef3953e345
volume: 2022
year: '2022'
...
---
_id: '12014'
abstract:
- lang: eng
text: We study the problem of high-dimensional multiple packing in Euclidean space.
Multiple packing is a natural generalization of sphere packing and is defined
as follows. Let N > 0 and L∈Z≥2. A multiple packing is a set C of points in Rn
such that any point in Rn lies in the intersection of at most L – 1 balls of radius
nN−−−√ around points in C. Given a well-known connection with coding theory, multiple
packings can be viewed as the Euclidean analog of list-decodable codes, which
are well-studied for finite fields. In this paper, we exactly pin down the asymptotic
density of (expurgated) Poisson Point Processes under a stronger notion called
average-radius multiple packing. To this end, we apply tools from high-dimensional
geometry and large deviation theory. This gives rise to the best known lower bound
on the largest multiple packing density. Our result corrects a mistake in a previous
paper by Blinovsky [Bli05].
article_processing_charge: No
author:
- first_name: Yihan
full_name: Zhang, Yihan
id: 2ce5da42-b2ea-11eb-bba5-9f264e9d002c
last_name: Zhang
orcid: 0000-0002-6465-6258
- first_name: Shashank
full_name: Vatedka, Shashank
last_name: Vatedka
citation:
ama: 'Zhang Y, Vatedka S. List-decodability of Poisson Point Processes. In: 2022
IEEE International Symposium on Information Theory. Vol 2022. IEEE; 2022:2559-2564.
doi:10.1109/ISIT50566.2022.9834512'
apa: 'Zhang, Y., & Vatedka, S. (2022). List-decodability of Poisson Point Processes.
In 2022 IEEE International Symposium on Information Theory (Vol. 2022,
pp. 2559–2564). Espoo, Finland: IEEE. https://doi.org/10.1109/ISIT50566.2022.9834512'
chicago: Zhang, Yihan, and Shashank Vatedka. “List-Decodability of Poisson Point
Processes.” In 2022 IEEE International Symposium on Information Theory,
2022:2559–64. IEEE, 2022. https://doi.org/10.1109/ISIT50566.2022.9834512.
ieee: Y. Zhang and S. Vatedka, “List-decodability of Poisson Point Processes,” in
2022 IEEE International Symposium on Information Theory, Espoo, Finland,
2022, vol. 2022, pp. 2559–2564.
ista: 'Zhang Y, Vatedka S. 2022. List-decodability of Poisson Point Processes. 2022
IEEE International Symposium on Information Theory. ISIT: International Symposium
on Information Theory vol. 2022, 2559–2564.'
mla: Zhang, Yihan, and Shashank Vatedka. “List-Decodability of Poisson Point Processes.”
2022 IEEE International Symposium on Information Theory, vol. 2022, IEEE,
2022, pp. 2559–64, doi:10.1109/ISIT50566.2022.9834512.
short: Y. Zhang, S. Vatedka, in:, 2022 IEEE International Symposium on Information
Theory, IEEE, 2022, pp. 2559–2564.
conference:
end_date: 2022-07-01
location: Espoo, Finland
name: 'ISIT: International Symposium on Information Theory'
start_date: 2022-06-26
date_created: 2022-09-04T22:02:04Z
date_published: 2022-08-03T00:00:00Z
date_updated: 2025-09-10T09:41:24Z
day: '03'
department:
- _id: MaMo
doi: 10.1109/ISIT50566.2022.9834512
external_id:
isi:
- '001254261902120'
intvolume: ' 2022'
isi: 1
language:
- iso: eng
month: '08'
oa_version: None
page: 2559-2564
publication: 2022 IEEE International Symposium on Information Theory
publication_identifier:
isbn:
- '9781665421591'
issn:
- 2157-8095
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: List-decodability of Poisson Point Processes
type: conference
user_id: 317138e5-6ab7-11ef-aa6d-ffef3953e345
volume: 2022
year: '2022'
...
---
_id: '12015'
abstract:
- lang: eng
text: We study the problem of high-dimensional multiple packing in Euclidean space.
Multiple packing is a natural generalization of sphere packing and is defined
as follows. Let P, N > 0 and L∈Z≥2. A multiple packing is a set C of points in
Bn(0–,nP−−−√) such that any point in ℝ n lies in the intersection of at most L
– 1 balls of radius nN−−−√ around points in C. 1 In this paper, we derive two
lower bounds on the largest possible density of a multiple packing. These bounds
are obtained through a stronger notion called average-radius multiple packing.
Specifically, we exactly pin down the asymptotics of (expurgated) Gaussian codes
and (expurgated) spherical codes under average-radius multiple packing. To this
end, we apply tools from high-dimensional geometry and large deviation theory.
The bound for spherical codes matches the previous best known bound which was
obtained for the standard (weaker) notion of multiple packing through a curious
connection with error exponents [Bli99], [ZV21]. The bound for Gaussian codes
suggests that they are strictly inferior to spherical codes.
article_processing_charge: No
author:
- first_name: Yihan
full_name: Zhang, Yihan
id: 2ce5da42-b2ea-11eb-bba5-9f264e9d002c
last_name: Zhang
orcid: 0000-0002-6465-6258
- first_name: Shashank
full_name: Vatedka, Shashank
last_name: Vatedka
citation:
ama: 'Zhang Y, Vatedka S. Lower bounds for multiple packing. In: 2022 IEEE International
Symposium on Information Theory. Vol 2022. IEEE; 2022:3085-3090. doi:10.1109/ISIT50566.2022.9834443'
apa: 'Zhang, Y., & Vatedka, S. (2022). Lower bounds for multiple packing. In
2022 IEEE International Symposium on Information Theory (Vol. 2022, pp.
3085–3090). Espoo, Finland: IEEE. https://doi.org/10.1109/ISIT50566.2022.9834443'
chicago: Zhang, Yihan, and Shashank Vatedka. “Lower Bounds for Multiple Packing.”
In 2022 IEEE International Symposium on Information Theory, 2022:3085–90.
IEEE, 2022. https://doi.org/10.1109/ISIT50566.2022.9834443.
ieee: Y. Zhang and S. Vatedka, “Lower bounds for multiple packing,” in 2022 IEEE
International Symposium on Information Theory, Espoo, Finland, 2022, vol.
2022, pp. 3085–3090.
ista: 'Zhang Y, Vatedka S. 2022. Lower bounds for multiple packing. 2022 IEEE International
Symposium on Information Theory. ISIT: International Symposium on Information
Theory vol. 2022, 3085–3090.'
mla: Zhang, Yihan, and Shashank Vatedka. “Lower Bounds for Multiple Packing.” 2022
IEEE International Symposium on Information Theory, vol. 2022, IEEE, 2022,
pp. 3085–90, doi:10.1109/ISIT50566.2022.9834443.
short: Y. Zhang, S. Vatedka, in:, 2022 IEEE International Symposium on Information
Theory, IEEE, 2022, pp. 3085–3090.
conference:
end_date: 2022-07-01
location: Espoo, Finland
name: 'ISIT: International Symposium on Information Theory'
start_date: 2022-06-26
date_created: 2022-09-04T22:02:05Z
date_published: 2022-08-03T00:00:00Z
date_updated: 2025-09-10T09:44:24Z
day: '03'
department:
- _id: MaMo
doi: 10.1109/ISIT50566.2022.9834443
external_id:
isi:
- '001254261903042'
intvolume: ' 2022'
isi: 1
language:
- iso: eng
month: '08'
oa_version: None
page: 3085-3090
publication: 2022 IEEE International Symposium on Information Theory
publication_identifier:
isbn:
- '9781665421591'
issn:
- 2157-8095
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: Lower bounds for multiple packing
type: conference
user_id: 317138e5-6ab7-11ef-aa6d-ffef3953e345
volume: 2022
year: '2022'
...
---
_id: '12016'
abstract:
- lang: eng
text: We consider the problem of coded distributed computing using polar codes.
The average execution time of a coded computing system is related to the error
probability for transmission over the binary erasure channel in recent work by
Soleymani, Jamali and Mahdavifar, where the performance of binary linear codes
is investigated. In this paper, we focus on polar codes and unveil a connection
between the average execution time and the scaling exponent μ of the family of
codes. In the finite-length characterization of polar codes, the scaling exponent
is a key object capturing the speed of convergence to capacity. In particular,
we show that (i) the gap between the normalized average execution time of polar
codes and that of optimal MDS codes is O(n –1/μ ), and (ii) this upper bound can
be improved to roughly O(n –1/2 ) by considering polar codes with large kernels.
We conjecture that these bounds could be improved to O(n –2/μ ) and O(n –1 ),
respectively, and provide a heuristic argument as well as numerical evidence supporting
this view.
acknowledgement: D. Fathollahi and M. Mondelli were partially supported by the 2019
Lopez-Loreta Prize. The authors thank Hamed Hassani and Hessam Mahdavifar for helpful
discussions.
article_processing_charge: No
arxiv: 1
author:
- first_name: Dorsa
full_name: Fathollahi, Dorsa
last_name: Fathollahi
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
citation:
ama: 'Fathollahi D, Mondelli M. Polar coded computing: The role of the scaling exponent.
In: 2022 IEEE International Symposium on Information Theory. Vol 2022.
IEEE; 2022:2154-2159. doi:10.1109/ISIT50566.2022.9834712'
apa: 'Fathollahi, D., & Mondelli, M. (2022). Polar coded computing: The role
of the scaling exponent. In 2022 IEEE International Symposium on Information
Theory (Vol. 2022, pp. 2154–2159). Espoo, Finland: IEEE. https://doi.org/10.1109/ISIT50566.2022.9834712'
chicago: 'Fathollahi, Dorsa, and Marco Mondelli. “Polar Coded Computing: The Role
of the Scaling Exponent.” In 2022 IEEE International Symposium on Information
Theory, 2022:2154–59. IEEE, 2022. https://doi.org/10.1109/ISIT50566.2022.9834712.'
ieee: 'D. Fathollahi and M. Mondelli, “Polar coded computing: The role of the scaling
exponent,” in 2022 IEEE International Symposium on Information Theory,
Espoo, Finland, 2022, vol. 2022, pp. 2154–2159.'
ista: 'Fathollahi D, Mondelli M. 2022. Polar coded computing: The role of the scaling
exponent. 2022 IEEE International Symposium on Information Theory. ISIT: International
Symposium on Information Theory vol. 2022, 2154–2159.'
mla: 'Fathollahi, Dorsa, and Marco Mondelli. “Polar Coded Computing: The Role of
the Scaling Exponent.” 2022 IEEE International Symposium on Information Theory,
vol. 2022, IEEE, 2022, pp. 2154–59, doi:10.1109/ISIT50566.2022.9834712.'
short: D. Fathollahi, M. Mondelli, in:, 2022 IEEE International Symposium on Information
Theory, IEEE, 2022, pp. 2154–2159.
conference:
end_date: 2022-07-01
location: Espoo, Finland
name: 'ISIT: International Symposium on Information Theory'
start_date: 2022-06-26
date_created: 2022-09-04T22:02:05Z
date_published: 2022-08-03T00:00:00Z
date_updated: 2025-09-10T09:43:32Z
day: '03'
department:
- _id: MaMo
doi: 10.1109/ISIT50566.2022.9834712
external_id:
arxiv:
- '2201.10082'
isi:
- '001254261902052'
intvolume: ' 2022'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.48550/arXiv.2201.10082
month: '08'
oa: 1
oa_version: Preprint
page: 2154-2159
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: 2022 IEEE International Symposium on Information Theory
publication_identifier:
isbn:
- '9781665421591'
issn:
- 2157-8095
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'Polar coded computing: The role of the scaling exponent'
type: conference
user_id: 317138e5-6ab7-11ef-aa6d-ffef3953e345
volume: 2022
year: '2022'
...
---
_id: '12480'
abstract:
- lang: eng
text: 'We consider the problem of estimating a signal from measurements obtained
via a generalized linear model. We focus on estimators based on approximate message
passing (AMP), a family of iterative algorithms with many appealing features:
the performance of AMP in the high-dimensional limit can be succinctly characterized
under suitable model assumptions; AMP can also be tailored to the empirical distribution
of the signal entries, and for a wide class of estimation problems, AMP is conjectured
to be optimal among all polynomial-time algorithms. However, a major issue of
AMP is that in many models (such as phase retrieval), it requires an initialization
correlated with the ground-truth signal and independent from the measurement matrix.
Assuming that such an initialization is available is typically not realistic.
In this paper, we solve this problem by proposing an AMP algorithm initialized
with a spectral estimator. With such an initialization, the standard AMP analysis
fails since the spectral estimator depends in a complicated way on the design
matrix. Our main contribution is a rigorous characterization of the performance
of AMP with spectral initialization in the high-dimensional limit. The key technical
idea is to define and analyze a two-phase artificial AMP algorithm that first
produces the spectral estimator, and then closely approximates the iterates of
the true AMP. We also provide numerical results that demonstrate the validity
of the proposed approach.'
acknowledgement: "The authors would like to thank Andrea Montanari for helpful discussions.\r\nM
Mondelli was partially supported by the 2019 Lopez-Loreta Prize. R Venkataramanan
was partially supported by the Alan Turing Institute under the EPSRC Grant\r\nEP/N510129/1."
article_number: '114003'
article_processing_charge: Yes (via OA deal)
article_type: original
author:
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: Ramji
full_name: Venkataramanan, Ramji
last_name: Venkataramanan
citation:
ama: 'Mondelli M, Venkataramanan R. Approximate message passing with spectral initialization
for generalized linear models. Journal of Statistical Mechanics: Theory and
Experiment. 2022;2022(11). doi:10.1088/1742-5468/ac9828'
apa: 'Mondelli, M., & Venkataramanan, R. (2022). Approximate message passing
with spectral initialization for generalized linear models. Journal of Statistical
Mechanics: Theory and Experiment. IOP Publishing. https://doi.org/10.1088/1742-5468/ac9828'
chicago: 'Mondelli, Marco, and Ramji Venkataramanan. “Approximate Message Passing
with Spectral Initialization for Generalized Linear Models.” Journal of Statistical
Mechanics: Theory and Experiment. IOP Publishing, 2022. https://doi.org/10.1088/1742-5468/ac9828.'
ieee: 'M. Mondelli and R. Venkataramanan, “Approximate message passing with spectral
initialization for generalized linear models,” Journal of Statistical Mechanics:
Theory and Experiment, vol. 2022, no. 11. IOP Publishing, 2022.'
ista: 'Mondelli M, Venkataramanan R. 2022. Approximate message passing with spectral
initialization for generalized linear models. Journal of Statistical Mechanics:
Theory and Experiment. 2022(11), 114003.'
mla: 'Mondelli, Marco, and Ramji Venkataramanan. “Approximate Message Passing with
Spectral Initialization for Generalized Linear Models.” Journal of Statistical
Mechanics: Theory and Experiment, vol. 2022, no. 11, 114003, IOP Publishing,
2022, doi:10.1088/1742-5468/ac9828.'
short: 'M. Mondelli, R. Venkataramanan, Journal of Statistical Mechanics: Theory
and Experiment 2022 (2022).'
corr_author: '1'
date_created: 2023-02-02T08:31:57Z
date_published: 2022-11-24T00:00:00Z
date_updated: 2025-04-15T07:50:16Z
day: '24'
ddc:
- '510'
- '530'
department:
- _id: MaMo
doi: 10.1088/1742-5468/ac9828
external_id:
isi:
- '000889589900001'
file:
- access_level: open_access
checksum: 01411ffa76d3e380a0446baeb89b1ef7
content_type: application/pdf
creator: dernst
date_created: 2023-02-02T08:35:52Z
date_updated: 2023-02-02T08:35:52Z
file_id: '12481'
file_name: 2022_JourStatisticalMechanics_Mondelli.pdf
file_size: 1729997
relation: main_file
success: 1
file_date_updated: 2023-02-02T08:35:52Z
has_accepted_license: '1'
intvolume: ' 2022'
isi: 1
issue: '11'
keyword:
- Statistics
- Probability and Uncertainty
- Statistics and Probability
- Statistical and Nonlinear Physics
language:
- iso: eng
month: '11'
oa: 1
oa_version: Published Version
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: 'Journal of Statistical Mechanics: Theory and Experiment'
publication_identifier:
issn:
- 1742-5468
publication_status: published
publisher: IOP Publishing
quality_controlled: '1'
related_material:
record:
- id: '10598'
relation: earlier_version
status: public
scopus_import: '1'
status: public
title: Approximate message passing with spectral initialization for generalized linear
models
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: journal_article
user_id: 4359f0d1-fa6c-11eb-b949-802e58b17ae8
volume: 2022
year: '2022'
...
---
OA_place: repository
OA_type: green
_id: '12537'
abstract:
- lang: eng
text: 'The Neural Tangent Kernel (NTK) has emerged as a powerful tool to provide
memorization, optimization and generalization guarantees in deep neural networks.
A line of work has studied the NTK spectrum for two-layer and deep networks with
at least a layer with Ω(N) neurons, N being the number of training samples. Furthermore,
there is increasing evidence suggesting that deep networks with sub-linear layer
widths are powerful memorizers and optimizers, as long as the number of parameters
exceeds the number of samples. Thus, a natural open question is whether the NTK
is well conditioned in such a challenging sub-linear setup. In this paper, we
answer this question in the affirmative. Our key technical contribution is a lower
bound on the smallest NTK eigenvalue for deep networks with the minimum possible
over-parameterization: the number of parameters is roughly Ω(N) and, hence, the
number of neurons is as little as Ω(N−−√). To showcase the applicability of our
NTK bounds, we provide two results concerning memorization capacity and optimization
guarantees for gradient descent training.'
acknowledgement: "The authors were partially supported by the 2019 Lopez-Loreta prize,
and they would like to thank\r\nQuynh Nguyen, Mahdi Soltanolkotabi and Adel Javanmard
for helpful discussions.\r\n"
alternative_title:
- Advances in Neural Information Processing Systems
article_processing_charge: No
arxiv: 1
author:
- first_name: Simone
full_name: Bombari, Simone
id: ca726dda-de17-11ea-bc14-f9da834f63aa
last_name: Bombari
- first_name: Mohammad Hossein
full_name: Amani, Mohammad Hossein
last_name: Amani
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
citation:
ama: 'Bombari S, Amani MH, Mondelli M. Memorization and optimization in deep neural
networks with minimum over-parameterization. In: 36th Conference on Neural
Information Processing Systems. Vol 35. Neural Information Processing Systems
Foundation; 2022:7628-7640.'
apa: 'Bombari, S., Amani, M. H., & Mondelli, M. (2022). Memorization and optimization
in deep neural networks with minimum over-parameterization. In 36th Conference
on Neural Information Processing Systems (Vol. 35, pp. 7628–7640). New Orleans,
LA, United States: Neural Information Processing Systems Foundation.'
chicago: Bombari, Simone, Mohammad Hossein Amani, and Marco Mondelli. “Memorization
and Optimization in Deep Neural Networks with Minimum Over-Parameterization.”
In 36th Conference on Neural Information Processing Systems, 35:7628–40.
Neural Information Processing Systems Foundation, 2022.
ieee: S. Bombari, M. H. Amani, and M. Mondelli, “Memorization and optimization in
deep neural networks with minimum over-parameterization,” in 36th Conference
on Neural Information Processing Systems, New Orleans, LA, United States,
2022, vol. 35, pp. 7628–7640.
ista: 'Bombari S, Amani MH, Mondelli M. 2022. Memorization and optimization in deep
neural networks with minimum over-parameterization. 36th Conference on Neural
Information Processing Systems. NeurIPS: Neural Information Processing Systems,
Advances in Neural Information Processing Systems, vol. 35, 7628–7640.'
mla: Bombari, Simone, et al. “Memorization and Optimization in Deep Neural Networks
with Minimum Over-Parameterization.” 36th Conference on Neural Information
Processing Systems, vol. 35, Neural Information Processing Systems Foundation,
2022, pp. 7628–40.
short: S. Bombari, M.H. Amani, M. Mondelli, in:, 36th Conference on Neural Information
Processing Systems, Neural Information Processing Systems Foundation, 2022, pp.
7628–7640.
conference:
end_date: 2022-12-09
location: New Orleans, LA, United States
name: 'NeurIPS: Neural Information Processing Systems'
start_date: 2022-11-28
corr_author: '1'
date_created: 2023-02-10T13:46:37Z
date_published: 2022-07-24T00:00:00Z
date_updated: 2025-05-14T11:28:22Z
day: '24'
department:
- _id: MaMo
external_id:
arxiv:
- '2205.10217'
intvolume: ' 35'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: ' https://doi.org/10.48550/arXiv.2205.10217'
month: '07'
oa: 1
oa_version: Preprint
page: 7628-7640
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: 36th Conference on Neural Information Processing Systems
publication_identifier:
eissn:
- 1049-5258
isbn:
- '9781713871088'
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
status: public
title: Memorization and optimization in deep neural networks with minimum over-parameterization
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2022'
...
---
_id: '12538'
abstract:
- lang: eng
text: In this paper, we study the compression of a target two-layer neural network
with N nodes into a compressed network with MIEEE Information Theory Workshop.
2022:588-593. doi:10.1109/ITW54588.2022.9965870
apa: 'Amani, M. H., Bombari, S., Mondelli, M., Pukdee, R., & Rini, S. (2022).
Sharp asymptotics on the compression of two-layer neural networks. IEEE Information
Theory Workshop. Mumbai, India: IEEE. https://doi.org/10.1109/ITW54588.2022.9965870'
chicago: Amani, Mohammad Hossein, Simone Bombari, Marco Mondelli, Rattana Pukdee,
and Stefano Rini. “Sharp Asymptotics on the Compression of Two-Layer Neural Networks.”
IEEE Information Theory Workshop. IEEE, 2022. https://doi.org/10.1109/ITW54588.2022.9965870.
ieee: M. H. Amani, S. Bombari, M. Mondelli, R. Pukdee, and S. Rini, “Sharp asymptotics
on the compression of two-layer neural networks,” IEEE Information Theory Workshop.
IEEE, pp. 588–593, 2022.
ista: Amani MH, Bombari S, Mondelli M, Pukdee R, Rini S. 2022. Sharp asymptotics
on the compression of two-layer neural networks. IEEE Information Theory Workshop.,
588–593.
mla: Amani, Mohammad Hossein, et al. “Sharp Asymptotics on the Compression of Two-Layer
Neural Networks.” IEEE Information Theory Workshop, IEEE, 2022, pp. 588–93,
doi:10.1109/ITW54588.2022.9965870.
short: M.H. Amani, S. Bombari, M. Mondelli, R. Pukdee, S. Rini, IEEE Information
Theory Workshop (2022) 588–593.
conference:
end_date: 2022-11-09
location: Mumbai, India
name: 'ITW: Information Theory Workshop'
start_date: 2022-11-01
date_created: 2023-02-10T13:47:56Z
date_published: 2022-11-16T00:00:00Z
date_updated: 2025-09-10T09:53:31Z
day: '16'
department:
- _id: MaMo
doi: 10.1109/ITW54588.2022.9965870
external_id:
arxiv:
- '2205.08199'
isi:
- '000904341100099'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: ' https://doi.org/10.48550/arXiv.2205.08199'
month: '11'
oa: 1
oa_version: Preprint
page: 588-593
publication: IEEE Information Theory Workshop
publication_identifier:
isbn:
- '9781665483414'
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: Sharp asymptotics on the compression of two-layer neural networks
type: journal_article
user_id: 317138e5-6ab7-11ef-aa6d-ffef3953e345
year: '2022'
...
---
_id: '12540'
abstract:
- lang: eng
text: We consider the problem of signal estimation in generalized linear models
defined via rotationally invariant design matrices. Since these matrices can have
an arbitrary spectral distribution, this model is well suited for capturing complex
correlation structures which often arise in applications. We propose a novel family
of approximate message passing (AMP) algorithms for signal estimation, and rigorously
characterize their performance in the high-dimensional limit via a state evolution
recursion. Our rotationally invariant AMP has complexity of the same order as
the existing AMP derived under the restrictive assumption of a Gaussian design;
our algorithm also recovers this existing AMP as a special case. Numerical results
showcase a performance close to Vector AMP (which is conjectured to be Bayes-optimal
in some settings), but obtained with a much lower complexity, as the proposed
algorithm does not require a computationally expensive singular value decomposition.
acknowledgement: The authors would like to thank the anonymous reviewers for their
helpful comments. KK and MM were partially supported by the 2019 Lopez-Loreta Prize.
article_number: '22'
article_processing_charge: No
author:
- first_name: Ramji
full_name: Venkataramanan, Ramji
last_name: Venkataramanan
- first_name: Kevin
full_name: Kögler, Kevin
id: 94ec913c-dc85-11ea-9058-e5051ab2428b
last_name: Kögler
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
citation:
ama: 'Venkataramanan R, Kögler K, Mondelli M. Estimation in rotationally invariant
generalized linear models via approximate message passing. In: Proceedings
of the 39th International Conference on Machine Learning. Vol 162. ML Research
Press; 2022.'
apa: 'Venkataramanan, R., Kögler, K., & Mondelli, M. (2022). Estimation in rotationally
invariant generalized linear models via approximate message passing. In Proceedings
of the 39th International Conference on Machine Learning (Vol. 162). Baltimore,
MD, United States: ML Research Press.'
chicago: Venkataramanan, Ramji, Kevin Kögler, and Marco Mondelli. “Estimation in
Rotationally Invariant Generalized Linear Models via Approximate Message Passing.”
In Proceedings of the 39th International Conference on Machine Learning,
Vol. 162. ML Research Press, 2022.
ieee: R. Venkataramanan, K. Kögler, and M. Mondelli, “Estimation in rotationally
invariant generalized linear models via approximate message passing,” in Proceedings
of the 39th International Conference on Machine Learning, Baltimore, MD, United
States, 2022, vol. 162.
ista: 'Venkataramanan R, Kögler K, Mondelli M. 2022. Estimation in rotationally
invariant generalized linear models via approximate message passing. Proceedings
of the 39th International Conference on Machine Learning. ICML: International
Conference on Machine Learning vol. 162, 22.'
mla: Venkataramanan, Ramji, et al. “Estimation in Rotationally Invariant Generalized
Linear Models via Approximate Message Passing.” Proceedings of the 39th International
Conference on Machine Learning, vol. 162, 22, ML Research Press, 2022.
short: R. Venkataramanan, K. Kögler, M. Mondelli, in:, Proceedings of the 39th International
Conference on Machine Learning, ML Research Press, 2022.
conference:
end_date: 2022-07-23
location: Baltimore, MD, United States
name: 'ICML: International Conference on Machine Learning'
start_date: 2022-07-17
corr_author: '1'
date_created: 2023-02-10T13:49:04Z
date_published: 2022-01-01T00:00:00Z
date_updated: 2025-04-15T07:50:16Z
ddc:
- '000'
department:
- _id: MaMo
file:
- access_level: open_access
checksum: 67436eb0a660789514cdf9db79e84683
content_type: application/pdf
creator: dernst
date_created: 2023-02-13T10:53:11Z
date_updated: 2023-02-13T10:53:11Z
file_id: '12547'
file_name: 2022_PMLR_Venkataramanan.pdf
file_size: 2341343
relation: main_file
success: 1
file_date_updated: 2023-02-13T10:53:11Z
has_accepted_license: '1'
intvolume: ' 162'
language:
- iso: eng
oa: 1
oa_version: Published Version
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: Proceedings of the 39th International Conference on Machine Learning
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
status: public
title: Estimation in rotationally invariant generalized linear models via approximate
message passing
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 162
year: '2022'
...
---
_id: '12860'
abstract:
- lang: eng
text: 'Memorization of the relation between entities in a dataset can lead to privacy
issues when using a trained model for question answering. We introduce Relational
Memorization (RM) to understand, quantify and control this phenomenon. While bounding
general memorization can have detrimental effects on the performance of a trained
model, bounding RM does not prevent effective learning. The difference is most
pronounced when the data distribution is long-tailed, with many queries having
only few training examples: Impeding general memorization prevents effective learning,
while impeding only relational memorization still allows learning general properties
of the underlying concepts. We formalize the notion of Relational Privacy (RP)
and, inspired by Differential Privacy (DP), we provide a possible definition of
Differential Relational Privacy (DrP). These notions can be used to describe and
compute bounds on the amount of RM in a trained model. We illustrate Relational
Privacy concepts in experiments with large-scale models for Question Answering.'
article_number: '2203.16701'
article_processing_charge: No
arxiv: 1
author:
- first_name: Simone
full_name: Bombari, Simone
id: ca726dda-de17-11ea-bc14-f9da834f63aa
last_name: Bombari
- first_name: Alessandro
full_name: Achille, Alessandro
last_name: Achille
- first_name: Zijian
full_name: Wang, Zijian
last_name: Wang
- first_name: Yu-Xiang
full_name: Wang, Yu-Xiang
last_name: Wang
- first_name: Yusheng
full_name: Xie, Yusheng
last_name: Xie
- first_name: Kunwar Yashraj
full_name: Singh, Kunwar Yashraj
last_name: Singh
- first_name: Srikar
full_name: Appalaraju, Srikar
last_name: Appalaraju
- first_name: Vijay
full_name: Mahadevan, Vijay
last_name: Mahadevan
- first_name: Stefano
full_name: Soatto, Stefano
last_name: Soatto
citation:
ama: Bombari S, Achille A, Wang Z, et al. Towards differential relational privacy
and its use in question answering. arXiv. doi:10.48550/arXiv.2203.16701
apa: Bombari, S., Achille, A., Wang, Z., Wang, Y.-X., Xie, Y., Singh, K. Y., … Soatto,
S. (n.d.). Towards differential relational privacy and its use in question answering.
arXiv. https://doi.org/10.48550/arXiv.2203.16701
chicago: Bombari, Simone, Alessandro Achille, Zijian Wang, Yu-Xiang Wang, Yusheng
Xie, Kunwar Yashraj Singh, Srikar Appalaraju, Vijay Mahadevan, and Stefano Soatto.
“Towards Differential Relational Privacy and Its Use in Question Answering.” ArXiv,
n.d. https://doi.org/10.48550/arXiv.2203.16701.
ieee: S. Bombari et al., “Towards differential relational privacy and its
use in question answering,” arXiv. .
ista: Bombari S, Achille A, Wang Z, Wang Y-X, Xie Y, Singh KY, Appalaraju S, Mahadevan
V, Soatto S. Towards differential relational privacy and its use in question answering.
arXiv, 2203.16701.
mla: Bombari, Simone, et al. “Towards Differential Relational Privacy and Its Use
in Question Answering.” ArXiv, 2203.16701, doi:10.48550/arXiv.2203.16701.
short: S. Bombari, A. Achille, Z. Wang, Y.-X. Wang, Y. Xie, K.Y. Singh, S. Appalaraju,
V. Mahadevan, S. Soatto, ArXiv (n.d.).
date_created: 2023-04-23T16:11:48Z
date_published: 2022-03-30T00:00:00Z
date_updated: 2023-04-25T07:34:49Z
day: '30'
department:
- _id: GradSch
- _id: MaMo
doi: 10.48550/arXiv.2203.16701
external_id:
arxiv:
- '2203.16701'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.48550/arXiv.2203.16701
month: '03'
oa: 1
oa_version: Preprint
publication: arXiv
publication_status: submitted
status: public
title: Towards differential relational privacy and its use in question answering
type: preprint
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2022'
...
---
_id: '17086'
abstract:
- lang: eng
text: 'We consider a high-dimensional mean estimation problem over a binary hidden
Markov model, which illuminates the interplay between memory in data, sample size,
dimension, and signal strength in statistical inference. In this model, an estimator
observes n samples of a d-dimensional parameter vector θ∗∈Rd, multiplied by a
random sign Si (1≤i≤n), and corrupted by isotropic standard Gaussian noise. The
sequence of signs {Si}i∈[n]∈{−1,1}n is drawn from a stationary homogeneous Markov
chain with flip probability δ∈[0,1/2]. As δ varies, this model smoothly interpolates
two well-studied models: the Gaussian Location Model for which δ=0 and the Gaussian
Mixture Model for which δ=1/2. Assuming that the estimator knows δ, we establish
a nearly minimax optimal (up to logarithmic factors) estimation error rate, as
a function of ∥θ∗∥,δ,d,n. We then provide an upper bound to the case of estimating
δ, assuming a (possibly inaccurate) knowledge of θ∗. The bound is proved to be
tight when θ∗ is an accurately known constant. These results are then combined
to an algorithm which estimates θ∗ with δ unknown a priori, and theoretical guarantees
on its error are stated.'
acknowledgement: "Part of this work was done when YZ was a postdoc at Technion where
he received funding from\r\nthe European Union’s Horizon 2020 research and innovation
programme under grant agreement No 682203-ERC-[Inf-Speed-Tradeoff]. The work of
of NW was supported in part by the Israel Science Foundation (ISF) under Grant 1782/22.
NW is grateful to Guy Bresler for introducing him to this problem, for the initial
ideas that led to this research, and for many helpful discussions on the topic."
alternative_title:
- NeurIPS
article_processing_charge: No
arxiv: 1
author:
- first_name: Yihan
full_name: Zhang, Yihan
id: 2ce5da42-b2ea-11eb-bba5-9f264e9d002c
last_name: Zhang
orcid: 0000-0002-6465-6258
- first_name: Nir
full_name: Weinberger, Nir
last_name: Weinberger
citation:
ama: 'Zhang Y, Weinberger N. Mean estimation in high-dimensional binary Markov Gaussian
mixture models. In: 36th Conference on Neural Information Processing Systems.
Vol 35. ML Research Press; 2022.'
apa: 'Zhang, Y., & Weinberger, N. (2022). Mean estimation in high-dimensional
binary Markov Gaussian mixture models. In 36th Conference on Neural Information
Processing Systems (Vol. 35). New Orleans, LA, United States: ML Research
Press.'
chicago: Zhang, Yihan, and Nir Weinberger. “Mean Estimation in High-Dimensional
Binary Markov Gaussian Mixture Models.” In 36th Conference on Neural Information
Processing Systems, Vol. 35. ML Research Press, 2022.
ieee: Y. Zhang and N. Weinberger, “Mean estimation in high-dimensional binary Markov
Gaussian mixture models,” in 36th Conference on Neural Information Processing
Systems, New Orleans, LA, United States, 2022, vol. 35.
ista: 'Zhang Y, Weinberger N. 2022. Mean estimation in high-dimensional binary Markov
Gaussian mixture models. 36th Conference on Neural Information Processing Systems.
NeurIPS: Neural Information Processing Systems, NeurIPS, vol. 35.'
mla: Zhang, Yihan, and Nir Weinberger. “Mean Estimation in High-Dimensional Binary
Markov Gaussian Mixture Models.” 36th Conference on Neural Information Processing
Systems, vol. 35, ML Research Press, 2022.
short: Y. Zhang, N. Weinberger, in:, 36th Conference on Neural Information Processing
Systems, ML Research Press, 2022.
conference:
end_date: 2022-12-09
location: New Orleans, LA, United States
name: 'NeurIPS: Neural Information Processing Systems'
start_date: 2022-11-28
corr_author: '1'
date_created: 2024-05-29T06:37:16Z
date_published: 2022-12-01T00:00:00Z
date_updated: 2024-08-05T09:48:58Z
day: '01'
ddc:
- '000'
department:
- _id: MaMo
external_id:
arxiv:
- '2206.02455'
file:
- access_level: open_access
checksum: 05f6f9f8fc34e224e0cad045b9489030
content_type: application/pdf
creator: dernst
date_created: 2024-08-05T09:44:49Z
date_updated: 2024-08-05T09:44:49Z
file_id: '17392'
file_name: 2022_NeurIPS_Zhang.pdf
file_size: 476307
relation: main_file
success: 1
file_date_updated: 2024-08-05T09:44:49Z
has_accepted_license: '1'
intvolume: ' 35'
language:
- iso: eng
month: '12'
oa: 1
oa_version: Published Version
publication: 36th Conference on Neural Information Processing Systems
publication_identifier:
isbn:
- '9781713871088'
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
scopus_import: '1'
status: public
title: Mean estimation in high-dimensional binary Markov Gaussian mixture models
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2022'
...
---
_id: '10211'
abstract:
- lang: eng
text: "We study the problem of recovering an unknown signal \U0001D465\U0001D465
given measurements obtained from a generalized linear model with a Gaussian sensing
matrix. Two popular solutions are based on a linear estimator \U0001D465\U0001D465^L
and a spectral estimator \U0001D465\U0001D465^s. The former is a data-dependent
linear combination of the columns of the measurement matrix, and its analysis
is quite simple. The latter is the principal eigenvector of a data-dependent matrix,
and a recent line of work has studied its performance. In this paper, we show
how to optimally combine \U0001D465\U0001D465^L and \U0001D465\U0001D465^s. At
the heart of our analysis is the exact characterization of the empirical joint
distribution of (\U0001D465\U0001D465,\U0001D465\U0001D465^L,\U0001D465\U0001D465^s)
in the high-dimensional limit. This allows us to compute the Bayes-optimal combination
of \U0001D465\U0001D465^L and \U0001D465\U0001D465^s, given the limiting distribution
of the signal \U0001D465\U0001D465. When the distribution of the signal is Gaussian,
then the Bayes-optimal combination has the form \U0001D703\U0001D465\U0001D465^L+\U0001D465\U0001D465^s
and we derive the optimal combination coefficient. In order to establish the limiting
distribution of (\U0001D465\U0001D465,\U0001D465\U0001D465^L,\U0001D465\U0001D465^s),
we design and analyze an approximate message passing algorithm whose iterates
give \U0001D465\U0001D465^L and approach \U0001D465\U0001D465^s. Numerical simulations
demonstrate the improvement of the proposed combination with respect to the two
methods considered separately."
acknowledgement: M. Mondelli would like to thank Andrea Montanari for helpful discussions.
All the authors would like to thank the anonymous reviewers for their helpful comments.
article_processing_charge: Yes (via OA deal)
article_type: original
arxiv: 1
author:
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: Christos
full_name: Thrampoulidis, Christos
last_name: Thrampoulidis
- first_name: Ramji
full_name: Venkataramanan, Ramji
last_name: Venkataramanan
citation:
ama: Mondelli M, Thrampoulidis C, Venkataramanan R. Optimal combination of linear
and spectral estimators for generalized linear models. Foundations of Computational
Mathematics. 2022;22(5):1513-1566. doi:10.1007/s10208-021-09531-x
apa: Mondelli, M., Thrampoulidis, C., & Venkataramanan, R. (2022). Optimal combination
of linear and spectral estimators for generalized linear models. Foundations
of Computational Mathematics. Springer. https://doi.org/10.1007/s10208-021-09531-x
chicago: Mondelli, Marco, Christos Thrampoulidis, and Ramji Venkataramanan. “Optimal
Combination of Linear and Spectral Estimators for Generalized Linear Models.”
Foundations of Computational Mathematics. Springer, 2022. https://doi.org/10.1007/s10208-021-09531-x.
ieee: M. Mondelli, C. Thrampoulidis, and R. Venkataramanan, “Optimal combination
of linear and spectral estimators for generalized linear models,” Foundations
of Computational Mathematics, vol. 22, no. 5. Springer, pp. 1513–1566, 2022.
ista: Mondelli M, Thrampoulidis C, Venkataramanan R. 2022. Optimal combination of
linear and spectral estimators for generalized linear models. Foundations of Computational
Mathematics. 22(5), 1513–1566.
mla: Mondelli, Marco, et al. “Optimal Combination of Linear and Spectral Estimators
for Generalized Linear Models.” Foundations of Computational Mathematics,
vol. 22, no. 5, Springer, 2022, pp. 1513–66, doi:10.1007/s10208-021-09531-x.
short: M. Mondelli, C. Thrampoulidis, R. Venkataramanan, Foundations of Computational
Mathematics 22 (2022) 1513–1566.
date_created: 2021-11-03T10:59:08Z
date_published: 2022-10-01T00:00:00Z
date_updated: 2025-04-15T06:53:08Z
day: '01'
ddc:
- '510'
department:
- _id: MaMo
doi: 10.1007/s10208-021-09531-x
external_id:
arxiv:
- '2008.03326'
isi:
- '000685721000001'
file:
- access_level: open_access
checksum: 9ea12dd8045a0678000a3a59295221cb
content_type: application/pdf
creator: alisjak
date_created: 2021-12-13T15:47:54Z
date_updated: 2021-12-13T15:47:54Z
file_id: '10542'
file_name: 2021_Springer_Mondelli.pdf
file_size: 2305731
relation: main_file
success: 1
file_date_updated: 2021-12-13T15:47:54Z
has_accepted_license: '1'
intvolume: ' 22'
isi: 1
issue: '5'
keyword:
- Applied Mathematics
- Computational Theory and Mathematics
- Computational Mathematics
- Analysis
language:
- iso: eng
month: '10'
oa: 1
oa_version: Published Version
page: 1513-1566
project:
- _id: B67AFEDC-15C9-11EA-A837-991A96BB2854
name: IST Austria Open Access Fund
publication: Foundations of Computational Mathematics
publication_identifier:
eissn:
- 1615-3383
issn:
- 1615-3375
publication_status: published
publisher: Springer
quality_controlled: '1'
scopus_import: '1'
status: public
title: Optimal combination of linear and spectral estimators for generalized linear
models
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: journal_article
user_id: 3E5EF7F0-F248-11E8-B48F-1D18A9856A87
volume: 22
year: '2022'
...