---
_id: '17086'
abstract:
- lang: eng
text: 'We consider a high-dimensional mean estimation problem over a binary hidden
Markov model, which illuminates the interplay between memory in data, sample size,
dimension, and signal strength in statistical inference. In this model, an estimator
observes n samples of a d-dimensional parameter vector θ∗∈Rd, multiplied by a
random sign Si (1≤i≤n), and corrupted by isotropic standard Gaussian noise. The
sequence of signs {Si}i∈[n]∈{−1,1}n is drawn from a stationary homogeneous Markov
chain with flip probability δ∈[0,1/2]. As δ varies, this model smoothly interpolates
two well-studied models: the Gaussian Location Model for which δ=0 and the Gaussian
Mixture Model for which δ=1/2. Assuming that the estimator knows δ, we establish
a nearly minimax optimal (up to logarithmic factors) estimation error rate, as
a function of ∥θ∗∥,δ,d,n. We then provide an upper bound to the case of estimating
δ, assuming a (possibly inaccurate) knowledge of θ∗. The bound is proved to be
tight when θ∗ is an accurately known constant. These results are then combined
to an algorithm which estimates θ∗ with δ unknown a priori, and theoretical guarantees
on its error are stated.'
acknowledgement: "Part of this work was done when YZ was a postdoc at Technion where
he received funding from\r\nthe European Union’s Horizon 2020 research and innovation
programme under grant agreement No 682203-ERC-[Inf-Speed-Tradeoff]. The work of
of NW was supported in part by the Israel Science Foundation (ISF) under Grant 1782/22.
NW is grateful to Guy Bresler for introducing him to this problem, for the initial
ideas that led to this research, and for many helpful discussions on the topic."
alternative_title:
- NeurIPS
article_processing_charge: No
arxiv: 1
author:
- first_name: Yihan
full_name: Zhang, Yihan
id: 2ce5da42-b2ea-11eb-bba5-9f264e9d002c
last_name: Zhang
orcid: 0000-0002-6465-6258
- first_name: Nir
full_name: Weinberger, Nir
last_name: Weinberger
citation:
ama: 'Zhang Y, Weinberger N. Mean estimation in high-dimensional binary Markov Gaussian
mixture models. In: 36th Conference on Neural Information Processing Systems.
Vol 35. ML Research Press; 2022.'
apa: 'Zhang, Y., & Weinberger, N. (2022). Mean estimation in high-dimensional
binary Markov Gaussian mixture models. In 36th Conference on Neural Information
Processing Systems (Vol. 35). New Orleans, LA, United States: ML Research
Press.'
chicago: Zhang, Yihan, and Nir Weinberger. “Mean Estimation in High-Dimensional
Binary Markov Gaussian Mixture Models.” In 36th Conference on Neural Information
Processing Systems, Vol. 35. ML Research Press, 2022.
ieee: Y. Zhang and N. Weinberger, “Mean estimation in high-dimensional binary Markov
Gaussian mixture models,” in 36th Conference on Neural Information Processing
Systems, New Orleans, LA, United States, 2022, vol. 35.
ista: 'Zhang Y, Weinberger N. 2022. Mean estimation in high-dimensional binary Markov
Gaussian mixture models. 36th Conference on Neural Information Processing Systems.
NeurIPS: Neural Information Processing Systems, NeurIPS, vol. 35.'
mla: Zhang, Yihan, and Nir Weinberger. “Mean Estimation in High-Dimensional Binary
Markov Gaussian Mixture Models.” 36th Conference on Neural Information Processing
Systems, vol. 35, ML Research Press, 2022.
short: Y. Zhang, N. Weinberger, in:, 36th Conference on Neural Information Processing
Systems, ML Research Press, 2022.
conference:
end_date: 2022-12-09
location: New Orleans, LA, United States
name: 'NeurIPS: Neural Information Processing Systems'
start_date: 2022-11-28
corr_author: '1'
date_created: 2024-05-29T06:37:16Z
date_published: 2022-12-01T00:00:00Z
date_updated: 2024-08-05T09:48:58Z
day: '01'
ddc:
- '000'
department:
- _id: MaMo
external_id:
arxiv:
- '2206.02455'
file:
- access_level: open_access
checksum: 05f6f9f8fc34e224e0cad045b9489030
content_type: application/pdf
creator: dernst
date_created: 2024-08-05T09:44:49Z
date_updated: 2024-08-05T09:44:49Z
file_id: '17392'
file_name: 2022_NeurIPS_Zhang.pdf
file_size: 476307
relation: main_file
success: 1
file_date_updated: 2024-08-05T09:44:49Z
has_accepted_license: '1'
intvolume: ' 35'
language:
- iso: eng
month: '12'
oa: 1
oa_version: Published Version
publication: 36th Conference on Neural Information Processing Systems
publication_identifier:
isbn:
- '9781713871088'
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
scopus_import: '1'
status: public
title: Mean estimation in high-dimensional binary Markov Gaussian mixture models
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2022'
...
---
_id: '10211'
abstract:
- lang: eng
text: "We study the problem of recovering an unknown signal \U0001D465\U0001D465
given measurements obtained from a generalized linear model with a Gaussian sensing
matrix. Two popular solutions are based on a linear estimator \U0001D465\U0001D465^L
and a spectral estimator \U0001D465\U0001D465^s. The former is a data-dependent
linear combination of the columns of the measurement matrix, and its analysis
is quite simple. The latter is the principal eigenvector of a data-dependent matrix,
and a recent line of work has studied its performance. In this paper, we show
how to optimally combine \U0001D465\U0001D465^L and \U0001D465\U0001D465^s. At
the heart of our analysis is the exact characterization of the empirical joint
distribution of (\U0001D465\U0001D465,\U0001D465\U0001D465^L,\U0001D465\U0001D465^s)
in the high-dimensional limit. This allows us to compute the Bayes-optimal combination
of \U0001D465\U0001D465^L and \U0001D465\U0001D465^s, given the limiting distribution
of the signal \U0001D465\U0001D465. When the distribution of the signal is Gaussian,
then the Bayes-optimal combination has the form \U0001D703\U0001D465\U0001D465^L+\U0001D465\U0001D465^s
and we derive the optimal combination coefficient. In order to establish the limiting
distribution of (\U0001D465\U0001D465,\U0001D465\U0001D465^L,\U0001D465\U0001D465^s),
we design and analyze an approximate message passing algorithm whose iterates
give \U0001D465\U0001D465^L and approach \U0001D465\U0001D465^s. Numerical simulations
demonstrate the improvement of the proposed combination with respect to the two
methods considered separately."
acknowledgement: M. Mondelli would like to thank Andrea Montanari for helpful discussions.
All the authors would like to thank the anonymous reviewers for their helpful comments.
article_processing_charge: Yes (via OA deal)
article_type: original
arxiv: 1
author:
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: Christos
full_name: Thrampoulidis, Christos
last_name: Thrampoulidis
- first_name: Ramji
full_name: Venkataramanan, Ramji
last_name: Venkataramanan
citation:
ama: Mondelli M, Thrampoulidis C, Venkataramanan R. Optimal combination of linear
and spectral estimators for generalized linear models. Foundations of Computational
Mathematics. 2022;22(5):1513-1566. doi:10.1007/s10208-021-09531-x
apa: Mondelli, M., Thrampoulidis, C., & Venkataramanan, R. (2022). Optimal combination
of linear and spectral estimators for generalized linear models. Foundations
of Computational Mathematics. Springer. https://doi.org/10.1007/s10208-021-09531-x
chicago: Mondelli, Marco, Christos Thrampoulidis, and Ramji Venkataramanan. “Optimal
Combination of Linear and Spectral Estimators for Generalized Linear Models.”
Foundations of Computational Mathematics. Springer, 2022. https://doi.org/10.1007/s10208-021-09531-x.
ieee: M. Mondelli, C. Thrampoulidis, and R. Venkataramanan, “Optimal combination
of linear and spectral estimators for generalized linear models,” Foundations
of Computational Mathematics, vol. 22, no. 5. Springer, pp. 1513–1566, 2022.
ista: Mondelli M, Thrampoulidis C, Venkataramanan R. 2022. Optimal combination of
linear and spectral estimators for generalized linear models. Foundations of Computational
Mathematics. 22(5), 1513–1566.
mla: Mondelli, Marco, et al. “Optimal Combination of Linear and Spectral Estimators
for Generalized Linear Models.” Foundations of Computational Mathematics,
vol. 22, no. 5, Springer, 2022, pp. 1513–66, doi:10.1007/s10208-021-09531-x.
short: M. Mondelli, C. Thrampoulidis, R. Venkataramanan, Foundations of Computational
Mathematics 22 (2022) 1513–1566.
date_created: 2021-11-03T10:59:08Z
date_published: 2022-10-01T00:00:00Z
date_updated: 2025-04-15T06:53:08Z
day: '01'
ddc:
- '510'
department:
- _id: MaMo
doi: 10.1007/s10208-021-09531-x
external_id:
arxiv:
- '2008.03326'
isi:
- '000685721000001'
file:
- access_level: open_access
checksum: 9ea12dd8045a0678000a3a59295221cb
content_type: application/pdf
creator: alisjak
date_created: 2021-12-13T15:47:54Z
date_updated: 2021-12-13T15:47:54Z
file_id: '10542'
file_name: 2021_Springer_Mondelli.pdf
file_size: 2305731
relation: main_file
success: 1
file_date_updated: 2021-12-13T15:47:54Z
fulldoi: https://doi.org/10.1007/s10208-021-09531-x
has_accepted_license: '1'
intvolume: ' 22'
isi: 1
issue: '5'
keyword:
- Applied Mathematics
- Computational Theory and Mathematics
- Computational Mathematics
- Analysis
language:
- iso: eng
license: https://creativecommons.org/licenses/by/4.0/
month: '10'
oa: 1
oa_version: Published Version
page: 1513-1566
project:
- _id: B67AFEDC-15C9-11EA-A837-991A96BB2854
name: IST Austria Open Access Fund
publication: Foundations of Computational Mathematics
publication_identifier:
eissn:
- 1615-3383
issn:
- 1615-3375
publication_status: published
publisher: Springer
quality_controlled: '1'
scopus_import: '1'
status: public
title: Optimal combination of linear and spectral estimators for generalized linear
models
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: journal_article
user_id: 3E5EF7F0-F248-11E8-B48F-1D18A9856A87
volume: 22
year: '2022'
...
---
_id: '12536'
abstract:
- lang: eng
text: 'We consider the problem of estimating a rank-1 signal corrupted by structured
rotationally invariant noise, and address the following question: how well do
inference algorithms perform when the noise statistics is unknown and hence Gaussian
noise is assumed? While the matched Bayes-optimal setting with unstructured noise
is well understood, the analysis of this mismatched problem is only at its premises.
In this paper, we make a step towards understanding the effect of the strong source
of mismatch which is the noise statistics. Our main technical contribution is
the rigorous analysis of a Bayes estimator and of an approximate message passing
(AMP) algorithm, both of which incorrectly assume a Gaussian setup. The first
result exploits the theory of spherical integrals and of low-rank matrix perturbations;
the idea behind the second one is to design and analyze an artificial AMP which,
by taking advantage of the flexibility in the denoisers, is able to "correct"
the mismatch. Armed with these sharp asymptotic characterizations, we unveil a
rich and often unexpected phenomenology. For example, despite AMP is in principle
designed to efficiently compute the Bayes estimator, the former is outperformed
by the latter in terms of mean-square error. We show that this performance gap
is due to an incorrect estimation of the signal norm. In fact, when the SNR is
large enough, the overlaps of the AMP and the Bayes estimator coincide, and they
even match those of optimal estimators taking into account the structure of the
noise.'
acknowledgement: "M. Mondelli was partially supported by the 2019 Lopez-Loreta Prize.
The authors acknowledge\r\ndiscussions with A. Krajenbrink, M. Robinson, A. Depope,
N. Macris and F. Pourkamali.\r\n"
alternative_title:
- Advances in Neural Information Processing Systems
article_processing_charge: No
arxiv: 1
author:
- first_name: Jean
full_name: Barbier, Jean
last_name: Barbier
- first_name: TianQi
full_name: Hou, TianQi
last_name: Hou
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: Manuel
full_name: Saenz, Manuel
last_name: Saenz
citation:
ama: 'Barbier J, Hou T, Mondelli M, Saenz M. The price of ignorance: How much does
it cost to forget noise structure in low-rank matrix estimation? In: 36th Conference
on Neural Information Processing Systems. Vol 35. Neural Information Processing
Systems Foundation; 2022.'
apa: 'Barbier, J., Hou, T., Mondelli, M., & Saenz, M. (2022). The price of ignorance:
How much does it cost to forget noise structure in low-rank matrix estimation?
In 36th Conference on Neural Information Processing Systems (Vol. 35).
New Orleans, LA, United States: Neural Information Processing Systems Foundation.'
chicago: 'Barbier, Jean, TianQi Hou, Marco Mondelli, and Manuel Saenz. “The Price
of Ignorance: How Much Does It Cost to Forget Noise Structure in Low-Rank Matrix
Estimation?” In 36th Conference on Neural Information Processing Systems,
Vol. 35. Neural Information Processing Systems Foundation, 2022.'
ieee: 'J. Barbier, T. Hou, M. Mondelli, and M. Saenz, “The price of ignorance: How
much does it cost to forget noise structure in low-rank matrix estimation?,” in
36th Conference on Neural Information Processing Systems, New Orleans,
LA, United States, 2022, vol. 35.'
ista: 'Barbier J, Hou T, Mondelli M, Saenz M. 2022. The price of ignorance: How
much does it cost to forget noise structure in low-rank matrix estimation? 36th
Conference on Neural Information Processing Systems. NeurIPS: Neural Information
Processing Systems, Advances in Neural Information Processing Systems, vol. 35.'
mla: 'Barbier, Jean, et al. “The Price of Ignorance: How Much Does It Cost to Forget
Noise Structure in Low-Rank Matrix Estimation?” 36th Conference on Neural Information
Processing Systems, vol. 35, Neural Information Processing Systems Foundation,
2022.'
short: J. Barbier, T. Hou, M. Mondelli, M. Saenz, in:, 36th Conference on Neural
Information Processing Systems, Neural Information Processing Systems Foundation,
2022.
conference:
end_date: 2022-12-09
location: New Orleans, LA, United States
name: 'NeurIPS: Neural Information Processing Systems'
start_date: 2022-11-28
corr_author: '1'
das_tickbox: '1'
date_created: 2023-02-10T13:45:41Z
date_published: 2022-11-20T00:00:00Z
date_updated: 2026-07-07T06:38:45Z
day: '20'
department:
- _id: MaMo
external_id:
arxiv:
- '2205.10009'
intvolume: ' 35'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/2205.10009
month: '11'
oa: 1
oa_version: Preprint
publication: 36th Conference on Neural Information Processing Systems
publication_identifier:
isbn:
- '9781713871088'
issn:
- 1049-5258
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'The price of ignorance: How much does it cost to forget noise structure in
low-rank matrix estimation?'
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2022'
...
---
_id: '12017'
abstract:
- lang: eng
text: 'In the classic adversarial communication problem, two parties communicate
over a noisy channel in the presence of a malicious jamming adversary. The arbitrarily
varying channels (AVCs) offer an elegant framework to study a wide range of interesting
adversary models. The optimal throughput or capacity over such AVCs is intimately
tied to the underlying adversary model; in some cases, capacity is unknown and
the problem is known to be notoriously hard. The omniscient adversary, one which
knows the sender’s entire channel transmission a priori, is one of such classic
models of interest; the capacity under such an adversary remains an exciting open
problem. The myopic adversary is a generalization of that model where the adversary’s
observation may be corrupted over a noisy discrete memoryless channel. Through
the adversary’s myopicity, one can unify the slew of different adversary models,
ranging from the omniscient adversary to one that is completely blind to the transmission
(the latter is the well known oblivious model where the capacity is fully characterized).In
this work, we present new results on the capacity under both the omniscient and
myopic adversary models. We completely characterize the positive capacity threshold
over general AVCs with omniscient adversaries. The characterization is in terms
of two key combinatorial objects: the set of completely positive distributions
and the CP-confusability set. For omniscient AVCs with positive capacity, we present
non-trivial lower and upper bounds on the capacity; unlike some of the previous
bounds, our bounds hold under fairly general input and jamming constraints. Our
lower bound improves upon the generalized Gilbert-Varshamov bound for general
AVCs while the upper bound generalizes the well known Elias-Bassalygo bound (known
for binary and q-ary alphabets). For the myopic AVCs, we build on prior results
known for the so-called sufficiently myopic model, and present new results on
the positive rate communication threshold over the so-called insufficiently myopic
regime (a completely insufficient myopic adversary specializes to an omniscient
adversary). We present interesting examples for the widely studied models of adversarial
bit-flip and bit-erasure channels. In fact, for the bit-flip AVC with additive
adversarial noise as well as random noise, we completely characterize the omniscient
model capacity when the random noise is sufficiently large vis-a-vis the adversary’s
budget.'
article_processing_charge: No
author:
- first_name: Anuj Kumar
full_name: Yadav, Anuj Kumar
last_name: Yadav
- first_name: Mohammadreza
full_name: Alimohammadi, Mohammadreza
last_name: Alimohammadi
- first_name: Yihan
full_name: Zhang, Yihan
id: 2ce5da42-b2ea-11eb-bba5-9f264e9d002c
last_name: Zhang
orcid: 0000-0002-6465-6258
- first_name: Amitalok J.
full_name: Budkuley, Amitalok J.
last_name: Budkuley
- first_name: Sidharth
full_name: Jaggi, Sidharth
last_name: Jaggi
citation:
ama: 'Yadav AK, Alimohammadi M, Zhang Y, Budkuley AJ, Jaggi S. New results on AVCs
with omniscient and myopic adversaries. In: 2022 IEEE International Symposium
on Information Theory. Vol 2022. IEEE; 2022:2535-2540. doi:10.1109/ISIT50566.2022.9834632'
apa: 'Yadav, A. K., Alimohammadi, M., Zhang, Y., Budkuley, A. J., & Jaggi, S.
(2022). New results on AVCs with omniscient and myopic adversaries. In 2022
IEEE International Symposium on Information Theory (Vol. 2022, pp. 2535–2540).
Espoo, Finland: IEEE. https://doi.org/10.1109/ISIT50566.2022.9834632'
chicago: Yadav, Anuj Kumar, Mohammadreza Alimohammadi, Yihan Zhang, Amitalok J.
Budkuley, and Sidharth Jaggi. “New Results on AVCs with Omniscient and Myopic
Adversaries.” In 2022 IEEE International Symposium on Information Theory,
2022:2535–40. IEEE, 2022. https://doi.org/10.1109/ISIT50566.2022.9834632.
ieee: A. K. Yadav, M. Alimohammadi, Y. Zhang, A. J. Budkuley, and S. Jaggi, “New
results on AVCs with omniscient and myopic adversaries,” in 2022 IEEE International
Symposium on Information Theory, Espoo, Finland, 2022, vol. 2022, pp. 2535–2540.
ista: 'Yadav AK, Alimohammadi M, Zhang Y, Budkuley AJ, Jaggi S. 2022. New results
on AVCs with omniscient and myopic adversaries. 2022 IEEE International Symposium
on Information Theory. ISIT: Internation Symposium on Information Theory vol.
2022, 2535–2540.'
mla: Yadav, Anuj Kumar, et al. “New Results on AVCs with Omniscient and Myopic Adversaries.”
2022 IEEE International Symposium on Information Theory, vol. 2022, IEEE,
2022, pp. 2535–40, doi:10.1109/ISIT50566.2022.9834632.
short: A.K. Yadav, M. Alimohammadi, Y. Zhang, A.J. Budkuley, S. Jaggi, in:, 2022
IEEE International Symposium on Information Theory, IEEE, 2022, pp. 2535–2540.
conference:
end_date: 2022-07-01
location: Espoo, Finland
name: 'ISIT: Internation Symposium on Information Theory'
start_date: 2022-06-26
date_created: 2022-09-04T22:02:06Z
date_published: 2022-08-03T00:00:00Z
date_updated: 2026-08-12T06:36:54Z
day: '03'
department:
- _id: MaMo
doi: 10.1109/ISIT50566.2022.9834632
external_id:
isi:
- '001254261902116'
fulldoi: https://doi.org/10.1109/ISIT50566.2022.9834632
intvolume: ' 2022'
isi: 1
language:
- iso: eng
month: '08'
oa_version: None
page: 2535-2540
publication: 2022 IEEE International Symposium on Information Theory
publication_identifier:
isbn:
- '9781665421591'
issn:
- 2157-8095
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: New results on AVCs with omniscient and myopic adversaries
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 2022
year: '2022'
...
---
_id: '12018'
abstract:
- lang: eng
text: We study the problem of characterizing the maximal rates of list decoding
in Euclidean spaces for finite list sizes. For any positive integer L ≥ 2 and
real N > 0, we say that a subset C⊂Rn is an (N,L – 1)-multiple packing or an (N,L–
1)-list decodable code if every Euclidean ball of radius nN−−−√ in ℝ n contains
no more than L − 1 points of C. We study this problem with and without ℓ 2 norm
constraints on C, and derive the best-known lower bounds on the maximal rate for
(N,L−1) multiple packing. Our bounds are obtained via error exponents for list
decoding over Additive White Gaussian Noise (AWGN) channels. We establish a curious
inequality which relates the error exponent, a quantity of average-case nature,
to the list-decoding radius, a quantity of worst-case nature. We derive various
bounds on the error exponent for list decoding in both bounded and unbounded settings
which could be of independent interest beyond multiple packing.
article_processing_charge: No
author:
- first_name: Yihan
full_name: Zhang, Yihan
id: 2ce5da42-b2ea-11eb-bba5-9f264e9d002c
last_name: Zhang
orcid: 0000-0002-6465-6258
- first_name: Shashank
full_name: Vatedka, Shashank
last_name: Vatedka
citation:
ama: 'Zhang Y, Vatedka S. Lower bounds on list decoding capacity using error exponents.
In: 2022 IEEE International Symposium on Information Theory. Vol 2022.
IEEE; 2022:1324-1329. doi:10.1109/ISIT50566.2022.9834815'
apa: 'Zhang, Y., & Vatedka, S. (2022). Lower bounds on list decoding capacity
using error exponents. In 2022 IEEE International Symposium on Information
Theory (Vol. 2022, pp. 1324–1329). Espoo, Finland: IEEE. https://doi.org/10.1109/ISIT50566.2022.9834815'
chicago: Zhang, Yihan, and Shashank Vatedka. “Lower Bounds on List Decoding Capacity
Using Error Exponents.” In 2022 IEEE International Symposium on Information
Theory, 2022:1324–29. IEEE, 2022. https://doi.org/10.1109/ISIT50566.2022.9834815.
ieee: Y. Zhang and S. Vatedka, “Lower bounds on list decoding capacity using error
exponents,” in 2022 IEEE International Symposium on Information Theory,
Espoo, Finland, 2022, vol. 2022, pp. 1324–1329.
ista: 'Zhang Y, Vatedka S. 2022. Lower bounds on list decoding capacity using error
exponents. 2022 IEEE International Symposium on Information Theory. ISIT: International
Symposium on Information Theory vol. 2022, 1324–1329.'
mla: Zhang, Yihan, and Shashank Vatedka. “Lower Bounds on List Decoding Capacity
Using Error Exponents.” 2022 IEEE International Symposium on Information Theory,
vol. 2022, IEEE, 2022, pp. 1324–29, doi:10.1109/ISIT50566.2022.9834815.
short: Y. Zhang, S. Vatedka, in:, 2022 IEEE International Symposium on Information
Theory, IEEE, 2022, pp. 1324–1329.
conference:
end_date: 2022-07-01
location: Espoo, Finland
name: 'ISIT: International Symposium on Information Theory'
start_date: 2022-06-26
date_created: 2022-09-04T22:02:06Z
date_published: 2022-08-03T00:00:00Z
date_updated: 2026-08-12T06:37:09Z
day: '03'
department:
- _id: MaMo
doi: 10.1109/ISIT50566.2022.9834815
external_id:
isi:
- '001254261901080'
fulldoi: https://doi.org/10.1109/ISIT50566.2022.9834815
intvolume: ' 2022'
isi: 1
language:
- iso: eng
month: '08'
oa_version: None
page: 1324-1329
publication: 2022 IEEE International Symposium on Information Theory
publication_identifier:
isbn:
- '9781665421591'
issn:
- 2157-8095
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: Lower bounds on list decoding capacity using error exponents
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 2022
year: '2022'
...
---
_id: '12019'
abstract:
- lang: eng
text: This paper studies combinatorial properties of codes for the Z-channel. A
Z-channel with error fraction τ takes as input a length-n binary codeword and
injects in an adversarial manner up to nτ asymmetric errors, i.e., errors that
only zero out bits but do not flip 0’s to 1’s. It is known that the largest (L
− 1)-list-decodable code for the Z-channel with error fraction τ has exponential
(in n) size if τ is less than a critical value that we call the Plotkin point
and has constant size if τ is larger than the threshold. The (L−1)-list-decoding
Plotkin point is known to be L−1L−1−L−LL−1. In this paper, we show that the largest
(L−1)-list-decodable code ε-above the Plotkin point has size Θ L (ε −3/2 ) for
any L − 1 ≥ 1.
article_processing_charge: No
author:
- first_name: Nikita
full_name: Polyanskii, Nikita
last_name: Polyanskii
- first_name: Yihan
full_name: Zhang, Yihan
id: 2ce5da42-b2ea-11eb-bba5-9f264e9d002c
last_name: Zhang
orcid: 0000-0002-6465-6258
citation:
ama: 'Polyanskii N, Zhang Y. List-decodable zero-rate codes for the Z-channel. In:
2022 IEEE International Symposium on Information Theory. Vol 2022. IEEE;
2022:2553-2558. doi:10.1109/ISIT50566.2022.9834829'
apa: 'Polyanskii, N., & Zhang, Y. (2022). List-decodable zero-rate codes for
the Z-channel. In 2022 IEEE International Symposium on Information Theory
(Vol. 2022, pp. 2553–2558). Espoo, Finland: IEEE. https://doi.org/10.1109/ISIT50566.2022.9834829'
chicago: Polyanskii, Nikita, and Yihan Zhang. “List-Decodable Zero-Rate Codes for
the Z-Channel.” In 2022 IEEE International Symposium on Information Theory,
2022:2553–58. IEEE, 2022. https://doi.org/10.1109/ISIT50566.2022.9834829.
ieee: N. Polyanskii and Y. Zhang, “List-decodable zero-rate codes for the Z-channel,”
in 2022 IEEE International Symposium on Information Theory, Espoo, Finland,
2022, vol. 2022, pp. 2553–2558.
ista: 'Polyanskii N, Zhang Y. 2022. List-decodable zero-rate codes for the Z-channel.
2022 IEEE International Symposium on Information Theory. ISIT: International Symposium
on Information Theory vol. 2022, 2553–2558.'
mla: Polyanskii, Nikita, and Yihan Zhang. “List-Decodable Zero-Rate Codes for the
Z-Channel.” 2022 IEEE International Symposium on Information Theory, vol.
2022, IEEE, 2022, pp. 2553–58, doi:10.1109/ISIT50566.2022.9834829.
short: N. Polyanskii, Y. Zhang, in:, 2022 IEEE International Symposium on Information
Theory, IEEE, 2022, pp. 2553–2558.
conference:
end_date: 2022-07-01
location: Espoo, Finland
name: 'ISIT: International Symposium on Information Theory'
start_date: 2022-06-26
date_created: 2022-09-04T22:02:07Z
date_published: 2022-08-03T00:00:00Z
date_updated: 2026-08-12T06:37:29Z
day: '03'
department:
- _id: MaMo
doi: 10.1109/ISIT50566.2022.9834829
external_id:
isi:
- '001254261902119'
fulldoi: https://doi.org/10.1109/ISIT50566.2022.9834829
intvolume: ' 2022'
isi: 1
language:
- iso: eng
month: '08'
oa_version: None
page: 2553-2558
publication: 2022 IEEE International Symposium on Information Theory
publication_identifier:
isbn:
- '9781665421591'
issn:
- 2157-8095
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: List-decodable zero-rate codes for the Z-channel
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 2022
year: '2022'
...
---
_id: '12273'
abstract:
- lang: eng
text: We study communication in the presence of a jamming adversary where quadratic
power constraints are imposed on the transmitter and the jammer. The jamming signal
is allowed to be a function of the codebook, and a noncausal but noisy observation
of the transmitted codeword. For a certain range of the noise-to-signal ratios
(NSRs) of the transmitter and the jammer, we are able to characterize the capacity
of this channel under deterministic encoding or stochastic encoding, i.e., with
no common randomness between the encoder/decoder pair. For the remaining NSR regimes,
we determine the capacity under the assumption of a small amount of common randomness
(at most 2log(n) bits in one sub-regime, and at most Ω(n) bits in the other sub-regime)
available to the encoder-decoder pair. Our proof techniques involve a novel myopic
list-decoding result for achievability, and a Plotkin-type push attack for the
converse in a subregion of the NSRs, both of which may be of independent interest.
We also give bounds on the strong secrecy capacity of this channel assuming that
the jammer is simultaneously eavesdropping.
article_processing_charge: No
article_type: original
arxiv: 1
author:
- first_name: Yihan
full_name: Zhang, Yihan
id: 2ce5da42-b2ea-11eb-bba5-9f264e9d002c
last_name: Zhang
orcid: 0000-0002-6465-6258
- first_name: Shashank
full_name: Vatedka, Shashank
last_name: Vatedka
- first_name: Sidharth
full_name: Jaggi, Sidharth
last_name: Jaggi
- first_name: Anand D.
full_name: Sarwate, Anand D.
last_name: Sarwate
citation:
ama: Zhang Y, Vatedka S, Jaggi S, Sarwate AD. Quadratically constrained myopic adversarial
channels. IEEE Transactions on Information Theory. 2022;68(8):4901-4948.
doi:10.1109/tit.2022.3167554
apa: Zhang, Y., Vatedka, S., Jaggi, S., & Sarwate, A. D. (2022). Quadratically
constrained myopic adversarial channels. IEEE Transactions on Information Theory.
IEEE. https://doi.org/10.1109/tit.2022.3167554
chicago: Zhang, Yihan, Shashank Vatedka, Sidharth Jaggi, and Anand D. Sarwate. “Quadratically
Constrained Myopic Adversarial Channels.” IEEE Transactions on Information
Theory. IEEE, 2022. https://doi.org/10.1109/tit.2022.3167554.
ieee: Y. Zhang, S. Vatedka, S. Jaggi, and A. D. Sarwate, “Quadratically constrained
myopic adversarial channels,” IEEE Transactions on Information Theory,
vol. 68, no. 8. IEEE, pp. 4901–4948, 2022.
ista: Zhang Y, Vatedka S, Jaggi S, Sarwate AD. 2022. Quadratically constrained myopic
adversarial channels. IEEE Transactions on Information Theory. 68(8), 4901–4948.
mla: Zhang, Yihan, et al. “Quadratically Constrained Myopic Adversarial Channels.”
IEEE Transactions on Information Theory, vol. 68, no. 8, IEEE, 2022, pp.
4901–48, doi:10.1109/tit.2022.3167554.
short: Y. Zhang, S. Vatedka, S. Jaggi, A.D. Sarwate, IEEE Transactions on Information
Theory 68 (2022) 4901–4948.
corr_author: '1'
date_created: 2023-01-16T10:01:19Z
date_published: 2022-08-01T00:00:00Z
date_updated: 2026-08-12T06:41:20Z
day: '01'
department:
- _id: MaMo
doi: 10.1109/tit.2022.3167554
external_id:
arxiv:
- '1801.05951'
isi:
- '000838527100004'
fulldoi: https://doi.org/10.1109/tit.2022.3167554
intvolume: ' 68'
isi: 1
issue: '8'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.48550/arXiv.1801.05951
month: '08'
oa: 1
oa_version: Preprint
page: 4901-4948
publication: IEEE Transactions on Information Theory
publication_identifier:
eissn:
- 1557-9654
issn:
- 0018-9448
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: Quadratically constrained myopic adversarial channels
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 68
year: '2022'
...
---
_id: '12233'
abstract:
- lang: eng
text: A novel recursive list decoding (RLD) algorithm for Reed-Muller (RM) codes
based on successive permutations (SP) of the codeword is presented. A low-complexity
SP scheme applied to a subset of the symmetry group of RM codes is first proposed
to carefully select a good codeword permutation on the fly. Then, the proposed
SP technique is integrated into an improved RLD algorithm that initializes different
decoding paths with random codeword permutations, which are sampled from the full
symmetry group of RM codes. Finally, efficient latency and complexity reduction
schemes are introduced that virtually preserve the error-correction performance
of the proposed decoder. Simulation results demonstrate that at the target frame
error rate of 10−3 for the RM code of length 256 with 163 information bits, the
proposed decoder reduces 6% of the computational complexity and 22% of the decoding
latency of the state-of-the-art semi-parallel simplified successive-cancellation
decoder with fast Hadamard transform (SSC-FHT) that uses 96 permutations from
the full symmetry group of RM codes, while relatively maintaining the error-correction
performance and memory consumption of the semi-parallel permuted SSC-FHT decoder.
article_processing_charge: No
article_type: original
arxiv: 1
author:
- first_name: Nghia
full_name: Doan, Nghia
last_name: Doan
- first_name: Seyyed Ali
full_name: Hashemi, Seyyed Ali
last_name: Hashemi
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: Warren J.
full_name: Gross, Warren J.
last_name: Gross
citation:
ama: Doan N, Hashemi SA, Mondelli M, Gross WJ. Decoding Reed-Muller codes with successive
codeword permutations. IEEE Transactions on Communications. 2022;70(11):7134-7145.
doi:10.1109/tcomm.2022.3211101
apa: Doan, N., Hashemi, S. A., Mondelli, M., & Gross, W. J. (2022). Decoding
Reed-Muller codes with successive codeword permutations. IEEE Transactions
on Communications. IEEE. https://doi.org/10.1109/tcomm.2022.3211101
chicago: Doan, Nghia, Seyyed Ali Hashemi, Marco Mondelli, and Warren J. Gross. “Decoding
Reed-Muller Codes with Successive Codeword Permutations.” IEEE Transactions
on Communications. IEEE, 2022. https://doi.org/10.1109/tcomm.2022.3211101.
ieee: N. Doan, S. A. Hashemi, M. Mondelli, and W. J. Gross, “Decoding Reed-Muller
codes with successive codeword permutations,” IEEE Transactions on Communications,
vol. 70, no. 11. IEEE, pp. 7134–7145, 2022.
ista: Doan N, Hashemi SA, Mondelli M, Gross WJ. 2022. Decoding Reed-Muller codes
with successive codeword permutations. IEEE Transactions on Communications. 70(11),
7134–7145.
mla: Doan, Nghia, et al. “Decoding Reed-Muller Codes with Successive Codeword Permutations.”
IEEE Transactions on Communications, vol. 70, no. 11, IEEE, 2022, pp. 7134–45,
doi:10.1109/tcomm.2022.3211101.
short: N. Doan, S.A. Hashemi, M. Mondelli, W.J. Gross, IEEE Transactions on Communications
70 (2022) 7134–7145.
date_created: 2023-01-16T09:50:38Z
date_published: 2022-11-01T00:00:00Z
date_updated: 2026-08-12T06:41:04Z
day: '01'
department:
- _id: MaMo
doi: 10.1109/tcomm.2022.3211101
external_id:
arxiv:
- '2109.02122'
isi:
- '000937284600006'
fulldoi: https://doi.org/10.1109/tcomm.2022.3211101
intvolume: ' 70'
isi: 1
issue: '11'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: ' https://doi.org/10.48550/arXiv.2109.02122'
month: '11'
oa: 1
oa_version: Preprint
page: 7134-7145
publication: IEEE Transactions on Communications
publication_identifier:
eissn:
- 1558-0857
issn:
- 0090-6778
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: Decoding Reed-Muller codes with successive codeword permutations
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 70
year: '2022'
...
---
_id: '10364'
abstract:
- lang: eng
text: 'This paper characterizes the latency of the simplified successive-cancellation
(SSC) decoding scheme for polar codes under hardware resource constraints. In
particular, when the number of processing elements P that can perform SSC decoding
operations in parallel is limited, as is the case in practice, the latency of
SSC decoding is O(N1-1/μ + N/P log2 log2 N/P), where N is the block length of
the code and μ is the scaling exponent of the channel. Three direct consequences
of this bound are presented. First, in a fully-parallel implementation where P
= N/2, the latency of SSC decoding is O(N1-1/μ), which is sublinear in the block
length. This recovers a result from our earlier work. Second, in a fully-serial
implementation where P = 1, the latency of SSC decoding scales as O(N log2 log2
N). The multiplicative constant is also calculated: we show that the latency of
SSC decoding when P = 1 is given by (2 + o(1))N log2 log2 N. Third, in a semi-parallel
implementation, the smallest P that gives the same latency as that of the fully-parallel
implementation is P = N1/μ. The tightness of our bound on SSC decoding latency
and the applicability of the foregoing results is validated through extensive
simulations.'
acknowledgement: "S. A. Hashemi is supported by a Postdoctoral Fellowship from the
Natural Sciences and\r\nEngineering Research Council of Canada (NSERC) and by Huawei.
M. Mondelli is partially\r\nsupported by the 2019 Lopez-Loreta Prize. A. Fazeli
and A. Vardy were supported in part by\r\nthe National Science Foundation under
Grant CCF-1764104."
article_processing_charge: No
article_type: original
arxiv: 1
author:
- first_name: Seyyed Ali
full_name: Hashemi, Seyyed Ali
last_name: Hashemi
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: Arman
full_name: Fazeli, Arman
last_name: Fazeli
- first_name: Alexander
full_name: Vardy, Alexander
last_name: Vardy
- first_name: John
full_name: Cioffi, John
last_name: Cioffi
- first_name: Andrea
full_name: Goldsmith, Andrea
last_name: Goldsmith
citation:
ama: Hashemi SA, Mondelli M, Fazeli A, Vardy A, Cioffi J, Goldsmith A. Parallelism
versus latency in simplified successive-cancellation decoding of polar codes.
IEEE Transactions on Wireless Communications. 2022;21(6):3909-3920. doi:10.1109/TWC.2021.3125626
apa: Hashemi, S. A., Mondelli, M., Fazeli, A., Vardy, A., Cioffi, J., & Goldsmith,
A. (2022). Parallelism versus latency in simplified successive-cancellation decoding
of polar codes. IEEE Transactions on Wireless Communications. IEEE. https://doi.org/10.1109/TWC.2021.3125626
chicago: Hashemi, Seyyed Ali, Marco Mondelli, Arman Fazeli, Alexander Vardy, John
Cioffi, and Andrea Goldsmith. “Parallelism versus Latency in Simplified Successive-Cancellation
Decoding of Polar Codes.” IEEE Transactions on Wireless Communications.
IEEE, 2022. https://doi.org/10.1109/TWC.2021.3125626.
ieee: S. A. Hashemi, M. Mondelli, A. Fazeli, A. Vardy, J. Cioffi, and A. Goldsmith,
“Parallelism versus latency in simplified successive-cancellation decoding of
polar codes,” IEEE Transactions on Wireless Communications, vol. 21, no.
6. IEEE, pp. 3909–3920, 2022.
ista: Hashemi SA, Mondelli M, Fazeli A, Vardy A, Cioffi J, Goldsmith A. 2022. Parallelism
versus latency in simplified successive-cancellation decoding of polar codes.
IEEE Transactions on Wireless Communications. 21(6), 3909–3920.
mla: Hashemi, Seyyed Ali, et al. “Parallelism versus Latency in Simplified Successive-Cancellation
Decoding of Polar Codes.” IEEE Transactions on Wireless Communications,
vol. 21, no. 6, IEEE, 2022, pp. 3909–20, doi:10.1109/TWC.2021.3125626.
short: S.A. Hashemi, M. Mondelli, A. Fazeli, A. Vardy, J. Cioffi, A. Goldsmith,
IEEE Transactions on Wireless Communications 21 (2022) 3909–3920.
date_created: 2021-11-28T23:01:29Z
date_published: 2022-06-01T00:00:00Z
date_updated: 2026-08-12T06:43:01Z
day: '01'
department:
- _id: MaMo
doi: 10.1109/TWC.2021.3125626
external_id:
arxiv:
- '2012.13378'
isi:
- '000809406400028'
fulldoi: https://doi.org/10.1109/TWC.2021.3125626
intvolume: ' 21'
isi: 1
issue: '6'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/2012.13378
month: '06'
oa: 1
oa_version: Preprint
page: 3909-3920
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: IEEE Transactions on Wireless Communications
publication_identifier:
eissn:
- 1558-2248
issn:
- 1536-1276
publication_status: published
publisher: IEEE
quality_controlled: '1'
related_material:
record:
- id: '10053'
relation: earlier_version
status: public
scopus_import: '1'
status: public
title: Parallelism versus latency in simplified successive-cancellation decoding of
polar codes
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 21
year: '2022'
...
---
OA_place: repository
OA_type: green
_id: '12537'
abstract:
- lang: eng
text: 'The Neural Tangent Kernel (NTK) has emerged as a powerful tool to provide
memorization, optimization and generalization guarantees in deep neural networks.
A line of work has studied the NTK spectrum for two-layer and deep networks with
at least a layer with Ω(N) neurons, N being the number of training samples. Furthermore,
there is increasing evidence suggesting that deep networks with sub-linear layer
widths are powerful memorizers and optimizers, as long as the number of parameters
exceeds the number of samples. Thus, a natural open question is whether the NTK
is well conditioned in such a challenging sub-linear setup. In this paper, we
answer this question in the affirmative. Our key technical contribution is a lower
bound on the smallest NTK eigenvalue for deep networks with the minimum possible
over-parameterization: the number of parameters is roughly Ω(N) and, hence, the
number of neurons is as little as Ω(N−−√). To showcase the applicability of our
NTK bounds, we provide two results concerning memorization capacity and optimization
guarantees for gradient descent training.'
acknowledgement: "The authors were partially supported by the 2019 Lopez-Loreta prize,
and they would like to thank\r\nQuynh Nguyen, Mahdi Soltanolkotabi and Adel Javanmard
for helpful discussions.\r\n"
alternative_title:
- Advances in Neural Information Processing Systems
article_processing_charge: No
arxiv: 1
author:
- first_name: Simone
full_name: Bombari, Simone
id: ca726dda-de17-11ea-bc14-f9da834f63aa
last_name: Bombari
- first_name: Mohammad Hossein
full_name: Amani, Mohammad Hossein
last_name: Amani
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
citation:
ama: 'Bombari S, Amani MH, Mondelli M. Memorization and optimization in deep neural
networks with minimum over-parameterization. In: 36th Conference on Neural
Information Processing Systems. Vol 35. Neural Information Processing Systems
Foundation; 2022:7628-7640.'
apa: 'Bombari, S., Amani, M. H., & Mondelli, M. (2022). Memorization and optimization
in deep neural networks with minimum over-parameterization. In 36th Conference
on Neural Information Processing Systems (Vol. 35, pp. 7628–7640). New Orleans,
LA, United States: Neural Information Processing Systems Foundation.'
chicago: Bombari, Simone, Mohammad Hossein Amani, and Marco Mondelli. “Memorization
and Optimization in Deep Neural Networks with Minimum Over-Parameterization.”
In 36th Conference on Neural Information Processing Systems, 35:7628–40.
Neural Information Processing Systems Foundation, 2022.
ieee: S. Bombari, M. H. Amani, and M. Mondelli, “Memorization and optimization in
deep neural networks with minimum over-parameterization,” in 36th Conference
on Neural Information Processing Systems, New Orleans, LA, United States,
2022, vol. 35, pp. 7628–7640.
ista: 'Bombari S, Amani MH, Mondelli M. 2022. Memorization and optimization in deep
neural networks with minimum over-parameterization. 36th Conference on Neural
Information Processing Systems. NeurIPS: Neural Information Processing Systems,
Advances in Neural Information Processing Systems, vol. 35, 7628–7640.'
mla: Bombari, Simone, et al. “Memorization and Optimization in Deep Neural Networks
with Minimum Over-Parameterization.” 36th Conference on Neural Information
Processing Systems, vol. 35, Neural Information Processing Systems Foundation,
2022, pp. 7628–40.
short: S. Bombari, M.H. Amani, M. Mondelli, in:, 36th Conference on Neural Information
Processing Systems, Neural Information Processing Systems Foundation, 2022, pp.
7628–7640.
conference:
end_date: 2022-12-09
location: New Orleans, LA, United States
name: 'NeurIPS: Neural Information Processing Systems'
start_date: 2022-11-28
corr_author: '1'
date_created: 2023-02-10T13:46:37Z
date_published: 2022-07-24T00:00:00Z
date_updated: 2026-09-21T13:07:01Z
day: '24'
department:
- _id: MaMo
external_id:
arxiv:
- '2205.10217'
intvolume: ' 35'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: ' https://doi.org/10.48550/arXiv.2205.10217'
month: '07'
oa: 1
oa_version: Preprint
page: 7628-7640
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: 36th Conference on Neural Information Processing Systems
publication_identifier:
eissn:
- 1049-5258
isbn:
- '9781713871088'
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
related_material:
record:
- id: '22857'
relation: dissertation_contains
status: public
status: public
title: Memorization and optimization in deep neural networks with minimum over-parameterization
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2022'
...
---
_id: '11420'
abstract:
- lang: eng
text: 'Understanding the properties of neural networks trained via stochastic gradient
descent (SGD) is at the heart of the theory of deep learning. In this work, we
take a mean-field view, and consider a two-layer ReLU network trained via noisy-SGD
for a univariate regularized regression problem. Our main result is that SGD with
vanishingly small noise injected in the gradients is biased towards a simple solution:
at convergence, the ReLU network implements a piecewise linear map of the inputs,
and the number of “knot” points -- i.e., points where the tangent of the ReLU
network estimator changes -- between two consecutive training inputs is at most
three. In particular, as the number of neurons of the network grows, the SGD dynamics
is captured by the solution of a gradient flow and, at convergence, the distribution
of the weights approaches the unique minimizer of a related free energy, which
has a Gibbs form. Our key technical contribution consists in the analysis of the
estimator resulting from this minimizer: we show that its second derivative vanishes
everywhere, except at some specific locations which represent the “knot” points.
We also provide empirical evidence that knots at locations distinct from the data
points might occur, as predicted by our theory.'
acknowledgement: "We would like to thank Mert Pilanci for several exploratory discussions
in the early stage\r\nof the project, Jan Maas for clarifications about Jordan et
al. (1998), and Max Zimmer for\r\nsuggestive numerical experiments. A. Shevchenko
and M. Mondelli are partially supported\r\nby the 2019 Lopez-Loreta Prize. V. Kungurtsev
acknowledges support to the OP VVV\r\nproject CZ.02.1.01/0.0/0.0/16 019/0000765
Research Center for Informatics.\r\n"
article_processing_charge: No
article_type: original
arxiv: 1
author:
- first_name: Aleksandr
full_name: Shevchenko, Aleksandr
id: F2B06EC2-C99E-11E9-89F0-752EE6697425
last_name: Shevchenko
- first_name: Vyacheslav
full_name: Kungurtsev, Vyacheslav
last_name: Kungurtsev
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
citation:
ama: Shevchenko A, Kungurtsev V, Mondelli M. Mean-field analysis of piecewise linear
solutions for wide ReLU networks. Journal of Machine Learning Research.
2022;23(130):1-55.
apa: Shevchenko, A., Kungurtsev, V., & Mondelli, M. (2022). Mean-field analysis
of piecewise linear solutions for wide ReLU networks. Journal of Machine Learning
Research. Journal of Machine Learning Research.
chicago: Shevchenko, Alexander, Vyacheslav Kungurtsev, and Marco Mondelli. “Mean-Field
Analysis of Piecewise Linear Solutions for Wide ReLU Networks.” Journal of
Machine Learning Research. Journal of Machine Learning Research, 2022.
ieee: A. Shevchenko, V. Kungurtsev, and M. Mondelli, “Mean-field analysis of piecewise
linear solutions for wide ReLU networks,” Journal of Machine Learning Research,
vol. 23, no. 130. Journal of Machine Learning Research, pp. 1–55, 2022.
ista: Shevchenko A, Kungurtsev V, Mondelli M. 2022. Mean-field analysis of piecewise
linear solutions for wide ReLU networks. Journal of Machine Learning Research.
23(130), 1–55.
mla: Shevchenko, Alexander, et al. “Mean-Field Analysis of Piecewise Linear Solutions
for Wide ReLU Networks.” Journal of Machine Learning Research, vol. 23,
no. 130, Journal of Machine Learning Research, 2022, pp. 1–55.
short: A. Shevchenko, V. Kungurtsev, M. Mondelli, Journal of Machine Learning Research
23 (2022) 1–55.
corr_author: '1'
date_created: 2022-05-29T22:01:54Z
date_published: 2022-04-01T00:00:00Z
date_updated: 2026-09-25T22:30:18Z
day: '01'
ddc:
- '000'
department:
- _id: MaMo
- _id: DaAl
external_id:
arxiv:
- '2111.02278'
file:
- access_level: open_access
checksum: d4ff5d1affb34848b5c5e4002483fc62
content_type: application/pdf
creator: cchlebak
date_created: 2022-05-30T08:22:55Z
date_updated: 2022-05-30T08:22:55Z
file_id: '11422'
file_name: 21-1365.pdf
file_size: 1521701
relation: main_file
success: 1
file_date_updated: 2022-05-30T08:22:55Z
has_accepted_license: '1'
intvolume: ' 23'
issue: '130'
language:
- iso: eng
month: '04'
oa: 1
oa_version: Published Version
page: 1-55
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: Journal of Machine Learning Research
publication_identifier:
eissn:
- 1533-7928
issn:
- 1532-4435
publication_status: published
publisher: Journal of Machine Learning Research
quality_controlled: '1'
related_material:
link:
- relation: other
url: https://www.jmlr.org/papers/v23/21-1365.html
record:
- id: '17465'
relation: dissertation_contains
status: public
scopus_import: '1'
status: public
title: Mean-field analysis of piecewise linear solutions for wide ReLU networks
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: journal_article
user_id: 8b945eb4-e2f2-11eb-945a-df72226e66a9
volume: 23
year: '2022'
...
---
_id: '13146'
abstract:
- lang: eng
text: 'A recent line of work has analyzed the theoretical properties of deep neural
networks via the Neural Tangent Kernel (NTK). In particular, the smallest eigenvalue
of the NTK has been related to the memorization capacity, the global convergence
of gradient descent algorithms and the generalization of deep nets. However, existing
results either provide bounds in the two-layer setting or assume that the spectrum
of the NTK matrices is bounded away from 0 for multi-layer networks. In this paper,
we provide tight bounds on the smallest eigenvalue of NTK matrices for deep ReLU
nets, both in the limiting case of infinite widths and for finite widths. In the
finite-width setting, the network architectures we consider are fairly general:
we require the existence of a wide layer with roughly order of N neurons, N being
the number of data samples; and the scaling of the remaining layer widths is arbitrary
(up to logarithmic factors). To obtain our results, we analyze various quantities
of independent interest: we give lower bounds on the smallest singular value of
hidden feature matrices, and upper bounds on the Lipschitz constant of input-output
feature maps.'
acknowledgement: The authors would like to thank the anonymous reviewers for their
helpful comments. MM was partially supported by the 2019 Lopez-Loreta Prize. QN
and GM acknowledge support from the European Research Council (ERC) under the European
Union’s Horizon 2020 research and innovation programme (grant agreement no 757983).
article_processing_charge: No
arxiv: 1
author:
- first_name: Quynh
full_name: Nguyen, Quynh
last_name: Nguyen
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: Guido
full_name: Montufar, Guido
last_name: Montufar
citation:
ama: 'Nguyen Q, Mondelli M, Montufar G. Tight bounds on the smallest Eigenvalue
of the neural tangent kernel for deep ReLU networks. In: Proceedings of the
38th International Conference on Machine Learning. Vol 139. ML Research Press;
2021:8119-8129.'
apa: 'Nguyen, Q., Mondelli, M., & Montufar, G. (2021). Tight bounds on the smallest
Eigenvalue of the neural tangent kernel for deep ReLU networks. In Proceedings
of the 38th International Conference on Machine Learning (Vol. 139, pp. 8119–8129).
Virtual: ML Research Press.'
chicago: Nguyen, Quynh, Marco Mondelli, and Guido Montufar. “Tight Bounds on the
Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks.” In Proceedings
of the 38th International Conference on Machine Learning, 139:8119–29. ML
Research Press, 2021.
ieee: Q. Nguyen, M. Mondelli, and G. Montufar, “Tight bounds on the smallest Eigenvalue
of the neural tangent kernel for deep ReLU networks,” in Proceedings of the
38th International Conference on Machine Learning, Virtual, 2021, vol. 139,
pp. 8119–8129.
ista: 'Nguyen Q, Mondelli M, Montufar G. 2021. Tight bounds on the smallest Eigenvalue
of the neural tangent kernel for deep ReLU networks. Proceedings of the 38th International
Conference on Machine Learning. ICML: International Conference on Machine Learning
vol. 139, 8119–8129.'
mla: Nguyen, Quynh, et al. “Tight Bounds on the Smallest Eigenvalue of the Neural
Tangent Kernel for Deep ReLU Networks.” Proceedings of the 38th International
Conference on Machine Learning, vol. 139, ML Research Press, 2021, pp. 8119–29.
short: Q. Nguyen, M. Mondelli, G. Montufar, in:, Proceedings of the 38th International
Conference on Machine Learning, ML Research Press, 2021, pp. 8119–8129.
conference:
end_date: 2021-07-24
location: Virtual
name: 'ICML: International Conference on Machine Learning'
start_date: 2021-07-18
date_created: 2023-06-18T22:00:48Z
date_published: 2021-07-01T00:00:00Z
date_updated: 2025-07-10T11:50:36Z
day: '01'
ddc:
- '000'
department:
- _id: MaMo
external_id:
arxiv:
- '2012.11654'
file:
- access_level: open_access
checksum: 19489cf5e16a0596b1f92e317d97c9b0
content_type: application/pdf
creator: dernst
date_created: 2023-06-19T10:49:12Z
date_updated: 2023-06-19T10:49:12Z
file_id: '13155'
file_name: 2021_PMLR_Nguyen.pdf
file_size: 591332
relation: main_file
success: 1
file_date_updated: 2023-06-19T10:49:12Z
has_accepted_license: '1'
intvolume: ' 139'
language:
- iso: eng
month: '07'
oa: 1
oa_version: Published Version
page: 8119-8129
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: Proceedings of the 38th International Conference on Machine Learning
publication_identifier:
eissn:
- 2640-3498
isbn:
- '9781713845065'
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
scopus_import: '1'
status: public
title: Tight bounds on the smallest Eigenvalue of the neural tangent kernel for deep
ReLU networks
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 139
year: '2021'
...
---
_id: '15254'
abstract:
- lang: eng
text: We consider the problem of reliable communication over a network containing
a hidden myopic adversary who can eavesdrop on some zro links, jam some zwo links,
and do both on some zrw links. We provide the first information-theoretically
tight characterization of the optimal rate of communication possible under all
possible settings of the tuple (zro,zwo,zrw) by providing a novel coding scheme/analysis
for a subset of parameter regimes. In particular, our vanishing-error schemes
bypass the Network Singleton Bound (which requires a zero-error recovery criteria)
in a certain parameter regime where the capacity had been heretofore open. As
a direct corollary we also obtain the capacity of the corresponding problem where
information-theoretic secrecy against eavesdropping is required in addition to
reliable communication.
acknowledgement: The work of Rawad Bitar was supported in part by the Technical University
of Munich—Institute for Advanced Studies, funded by the German Excellence Initiative
and European Union Seventh Framework Programme under Grant 291763. The work of Sidharth
Jaggi was supported by the Hong Kong UGC GRF under Grant 14304418, Grant 14300617,
and Grant 14313116. The work of Yihan Zhang was supported by the European Union’s
Horizon 2020 Research and Innovation Programme under Grant 682203-ERC-[Inf-Speed-Tradeoff].
Preliminary results were presented at IEEE International Symposium on information
Theory (ISIT).
article_processing_charge: No
article_type: original
arxiv: 1
author:
- first_name: Sijie
full_name: Li, Sijie
last_name: Li
- first_name: Rawad
full_name: Bitar, Rawad
last_name: Bitar
- first_name: Sidharth
full_name: Jaggi, Sidharth
last_name: Jaggi
- first_name: Yihan
full_name: Zhang, Yihan
id: 2ce5da42-b2ea-11eb-bba5-9f264e9d002c
last_name: Zhang
orcid: 0000-0002-6465-6258
citation:
ama: Li S, Bitar R, Jaggi S, Zhang Y. Network coding with myopic adversaries. IEEE
Journal on Selected Areas in Information Theory. 2021;2(4):1108-1119. doi:10.1109/JSAIT.2021.3126474
apa: Li, S., Bitar, R., Jaggi, S., & Zhang, Y. (2021). Network coding with myopic
adversaries. IEEE Journal on Selected Areas in Information Theory. IEEE.
https://doi.org/10.1109/JSAIT.2021.3126474
chicago: Li, Sijie, Rawad Bitar, Sidharth Jaggi, and Yihan Zhang. “Network Coding
with Myopic Adversaries.” IEEE Journal on Selected Areas in Information Theory.
IEEE, 2021. https://doi.org/10.1109/JSAIT.2021.3126474.
ieee: S. Li, R. Bitar, S. Jaggi, and Y. Zhang, “Network coding with myopic adversaries,”
IEEE Journal on Selected Areas in Information Theory, vol. 2, no. 4. IEEE,
pp. 1108–1119, 2021.
ista: Li S, Bitar R, Jaggi S, Zhang Y. 2021. Network coding with myopic adversaries.
IEEE Journal on Selected Areas in Information Theory. 2(4), 1108–1119.
mla: Li, Sijie, et al. “Network Coding with Myopic Adversaries.” IEEE Journal
on Selected Areas in Information Theory, vol. 2, no. 4, IEEE, 2021, pp. 1108–19,
doi:10.1109/JSAIT.2021.3126474.
short: S. Li, R. Bitar, S. Jaggi, Y. Zhang, IEEE Journal on Selected Areas in Information
Theory 2 (2021) 1108–1119.
date_created: 2024-03-31T22:01:13Z
date_published: 2021-12-01T00:00:00Z
date_updated: 2024-04-02T08:31:59Z
day: '01'
department:
- _id: MaMo
doi: 10.1109/JSAIT.2021.3126474
external_id:
arxiv:
- '2102.09885'
fulldoi: https://doi.org/10.1109/JSAIT.2021.3126474
intvolume: ' 2'
issue: '4'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.48550/arXiv.2102.09885
month: '12'
oa: 1
oa_version: Preprint
page: 1108-1119
publication: IEEE Journal on Selected Areas in Information Theory
publication_identifier:
eissn:
- 2641-8770
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: Network coding with myopic adversaries
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 2
year: '2021'
...
---
_id: '10593'
abstract:
- lang: eng
text: 'We study the problem of estimating a rank-$1$ signal in the presence of rotationally
invariant noise-a class of perturbations more general than Gaussian noise. Principal
Component Analysis (PCA) provides a natural estimator, and sharp results on its
performance have been obtained in the high-dimensional regime. Recently, an Approximate
Message Passing (AMP) algorithm has been proposed as an alternative estimator
with the potential to improve the accuracy of PCA. However, the existing analysis
of AMP requires an initialization that is both correlated with the signal and
independent of the noise, which is often unrealistic in practice. In this work,
we combine the two methods, and propose to initialize AMP with PCA. Our main result
is a rigorous asymptotic characterization of the performance of this estimator.
Both the AMP algorithm and its analysis differ from those previously derived in
the Gaussian setting: at every iteration, our AMP algorithm requires a specific
term to account for PCA initialization, while in the Gaussian case, PCA initialization
affects only the first iteration of AMP. The proof is based on a two-phase artificial
AMP that first approximates the PCA estimator and then mimics the true AMP. Our
numerical simulations show an excellent agreement between AMP results and theoretical
predictions, and suggest an interesting open direction on achieving Bayes-optimal
performance.'
acknowledgement: "M. Mondelli would like to thank László Erdős for helpful discussions.
M. Mondelli was partially supported by the 2019 Lopez-Loreta Prize. R. Venkataramanan
was partially supported by the Alan Turing Institute under the EPSRC grant EP/N510129/1.\r\n"
article_processing_charge: No
arxiv: 1
author:
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: Ramji
full_name: Venkataramanan, Ramji
last_name: Venkataramanan
citation:
ama: 'Mondelli M, Venkataramanan R. PCA initialization for approximate message passing
in rotationally invariant models. In: 35th Conference on Neural Information
Processing Systems. Vol 35. Neural Information Processing Systems Foundation;
2021:29616-29629.'
apa: 'Mondelli, M., & Venkataramanan, R. (2021). PCA initialization for approximate
message passing in rotationally invariant models. In 35th Conference on Neural
Information Processing Systems (Vol. 35, pp. 29616–29629). Virtual: Neural
Information Processing Systems Foundation.'
chicago: Mondelli, Marco, and Ramji Venkataramanan. “PCA Initialization for Approximate
Message Passing in Rotationally Invariant Models.” In 35th Conference on Neural
Information Processing Systems, 35:29616–29. Neural Information Processing
Systems Foundation, 2021.
ieee: M. Mondelli and R. Venkataramanan, “PCA initialization for approximate message
passing in rotationally invariant models,” in 35th Conference on Neural Information
Processing Systems, Virtual, 2021, vol. 35, pp. 29616–29629.
ista: 'Mondelli M, Venkataramanan R. 2021. PCA initialization for approximate message
passing in rotationally invariant models. 35th Conference on Neural Information
Processing Systems. NeurIPS: Neural Information Processing Systems vol. 35, 29616–29629.'
mla: Mondelli, Marco, and Ramji Venkataramanan. “PCA Initialization for Approximate
Message Passing in Rotationally Invariant Models.” 35th Conference on Neural
Information Processing Systems, vol. 35, Neural Information Processing Systems
Foundation, 2021, pp. 29616–29.
short: M. Mondelli, R. Venkataramanan, in:, 35th Conference on Neural Information
Processing Systems, Neural Information Processing Systems Foundation, 2021, pp.
29616–29629.
conference:
end_date: 2021-12-14
location: Virtual
name: 'NeurIPS: Neural Information Processing Systems'
start_date: 2021-12-06
corr_author: '1'
date_created: 2022-01-03T10:50:02Z
date_published: 2021-12-01T00:00:00Z
date_updated: 2025-04-15T07:50:11Z
day: '01'
department:
- _id: MaMo
external_id:
arxiv:
- '2106.02356'
intvolume: ' 35'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/2106.02356
month: '12'
oa: 1
oa_version: Preprint
page: 29616-29629
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: 35th Conference on Neural Information Processing Systems
publication_identifier:
isbn:
- '9781713845393'
issn:
- 1049-5258
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
scopus_import: '1'
status: public
title: PCA initialization for approximate message passing in rotationally invariant
models
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2021'
...
---
_id: '10594'
abstract:
- lang: eng
text: 'The question of how and why the phenomenon of mode connectivity occurs in
training deep neural networks has gained remarkable attention in the research
community. From a theoretical perspective, two possible explanations have been
proposed: (i) the loss function has connected sublevel sets, and (ii) the solutions
found by stochastic gradient descent are dropout stable. While these explanations
provide insights into the phenomenon, their assumptions are not always satisfied
in practice. In particular, the first approach requires the network to have one
layer with order of N neurons (N being the number of training samples), while
the second one requires the loss to be almost invariant after removing half of
the neurons at each layer (up to some rescaling of the remaining ones). In this
work, we improve both conditions by exploiting the quality of the features at
every intermediate layer together with a milder over-parameterization condition.
More specifically, we show that: (i) under generic assumptions on the features
of intermediate layers, it suffices that the last two hidden layers have order
of N−−√ neurons, and (ii) if subsets of features at each layer are linearly separable,
then no over-parameterization is needed to show the connectivity. Our experiments
confirm that the proposed condition ensures the connectivity of solutions found
by stochastic gradient descent, even in settings where the previous requirements
do not hold.'
acknowledgement: MM was partially supported by the 2019 Lopez-Loreta Prize. QN and
PB acknowledge support from the European Research Council (ERC) under the European
Union’s Horizon 2020 research and innovation programme (grant agreement no 757983).
article_processing_charge: No
arxiv: 1
author:
- first_name: Quynh
full_name: Nguyen, Quynh
last_name: Nguyen
- first_name: Pierre
full_name: Bréchet, Pierre
last_name: Bréchet
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
citation:
ama: 'Nguyen Q, Bréchet P, Mondelli M. When are solutions connected in deep networks?
In: 35th Conference on Neural Information Processing Systems. Vol 35. Neural
Information Processing Systems Foundation; 2021.'
apa: 'Nguyen, Q., Bréchet, P., & Mondelli, M. (2021). When are solutions connected
in deep networks? In 35th Conference on Neural Information Processing Systems
(Vol. 35). Virtual: Neural Information Processing Systems Foundation.'
chicago: Nguyen, Quynh, Pierre Bréchet, and Marco Mondelli. “When Are Solutions
Connected in Deep Networks?” In 35th Conference on Neural Information Processing
Systems, Vol. 35. Neural Information Processing Systems Foundation, 2021.
ieee: Q. Nguyen, P. Bréchet, and M. Mondelli, “When are solutions connected in deep
networks?,” in 35th Conference on Neural Information Processing Systems,
Virtual, 2021, vol. 35.
ista: Nguyen Q, Bréchet P, Mondelli M. 2021. When are solutions connected in deep
networks? 35th Conference on Neural Information Processing Systems. 35th Conference
on Neural Information Processing Systems vol. 35.
mla: Nguyen, Quynh, et al. “When Are Solutions Connected in Deep Networks?” 35th
Conference on Neural Information Processing Systems, vol. 35, Neural Information
Processing Systems Foundation, 2021.
short: Q. Nguyen, P. Bréchet, M. Mondelli, in:, 35th Conference on Neural Information
Processing Systems, Neural Information Processing Systems Foundation, 2021.
conference:
end_date: 2021-12-14
location: Virtual
name: 35th Conference on Neural Information Processing Systems
start_date: 2021-12-06
corr_author: '1'
date_created: 2022-01-03T10:56:20Z
date_published: 2021-12-01T00:00:00Z
date_updated: 2025-04-15T07:50:11Z
day: '01'
department:
- _id: MaMo
external_id:
arxiv:
- '2102.09671'
intvolume: ' 35'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/2102.09671
month: '12'
oa: 1
oa_version: Preprint
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: 35th Conference on Neural Information Processing Systems
publication_identifier:
isbn:
- '9781713845393'
issn:
- 1049-5258
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
status: public
title: When are solutions connected in deep networks?
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2021'
...
---
_id: '10595'
abstract:
- lang: eng
text: 'A recent line of work has analyzed the theoretical properties of deep neural
networks via the Neural Tangent Kernel (NTK). In particular, the smallest eigenvalue
of the NTK has been related to the memorization capacity, the global convergence
of gradient descent algorithms and the generalization of deep nets. However, existing
results either provide bounds in the two-layer setting or assume that the spectrum
of the NTK matrices is bounded away from 0 for multi-layer networks. In this paper,
we provide tight bounds on the smallest eigenvalue of NTK matrices for deep ReLU
nets, both in the limiting case of infinite widths and for finite widths. In the
finite-width setting, the network architectures we consider are fairly general:
we require the existence of a wide layer with roughly order of $N$ neurons, $N$
being the number of data samples; and the scaling of the remaining layer widths
is arbitrary (up to logarithmic factors). To obtain our results, we analyze various
quantities of independent interest: we give lower bounds on the smallest singular
value of hidden feature matrices, and upper bounds on the Lipschitz constant of
input-output feature maps.'
acknowledgement: "The authors would like to thank the anonymous reviewers for their
helpful comments. MM was partially supported\r\nby the 2019 Lopez-Loreta Prize.
QN and GM acknowledge support from the European Research Council (ERC) under\r\nthe
European Union’s Horizon 2020 research and innovation programme (grant agreement
no 757983)."
alternative_title:
- Proceedings of Machine Learning Research
article_processing_charge: No
arxiv: 1
author:
- first_name: Quynh
full_name: Nguyen, Quynh
last_name: Nguyen
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: Guido F
full_name: Montufar, Guido F
last_name: Montufar
citation:
ama: 'Nguyen Q, Mondelli M, Montufar GF. Tight bounds on the smallest eigenvalue
of the neural tangent kernel for deep ReLU networks. In: Meila M, Zhang T, eds.
Proceedings of the 38th International Conference on Machine Learning. Vol
139. ML Research Press; 2021:8119-8129.'
apa: 'Nguyen, Q., Mondelli, M., & Montufar, G. F. (2021). Tight bounds on the
smallest eigenvalue of the neural tangent kernel for deep ReLU networks. In M.
Meila & T. Zhang (Eds.), Proceedings of the 38th International Conference
on Machine Learning (Vol. 139, pp. 8119–8129). Virtual: ML Research Press.'
chicago: Nguyen, Quynh, Marco Mondelli, and Guido F Montufar. “Tight Bounds on the
Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks.” In Proceedings
of the 38th International Conference on Machine Learning, edited by Marina
Meila and Tong Zhang, 139:8119–29. ML Research Press, 2021.
ieee: Q. Nguyen, M. Mondelli, and G. F. Montufar, “Tight bounds on the smallest
eigenvalue of the neural tangent kernel for deep ReLU networks,” in Proceedings
of the 38th International Conference on Machine Learning, Virtual, 2021, vol.
139, pp. 8119–8129.
ista: 'Nguyen Q, Mondelli M, Montufar GF. 2021. Tight bounds on the smallest eigenvalue
of the neural tangent kernel for deep ReLU networks. Proceedings of the 38th International
Conference on Machine Learning. ICML: International Conference on Machine Learning,
Proceedings of Machine Learning Research, vol. 139, 8119–8129.'
mla: Nguyen, Quynh, et al. “Tight Bounds on the Smallest Eigenvalue of the Neural
Tangent Kernel for Deep ReLU Networks.” Proceedings of the 38th International
Conference on Machine Learning, edited by Marina Meila and Tong Zhang, vol.
139, ML Research Press, 2021, pp. 8119–29.
short: Q. Nguyen, M. Mondelli, G.F. Montufar, in:, M. Meila, T. Zhang (Eds.), Proceedings
of the 38th International Conference on Machine Learning, ML Research Press, 2021,
pp. 8119–8129.
conference:
end_date: 2021-07-24
location: Virtual
name: 'ICML: International Conference on Machine Learning'
start_date: 2021-07-18
date_created: 2022-01-03T10:57:49Z
date_published: 2021-01-01T00:00:00Z
date_updated: 2026-06-18T08:43:54Z
day: '01'
ddc:
- '000'
department:
- _id: MaMo
editor:
- first_name: Marina
full_name: Meila, Marina
last_name: Meila
- first_name: Tong
full_name: Zhang, Tong
last_name: Zhang
external_id:
arxiv:
- '2012.11654'
intvolume: ' 139'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: http://proceedings.mlr.press/v139/nguyen21g.html
month: '01'
oa: 1
oa_version: Published Version
page: 8119-8129
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: Proceedings of the 38th International Conference on Machine Learning
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
status: public
title: Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep
ReLU networks
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 139
year: '2021'
...
---
_id: '10598'
abstract:
- lang: eng
text: ' We consider the problem of estimating a signal from measurements obtained
via a generalized linear model. We focus on estimators based on approximate message
passing (AMP), a family of iterative algorithms with many appealing features:
the performance of AMP in the high-dimensional limit can be succinctly characterized
under suitable model assumptions; AMP can also be tailored to the empirical distribution
of the signal entries, and for a wide class of estimation problems, AMP is conjectured
to be optimal among all polynomial-time algorithms. However, a major issue of
AMP is that in many models (such as phase retrieval), it requires an initialization
correlated with the ground-truth signal and independent from the measurement matrix.
Assuming that such an initialization is available is typically not realistic.
In this paper, we solve this problem by proposing an AMP algorithm initialized
with a spectral estimator. With such an initialization, the standard AMP analysis
fails since the spectral estimator depends in a complicated way on the design
matrix. Our main contribution is a rigorous characterization of the performance
of AMP with spectral initialization in the high-dimensional limit. The key technical
idea is to define and analyze a two-phase artificial AMP algorithm that first
produces the spectral estimator, and then closely approximates the iterates of
the true AMP. We also provide numerical results that demonstrate the validity
of the proposed approach. '
acknowledgement: The authors would like to thank Andrea Montanari for helpful discussions.
M. Mondelli was partially supported by the 2019 Lopez-Loreta Prize. R. Venkataramanan
was partially supported by the Alan Turing Institute under the EPSRC grant EP/N510129/1.
alternative_title:
- Proceedings of Machine Learning Research
article_processing_charge: Yes (via OA deal)
arxiv: 1
author:
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: Ramji
full_name: Venkataramanan, Ramji
last_name: Venkataramanan
citation:
ama: 'Mondelli M, Venkataramanan R. Approximate message passing with spectral initialization
for generalized linear models. In: Banerjee A, Fukumizu K, eds. Proceedings
of The 24th International Conference on Artificial Intelligence and Statistics.
Vol 130. ML Research Press; 2021:397-405.'
apa: 'Mondelli, M., & Venkataramanan, R. (2021). Approximate message passing
with spectral initialization for generalized linear models. In A. Banerjee &
K. Fukumizu (Eds.), Proceedings of The 24th International Conference on Artificial
Intelligence and Statistics (Vol. 130, pp. 397–405). Virtual, San Diego, CA,
United States: ML Research Press.'
chicago: Mondelli, Marco, and Ramji Venkataramanan. “Approximate Message Passing
with Spectral Initialization for Generalized Linear Models.” In Proceedings
of The 24th International Conference on Artificial Intelligence and Statistics,
edited by Arindam Banerjee and Kenji Fukumizu, 130:397–405. ML Research Press,
2021.
ieee: M. Mondelli and R. Venkataramanan, “Approximate message passing with spectral
initialization for generalized linear models,” in Proceedings of The 24th International
Conference on Artificial Intelligence and Statistics, Virtual, San Diego,
CA, United States, 2021, vol. 130, pp. 397–405.
ista: 'Mondelli M, Venkataramanan R. 2021. Approximate message passing with spectral
initialization for generalized linear models. Proceedings of The 24th International
Conference on Artificial Intelligence and Statistics. AISTATS: Artificial Intelligence
and Statistics, Proceedings of Machine Learning Research, vol. 130, 397–405.'
mla: Mondelli, Marco, and Ramji Venkataramanan. “Approximate Message Passing with
Spectral Initialization for Generalized Linear Models.” Proceedings of The
24th International Conference on Artificial Intelligence and Statistics, edited
by Arindam Banerjee and Kenji Fukumizu, vol. 130, ML Research Press, 2021, pp.
397–405.
short: M. Mondelli, R. Venkataramanan, in:, A. Banerjee, K. Fukumizu (Eds.), Proceedings
of The 24th International Conference on Artificial Intelligence and Statistics,
ML Research Press, 2021, pp. 397–405.
conference:
end_date: 2021-04-15
location: Virtual, San Diego, CA, United States
name: 'AISTATS: Artificial Intelligence and Statistics'
start_date: 2021-04-13
date_created: 2022-01-03T11:34:22Z
date_published: 2021-04-01T00:00:00Z
date_updated: 2025-04-15T07:50:16Z
day: '01'
department:
- _id: MaMo
editor:
- first_name: Arindam
full_name: Banerjee, Arindam
last_name: Banerjee
- first_name: Kenji
full_name: Fukumizu, Kenji
last_name: Fukumizu
external_id:
arxiv:
- '2010.03460'
intvolume: ' 130'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://proceedings.mlr.press/v130/mondelli21a.html
month: '04'
oa: 1
oa_version: Preprint
page: 397-405
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: Proceedings of The 24th International Conference on Artificial Intelligence
and Statistics
publication_identifier:
issn:
- 2640-3498
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
related_material:
record:
- id: '12480'
relation: later_version
status: public
scopus_import: '1'
status: public
title: Approximate message passing with spectral initialization for generalized linear
models
type: conference
user_id: 3E5EF7F0-F248-11E8-B48F-1D18A9856A87
volume: 130
year: '2021'
...
---
OA_place: repository
OA_type: green
_id: '9002'
abstract:
- lang: eng
text: ' We prove that, for the binary erasure channel (BEC), the polar-coding paradigm
gives rise to codes that not only approach the Shannon limit but do so under the
best possible scaling of their block length as a function of the gap to capacity.
This result exhibits the first known family of binary codes that attain both optimal
scaling and quasi-linear complexity of encoding and decoding. Our proof is based
on the construction and analysis of binary polar codes with large kernels. When
communicating reliably at rates within ε>0 of capacity, the code length n often
scales as O(1/εμ), where the constant μ is called the scaling exponent. It is
known that the optimal scaling exponent is μ=2, and it is achieved by random linear
codes. The scaling exponent of conventional polar codes (based on the 2×2 kernel)
on the BEC is μ=3.63. This falls far short of the optimal scaling guaranteed by
random codes. Our main contribution is a rigorous proof of the following result:
for the BEC, there exist ℓ×ℓ binary kernels, such that polar codes constructed
from these kernels achieve scaling exponent μ(ℓ) that tends to the optimal value
of 2 as ℓ grows. We furthermore characterize precisely how large ℓ needs to be
as a function of the gap between μ(ℓ) and 2. The resulting binary codes maintain
the recursive structure of conventional polar codes, and thereby achieve construction
complexity O(n) and encoding/decoding complexity O(nlogn).'
article_processing_charge: No
article_type: original
arxiv: 1
author:
- first_name: Arman
full_name: Fazeli, Arman
last_name: Fazeli
- first_name: Hamed
full_name: Hassani, Hamed
last_name: Hassani
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: Alexander
full_name: Vardy, Alexander
last_name: Vardy
citation:
ama: 'Fazeli A, Hassani H, Mondelli M, Vardy A. Binary linear codes with optimal
scaling: Polar codes with large kernels. IEEE Transactions on Information Theory.
2021;67(9):5693-5710. doi:10.1109/TIT.2020.3038806'
apa: 'Fazeli, A., Hassani, H., Mondelli, M., & Vardy, A. (2021). Binary linear
codes with optimal scaling: Polar codes with large kernels. IEEE Transactions
on Information Theory. IEEE. https://doi.org/10.1109/TIT.2020.3038806'
chicago: 'Fazeli, Arman, Hamed Hassani, Marco Mondelli, and Alexander Vardy. “Binary
Linear Codes with Optimal Scaling: Polar Codes with Large Kernels.” IEEE Transactions
on Information Theory. IEEE, 2021. https://doi.org/10.1109/TIT.2020.3038806.'
ieee: 'A. Fazeli, H. Hassani, M. Mondelli, and A. Vardy, “Binary linear codes with
optimal scaling: Polar codes with large kernels,” IEEE Transactions on Information
Theory, vol. 67, no. 9. IEEE, pp. 5693–5710, 2021.'
ista: 'Fazeli A, Hassani H, Mondelli M, Vardy A. 2021. Binary linear codes with
optimal scaling: Polar codes with large kernels. IEEE Transactions on Information
Theory. 67(9), 5693–5710.'
mla: 'Fazeli, Arman, et al. “Binary Linear Codes with Optimal Scaling: Polar Codes
with Large Kernels.” IEEE Transactions on Information Theory, vol. 67,
no. 9, IEEE, 2021, pp. 5693–710, doi:10.1109/TIT.2020.3038806.'
short: A. Fazeli, H. Hassani, M. Mondelli, A. Vardy, IEEE Transactions on Information
Theory 67 (2021) 5693–5710.
date_created: 2021-01-10T23:01:18Z
date_published: 2021-09-01T00:00:00Z
date_updated: 2025-09-10T09:59:12Z
day: '01'
department:
- _id: MaMo
doi: 10.1109/TIT.2020.3038806
external_id:
arxiv:
- '1711.01339'
isi:
- '000690440100007'
fulldoi: https://doi.org/10.1109/TIT.2020.3038806
intvolume: ' 67'
isi: 1
issue: '9'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.48550/arXiv.1711.01339
month: '09'
oa: 1
oa_version: Preprint
page: 5693-5710
publication: IEEE Transactions on Information Theory
publication_identifier:
eissn:
- 1557-9654
issn:
- 0018-9448
publication_status: published
publisher: IEEE
quality_controlled: '1'
related_material:
record:
- id: '6665'
relation: earlier_version
status: public
scopus_import: '1'
status: public
title: 'Binary linear codes with optimal scaling: Polar codes with large kernels'
type: journal_article
user_id: 317138e5-6ab7-11ef-aa6d-ffef3953e345
volume: 67
year: '2021'
...
---
_id: '10599'
abstract:
- lang: eng
text: A two-part successive syndrome-check decoding of polar codes is proposed with
the first part successively refining the received codeword and the second part
checking its syndrome. A new formulation of the successive-cancellation (SC) decoding
algorithm is presented that allows for successively refining the received codeword
by comparing the log-likelihood ratio value of a frozen bit with its predefined
value. The syndrome of the refined received codeword is then checked for possible
errors. In case there are no errors, the decoding process is terminated. Otherwise,
the decoder continues to refine the received codeword. The proposed method is
extended to the case of SC list (SCL) decoding by terminating the decoding process
when the syndrome of the best candidate in the list indicates no errors. Simulation
results show that the proposed method reduces the time-complexity of SC and SCL
decoders and their fast variants, especially at high signal-to-noise ratios.
acknowledgement: This work is supported in part by ONR grant N00014-18-1-2191. S.
A. Hashemi was supported by a Postdoctoral Fellowship from the Natural Sciences
and Engineering Research Council of Canada (NSERC) and by Huawei. M. Mondelli was
partially supported by the 2019 Lopez-Loreta Prize.
article_processing_charge: No
arxiv: 1
author:
- first_name: Seyyed Ali
full_name: Hashemi, Seyyed Ali
last_name: Hashemi
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: John
full_name: Cioffi, John
last_name: Cioffi
- first_name: Andrea
full_name: Goldsmith, Andrea
last_name: Goldsmith
citation:
ama: 'Hashemi SA, Mondelli M, Cioffi J, Goldsmith A. Successive syndrome-check decoding
of polar codes. In: Proceedings of the 55th Asilomar Conference on Signals,
Systems, and Computers. Vol 2021-October. IEEE; 2021:943-947. doi:10.1109/IEEECONF53345.2021.9723394'
apa: 'Hashemi, S. A., Mondelli, M., Cioffi, J., & Goldsmith, A. (2021). Successive
syndrome-check decoding of polar codes. In Proceedings of the 55th Asilomar
Conference on Signals, Systems, and Computers (Vol. 2021–October, pp. 943–947).
Virtual, Pacific Grove, CA, United States: IEEE. https://doi.org/10.1109/IEEECONF53345.2021.9723394'
chicago: Hashemi, Seyyed Ali, Marco Mondelli, John Cioffi, and Andrea Goldsmith.
“Successive Syndrome-Check Decoding of Polar Codes.” In Proceedings of the
55th Asilomar Conference on Signals, Systems, and Computers, 2021–October:943–47.
IEEE, 2021. https://doi.org/10.1109/IEEECONF53345.2021.9723394.
ieee: S. A. Hashemi, M. Mondelli, J. Cioffi, and A. Goldsmith, “Successive syndrome-check
decoding of polar codes,” in Proceedings of the 55th Asilomar Conference on
Signals, Systems, and Computers, Virtual, Pacific Grove, CA, United States,
2021, vol. 2021–October, pp. 943–947.
ista: 'Hashemi SA, Mondelli M, Cioffi J, Goldsmith A. 2021. Successive syndrome-check
decoding of polar codes. Proceedings of the 55th Asilomar Conference on Signals,
Systems, and Computers. ACSSC: Asilomar Conference on Signals, Systems, and Computers
vol. 2021–October, 943–947.'
mla: Hashemi, Seyyed Ali, et al. “Successive Syndrome-Check Decoding of Polar Codes.”
Proceedings of the 55th Asilomar Conference on Signals, Systems, and Computers,
vol. 2021–October, IEEE, 2021, pp. 943–47, doi:10.1109/IEEECONF53345.2021.9723394.
short: S.A. Hashemi, M. Mondelli, J. Cioffi, A. Goldsmith, in:, Proceedings of the
55th Asilomar Conference on Signals, Systems, and Computers, IEEE, 2021, pp. 943–947.
conference:
end_date: 2021-11-03
location: Virtual, Pacific Grove, CA, United States
name: 'ACSSC: Asilomar Conference on Signals, Systems, and Computers'
start_date: 2021-10-31
date_created: 2022-01-03T11:39:51Z
date_published: 2021-11-01T00:00:00Z
date_updated: 2026-08-12T06:36:04Z
day: '01'
department:
- _id: MaMo
doi: 10.1109/IEEECONF53345.2021.9723394
external_id:
arxiv:
- '2112.00057'
fulldoi: https://doi.org/10.1109/IEEECONF53345.2021.9723394
language:
- iso: eng
main_file_link:
- open_access: '1'
url: ' https://doi.org/10.48550/arXiv.2112.00057'
month: '11'
oa: 1
oa_version: Preprint
page: 943-947
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: Proceedings of the 55th Asilomar Conference on Signals, Systems, and
Computers
publication_identifier:
isbn:
- '9781665458283'
issn:
- 1058-6393
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: Successive syndrome-check decoding of polar codes
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 2021-October
year: '2021'
...
---
OA_place: repository
OA_type: green
_id: '10597'
abstract:
- lang: eng
text: We thank Emmanuel Abbe and Min Ye for providing us the implementation of RPA
decoding. D. Fathollahi and M. Mondelli are partially supported by the 2019 Lopez-Loreta
Prize. N. Farsad is supported by Discovery Grant from the Natural Sciences and
Engineering Research Council of Canada (NSERC) and Canada Foundation for Innovation
(CFI), John R. Evans Leader Fund. S. A. Hashemi is supported by a Postdoctoral
Fellowship from NSERC.
article_processing_charge: No
arxiv: 1
author:
- first_name: Dorsa
full_name: Fathollahi, Dorsa
id: 712472af-8a7e-11f1-848f-9768b0c2fdf8
last_name: Fathollahi
- first_name: Nariman
full_name: Farsad, Nariman
last_name: Farsad
- first_name: Seyyed Ali
full_name: Hashemi, Seyyed Ali
last_name: Hashemi
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
citation:
ama: 'Fathollahi D, Farsad N, Hashemi SA, Mondelli M. Sparse multi-decoder recursive
projection aggregation for Reed-Muller codes. In: 2021 IEEE International Symposium
on Information Theory. IEEE; 2021:1082-1087. doi:10.1109/isit45174.2021.9517887'
apa: 'Fathollahi, D., Farsad, N., Hashemi, S. A., & Mondelli, M. (2021). Sparse
multi-decoder recursive projection aggregation for Reed-Muller codes. In 2021
IEEE International Symposium on Information Theory (pp. 1082–1087). Virtual,
Melbourne, Australia: IEEE. https://doi.org/10.1109/isit45174.2021.9517887'
chicago: Fathollahi, Dorsa, Nariman Farsad, Seyyed Ali Hashemi, and Marco Mondelli.
“Sparse Multi-Decoder Recursive Projection Aggregation for Reed-Muller Codes.”
In 2021 IEEE International Symposium on Information Theory, 1082–87. IEEE,
2021. https://doi.org/10.1109/isit45174.2021.9517887.
ieee: D. Fathollahi, N. Farsad, S. A. Hashemi, and M. Mondelli, “Sparse multi-decoder
recursive projection aggregation for Reed-Muller codes,” in 2021 IEEE International
Symposium on Information Theory, Virtual, Melbourne, Australia, 2021, pp.
1082–1087.
ista: 'Fathollahi D, Farsad N, Hashemi SA, Mondelli M. 2021. Sparse multi-decoder
recursive projection aggregation for Reed-Muller codes. 2021 IEEE International
Symposium on Information Theory. ISIT: International Symposium on Information
Theory, 1082–1087.'
mla: Fathollahi, Dorsa, et al. “Sparse Multi-Decoder Recursive Projection Aggregation
for Reed-Muller Codes.” 2021 IEEE International Symposium on Information Theory,
IEEE, 2021, pp. 1082–87, doi:10.1109/isit45174.2021.9517887.
short: D. Fathollahi, N. Farsad, S.A. Hashemi, M. Mondelli, in:, 2021 IEEE International
Symposium on Information Theory, IEEE, 2021, pp. 1082–1087.
conference:
end_date: 2021-07-20
location: Virtual, Melbourne, Australia
name: 'ISIT: International Symposium on Information Theory'
start_date: 2021-07-12
date_created: 2022-01-03T11:31:26Z
date_published: 2021-09-01T00:00:00Z
date_updated: 2026-08-12T06:36:19Z
day: '01'
department:
- _id: MaMo
doi: 10.1109/isit45174.2021.9517887
external_id:
arxiv:
- '2011.12882'
isi:
- '000701502201029'
fulldoi: https://doi.org/10.1109/isit45174.2021.9517887
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/2011.12882
month: '09'
oa: 1
oa_version: Preprint
page: 1082-1087
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: 2021 IEEE International Symposium on Information Theory
publication_identifier:
eisbn:
- 978-1-5386-8209-8
isbn:
- 978-1-5386-8210-4
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: Sparse multi-decoder recursive projection aggregation for Reed-Muller codes
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2021'
...