---
_id: '7130'
abstract:
- lang: eng
  text: "We show that statistical criticality, i.e. the occurrence of power law frequency
    distributions, arises in samples that are maximally informative about the underlying
    generating process. In order to reach this conclusion, we first identify the frequency
    with which different outcomes occur in a sample, as the variable carrying useful
    information on the generative process. The entropy of the frequency, that we call
    relevance, provides an upper bound to the number of informative bits. This differs
    from the entropy of the data, that we take as a measure of resolution. Samples
    that maximise relevance at a given resolution—that we call maximally informative
    samples—exhibit statistical criticality. In particular, Zipf's law arises at the
    optimal trade-off between resolution (i.e. compression) and relevance. As a byproduct,
    we derive a bound of the maximal number of parameters that can be estimated from
    a dataset, in the absence of prior knowledge on the generative model.\r\n\r\nFurthermore,
    we relate criticality to the statistical properties of the representation of the
    data generating process. We show that, as a consequence of the concentration property
    of the asymptotic equipartition property, representations that are maximally informative
    about the data generating process are characterised by an exponential distribution
    of energy levels. This arises from a principle of minimal entropy, that is conjugate
    of the maximum entropy principle in statistical mechanics. This explains why statistical
    criticality requires no parameter fine tuning in maximally informative samples."
acknowledgement: We acknowledge interesting discussions with M Abbott, E Aurell, J
  Barbier, R Monasson, T Mora, I Nemenman, N Tishby and R Zecchina. This research
  was supported by the Kavli Foundation and the Centre of Excellence scheme of the
  Research Council of Norway (Centre for Neural Computation) (RJC and YR), by the
  Basic Science Research Program through the National Research Foundation of Korea
  (NRF), funded by the Ministry of Education (2016R1D1A1B03932264) (JJ), and, in part,
  by the ICTP through the OEA-AC-98 (JS).
article_number: '063402'
article_processing_charge: No
article_type: original
arxiv: 1
author:
- first_name: Ryan J
  full_name: Cubero, Ryan J
  id: 850B2E12-9CD4-11E9-837F-E719E6697425
  last_name: Cubero
  orcid: 0000-0003-0002-1867
- first_name: Junghyo
  full_name: Jo, Junghyo
  last_name: Jo
- first_name: Matteo
  full_name: Marsili, Matteo
  last_name: Marsili
- first_name: Yasser
  full_name: Roudi, Yasser
  last_name: Roudi
- first_name: Juyong
  full_name: Song, Juyong
  last_name: Song
citation:
  ama: 'Cubero RJ, Jo J, Marsili M, Roudi Y, Song J. Statistical criticality arises
    in most informative representations. <i>Journal of Statistical Mechanics: Theory
    and Experiment</i>. 2019;2019(6). doi:<a href="https://doi.org/10.1088/1742-5468/ab16c8">10.1088/1742-5468/ab16c8</a>'
  apa: 'Cubero, R. J., Jo, J., Marsili, M., Roudi, Y., &#38; Song, J. (2019). Statistical
    criticality arises in most informative representations. <i>Journal of Statistical
    Mechanics: Theory and Experiment</i>. IOP Publishing. <a href="https://doi.org/10.1088/1742-5468/ab16c8">https://doi.org/10.1088/1742-5468/ab16c8</a>'
  chicago: 'Cubero, Ryan J, Junghyo Jo, Matteo Marsili, Yasser Roudi, and Juyong Song.
    “Statistical Criticality Arises in Most Informative Representations.” <i>Journal
    of Statistical Mechanics: Theory and Experiment</i>. IOP Publishing, 2019. <a
    href="https://doi.org/10.1088/1742-5468/ab16c8">https://doi.org/10.1088/1742-5468/ab16c8</a>.'
  ieee: 'R. J. Cubero, J. Jo, M. Marsili, Y. Roudi, and J. Song, “Statistical criticality
    arises in most informative representations,” <i>Journal of Statistical Mechanics:
    Theory and Experiment</i>, vol. 2019, no. 6. IOP Publishing, 2019.'
  ista: 'Cubero RJ, Jo J, Marsili M, Roudi Y, Song J. 2019. Statistical criticality
    arises in most informative representations. Journal of Statistical Mechanics:
    Theory and Experiment. 2019(6), 063402.'
  mla: 'Cubero, Ryan J., et al. “Statistical Criticality Arises in Most Informative
    Representations.” <i>Journal of Statistical Mechanics: Theory and Experiment</i>,
    vol. 2019, no. 6, 063402, IOP Publishing, 2019, doi:<a href="https://doi.org/10.1088/1742-5468/ab16c8">10.1088/1742-5468/ab16c8</a>.'
  short: 'R.J. Cubero, J. Jo, M. Marsili, Y. Roudi, J. Song, Journal of Statistical
    Mechanics: Theory and Experiment 2019 (2019).'
date_created: 2019-11-26T22:36:09Z
date_published: 2019-06-17T00:00:00Z
date_updated: 2021-01-12T08:11:57Z
day: '17'
doi: 10.1088/1742-5468/ab16c8
extern: '1'
external_id:
  arxiv:
  - '1808.00249'
intvolume: '      2019'
issue: '6'
keyword:
- optimization under uncertainty
- source coding
- large deviation
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://arxiv.org/abs/1808.00249
month: '06'
oa: 1
oa_version: Preprint
publication: 'Journal of Statistical Mechanics: Theory and Experiment'
publication_identifier:
  issn:
  - 1742-5468
publication_status: published
publisher: IOP Publishing
quality_controlled: '1'
status: public
title: Statistical criticality arises in most informative representations
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 2019
year: '2019'
...