A statistical approach to Monte Carlo denoising - Sakai, Hiroyuki; Freude, Christian; Auzinger, Thomas; Hahn, David; Wimmer, Michael (Proceedings - SIGGRAPH Asia 2024 Conference Papers)
The stochastic nature of modern Monte Carlo (MC) rendering methods inevitably produces noise in rendered images for a practical number of samples per pixel. The problem of denoising these images has been widely studied, with most recent methods relying on data-driven, pretrained neural networks. In contrast, in this paper we propose a statistical approach to the denoising problem, treating each pixel as a random variable and reasoning about its distribution. Considering a pixel of the noisy rendered image, we formulate fast pair-wise statistical tests—based on online estimators—to decide which of the nearby pixels to exclude from the denoising filter. We show that for symmetric pixel weights and normally distributed samples, the classical Welch t-test is optimal in terms of mean squared error. We then show how to extend this result to handle non-normal distributions, using more recent confidence-interval formulations in combination with the Box-Cox transformation. Our results show that our statistical denoising approach matches the performance of state-of-the-art neural image denoising without having to resort to any computation-intensive pretraining. Furthermore, our approach easily generalizes to other quantities besides pixel intensity, which we demonstrate by showing additional applications to Russian roulette path termination and multiple importance sampling.

The polynomial sieve and equal sums of like polynomials - Browning, Timothy D (International Mathematics Research Notices)
This paper corrects an error in an earlier work of the author.

Can LLMs separate instructions from data? And what do we even mean by that? - Zverev, Egor; Abdelnabi, Sahar; Tabesh, Soroush; Fritz, Mario; Lampert, Christoph (arXiv)
Instruction-tuned Large Language Models (LLMs) show impressive results in numerous practical applications, but they lack essential safety features that are common in other areas of computer science, particularly an explicit separation of instructions and data. This makes them vulnerable to manipulations such as indirect prompt injections and generally unsuitable for safety-critical tasks. Surprisingly, there is currently no established definition or benchmark to quantify this phenomenon. In this work, we close this gap by introducing a formal measure for instruction-data separation and an empirical variant that is calculable from a model's outputs. We also present a new dataset, SEP, that allows estimating the measure for real-world models. Our results on various LLMs show that the problem of instruction-data separation is real: all models fail to achieve high separation, and canonical mitigation techniques, such as prompt engineering and fine-tuning, either fail to substantially improve separation or reduce model utility. The source code and SEP dataset are openly accessible at https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed.

Data - The unreasonable efficiency of total rain evaporation removal in triggering convective self-aggregation - Hwong, Yi-Ling; Muller, Caroline J
This repository contains the data, scripts, SAM codes and files required to reproduce the results of the manuscript "The Unreasonable Efficiency of Total Rain Evaporation Removal in Triggering Convective Self-Aggregation" submitted to the Geophysical Research Letters (GRL). Brief description of project: This project aims to examine the impact of rain evaporation removal or reduction in the planetary boundary layer (PBL) on convective self aggregation (CSA). Non-rotating radiative-convective equilibrium (RCE) simulations were conducted with the System for Atmospheric Modeling (SAM) cloud resolving model. Rain evaporation in the lowest 1 km was progressively reduced and the effect on CSA was investigated. The physical processes underlying this type of aggregation (referred to in the manuscript as no-evaporation CSA, or NE-CSA) were analyzed and described. The default SAM code base (version 6.10.8) can be downloaded from here: http://rossby.msrc.sunysb.edu/~marat/SAM.html

Continual learning: Applications and the road forward - Verwimp, Eli; Aljundi, Rahaf; Ben-David, Shai; Bethge, Matthias; Cossu, Andrea; Gepperth, Alexander; Hayes, Tyler L.; Hüllermeier, Eyke; Kanan, Christopher; Kudithipudi, Dhireesha; Lampert, Christoph; Mundt, Martin; et al. (Transactions on Machine Learning Research)
Continual learning is a subfield of machine learning, which aims to allow machine learning models to continuously learn on new data, by accumulating knowledge without forgetting what was learned in the past. In this work, we take a step back, and ask: "Why should one care about continual learning in the first place?". We set the stage by examining recent continual learning papers published at four major machine learning conferences, and show that memory-constrained settings dominate the field. Then, we discuss five open problems in machine learning, and even though they might seem unrelated to continual learning at first sight, we show that continual learning will inevitably be part of their solution. These problems are model editing, personalization and specialization, on-device learning, faster (re-)training and reinforcement learning. Finally, by comparing the desiderata from these unsolved problems and the current assumptions in continual learning, we highlight and discuss four future directions for continual learning research. We hope that this work offers an interesting perspective on the future of continual learning, while displaying its potential value and the paths we have to pursue in order to make it successful. This work is the result of the many discussions the authors had at the Dagstuhl seminar on Deep Continual Learning, in March 2023.

Anomalous multi-gap topological phases in periodically driven quantum rotors - Karle, Volker; Lemeshko, Mikhail; Bouhon, Adrien; Slager, Robert-Jan; Ünal, F. Nur (arXiv)
We demonstrate that periodically driven quantum rotors provide a promising and broadly applicable platform to implement multi-gap topological phases, where groups of bands can acquire topological invariants due to non-Abelian braiding of band degeneracies. By adiabatically varying the periodic kicks to the rotor we find nodal-line braiding, which causes sign flips of topological charges of band nodes and can prevent them from annihilating, indicated by non-zero values of the %non-Abelian patch Euler class. In particular, we report on the emergence of an anomalous Dirac string phase arising in the strongly driven regime, a truly out-of-equilibrium phase of the quantum rotor. This phase emanates from braiding processes involving all (quasienergy) gaps and manifests itself with edge states at zero angular momentum. Our results reveal direct applications in state-of-the-art experiments of quantum rotors, such as linear molecules driven by periodic far-off-resonant laser pulses or artificial quantum rotors in optical lattices, whose extensive versatility offers precise modification and observation of novel non-Abelian topological properties.

Large-scale population data enrichment in mental health research - Nees, Frauke; Renner, Paul; Holz, Nathalie E.; Polemiti, Elli; Siehl, Sebastian; Hese, Sören; Schepanski, Kerstin; Schumann, Gunter; Walter, Henrik; Heinz, Andreas; Ralser, Markus; Twardziok, Sven; et al. (Nature Mental Health)
This Comment explores new approaches to enrich large-scale population data, including incorporating macro-environmental and digital health measures.

Acute and circadian feedforward regulation of agouti-related peptide hunger neurons - Douglass, Amelia May Barnett; Kucukdereli, Hakan; Madara, Joseph C.; Wang, Daqing; Wu, Chen; Lowenstein, Elijah D.; Tao, Jenkang; Lowell, Bradford B. (Cell Metabolism)
When food is freely available, eating occurs without energy deficit. While agouti-related peptide (AgRP) neurons are likely involved, their activation is thought to require negative energy balance. To investigate this, we implemented long-term, continuous in vivo fiber-photometry recordings in mice. We discovered new forms of AgRP neuron regulation, including fast pre-ingestive decreases in activity and unexpectedly rapid activation by fasting. Furthermore, AgRP neuron activity has a circadian rhythm that peaks concurrent with the daily feeding onset. Importantly, this rhythm persists when nutrition is provided via constant-rate gastric infusions. Hence, it is not secondary to a circadian feeding rhythm. The AgRP neuron rhythm is driven by the circadian clock, the suprachiasmatic nucleus (SCN), as SCN ablation abolishes the circadian rhythm in AgRP neuron activity and feeding. The SCN activates AgRP neurons via excitatory afferents from thyrotrophin-releasing hormone-expressing neurons in the dorsomedial hypothalamus (DMHTrh neurons) to drive daily feeding rhythms.

The 3-isogeny selmer groups of the elliptic curves y2=x3+n2 - Chan, Yik Tung (International Mathematics Research Notices)
Consider the family of elliptic curves En:y2=x3+n2, where n varies over positive cubefree integers. There is a rational 3-isogeny ϕ from En to E^n:y2=x3−27n2 and a dual isogeny ϕ^:E^n→En. We show that for almost all n, the rank of Selϕ(En) is 0, and the rank of Selϕ^(E^n) is determined by the number of prime factors of n that are congruent to 2mod3 and the congruence class of nmod9.

MICROADAM: Accurate adaptive optimization with low space overhead and provable convergence - Modoranu, Ionut-Vlad; Safaryan, Mher; Malinovsky, Grigory; Kurtic, Eldar; Robert, Thomas; Richtárik, Peter; Alistarh, Dan-Adrian (38th Conference on Neural Information Processing Systems)
We propose a new variant of the Adam optimizer [Kingma and Ba, 2014] called MICROADAM that specifically minimizes memory overheads, while maintaining theoretical convergence guarantees. We achieve this by compressing the gradient information before it is fed into the optimizer state, thereby reducing its memory footprint significantly. We control the resulting compression error via a novel instance of the classical error feedback mechanism from distributed optimization [Seide et al., 2014, Alistarh et al., 2018, Karimireddy et al., 2019] in which the error correction information is itself compressed to allow for practical memory gains. We prove that the resulting approach maintains theoretical convergence guarantees competitive to those of AMSGrad, while providing good practical performance. Specifically, we show that MICROADAM can be implemented efficiently on GPUs: on both million-scale (BERT) and billion-scale (LLaMA) models, MICROADAM provides practical convergence competitive to that of the uncompressed Adam baseline, with lower memory usage and similar running time. Our code is available at https://github.com/IST-DASLab/MicroAdam.

QuaRot: Outlier-free 4-bit inference in rotated LLMs - Ashkboos, Saleh; Mohtashami, Amirkeivan; Croci, Maximilian L.; Li, Bo; Cameron, Pashmina; Jaggi, Martin; Alistarh, Dan-Adrian; Hoefler, Torsten; Hensman, James (38th Conference on Neural Information Processing Systems)
We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism, and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4 bits, without any channels identified for retention in higher precision. Our 4-bit quantized LLAMA2-70B model has losses of at most 0.47 WikiText-2 perplexity and retains 99% of the zero-shot performance. We also show that QuaRot can provide lossless 6 and 8 bit LLAMA-2 models without any calibration data using round-to-nearest quantization. Code is available at github.com/spcl/QuaRot.

Continual counting with gradual privacy expiration - Andersson, Joel Daniel; Henzinger, Monika H; Pagh, Rasmus; Steiner, Teresa Anna; Upadhyay, Jalaj (38th Conference on Neural Information Processing Systems)
Differential privacy with gradual expiration models the setting where data items arrive in a stream and at a given time t the privacy loss guaranteed for a data item seen at time (t − d) is εg(d), where g is a monotonically non-decreasing function. We study the fundamental continual (binary) counting problem where each data item consists of a bit, and the algorithm needs to output at each time step the sum of all the bits streamed so far. For a stream of length T and privacy without expiration continual counting is possible with maximum (over all time steps) additive error O(log2 (T)/ε) and the best known lower bound is Ω(log(T)/ε); closing this gap is a challenging open problem. We show that the situation is very different for privacy with gradual expiration by giving upper and lower bounds for a large set of expiration functions g. Specifically, our algorithm achieves an additive error of O(log(T)/ε) for a large set of privacy expiration functions. We also give a lower bound that shows that if C is the additive error of any ε-DP algorithm for this problem, then the product of C and the privacy expiration function after 2C steps must be Ω(log(T)/ε). Our algorithm matches this lower bound as its additive error is O(log(T)/ε), even when g(2C) = O(1). Our empirical evaluation shows that we achieve a slowly growing privacy loss with significantly smaller empirical privacy loss for large values of d than a natural baseline algorithm.

Latent functional maps: A spectral framework for representation alignment - Fumero, Marco; Pegoraro, Marco; Maiorca, Valentino; Locatello, Francesco; Rodolà, Emanuele (38th Conference on Neural Information Processing Systems)
Neural models learn data representations that lie on low-dimensional manifolds, yet modeling the relation between these representational spaces is an ongoing challenge. By integrating spectral geometry principles into neural modeling, we show that this problem can be better addressed in the functional domain, mitigating complexity, while enhancing interpretability and performances on downstream tasks. To this end, we introduce a multi-purpose framework to the representation learning community, which allows to: (i) compare different spaces in an interpretable way and measure their intrinsic similarity; (ii) find correspondences between them, both in unsupervised and weakly supervised settings, and (iii) to effectively transfer representations between distinct spaces. We validate our framework on various applications, ranging from stitching to retrieval tasks, and on multiple modalities, demonstrating that Latent Functional Maps can serve as a swiss-army knife for representation alignment

C2M3: Cycle-consistent multi-model merging - Crisostomi, Donato; Fumero, Marco; Baieri, Daniele; Bernard, Florian; Rodolà, Emanuele (38th Conference on Neural Information Processing Systems)
In this paper, we present a novel data-free method for merging neural networks in weight space. Differently from most existing works, our method optimizes for the permutations of network neurons globally across all layers. This allows us to enforce cycle consistency of the permutations when merging n ≥ 3 models, allowing circular compositions of permutations to be computed without accumulating error along the path. We qualitatively and quantitatively motivate the need for such a constraint, showing its benefits when merging sets of models in scenarios spanning varying architectures and datasets. We finally show that, when coupled with activation renormalization, our approach yields the best results in the task.

The iterative optimal brain surgeon: Faster sparse recovery by leveraging second-order information - Wu, Diyuan; Modoranu, Ionut-Vlad; Safaryan, Mher; Kuznedelev, Denis; Alistarh, Dan-Adrian (38th Conference on Neural Information Processing Systems)
The rising footprint of machine learning has led to a focus on imposing model sparsity as a means of reducing computational and memory costs. For deep neural networks (DNNs), the state-of-the-art accuracy-vs-sparsity is achieved by heuristics inspired by the classical Optimal Brain Surgeon (OBS) framework [LeCun et al., 1989, Hassibi and Stork, 1992, Hassibi et al., 1993], which leverages loss curvature information to make better pruning decisions. Yet, these results still lack a solid theoretical understanding, and it is unclear whether they can be improved by leveraging connections to the wealth of work on sparse recovery algorithms. In this paper, we draw new connections between these two areas and present new sparse recovery algorithms inspired by the OBS framework that comes with theoretical guarantees under reasonable assumptions and have strong practical performance. Specifically, our work starts from the observation that we can leverage curvature information in OBS-like fashion upon the projection step of classic iterative sparse recovery algorithms such as IHT. We show for the first time that this leads both to improved convergence bounds under standard assumptions. Furthermore, we present extensions of this approach to the practical task of obtaining accurate sparse DNNs, and validate it experimentally at scale for Transformer-based models on vision and language tasks.

PV-tuning: Beyond straight-through estimation for extreme LLM compression - Malinovskii, Vladimir; Mazur, Denis; Ilin, Ivan; Kuznedelev, Denis; Burlachenko, Konstantin; Yi, Kai; Alistarh, Dan-Adrian; Richtarik, Peter (38th Conference on Neural Information Processing Systems)
There has been significant interest in "extreme" compression of large language models (LLMs), i.e. to 1-2 bits per parameter, which allows such models to be executed efficiently on resource-constrained devices. Existing work focused on improved one-shot quantization techniques and weight representations; yet, purely post-training approaches are reaching diminishing returns in terms of the accuracy-vs-bit-width trade-off. State-of-the-art quantization methods such as QuIP# and AQLM include fine-tuning (part of) the compressed parameters over a limited amount of calibration data; however, such fine-tuning techniques over compressed weights often make exclusive use of straight-through estimators (STE), whose performance is not well-understood in this setting. In this work, we question the use of STE for extreme LLM compression, showing that it can be sub-optimal, and perform a systematic study of quantization-aware fine-tuning strategies for LLMs.We propose PV-Tuning - a representation-agnostic framework that generalizes and improves upon existing fine-tuning strategies, and provides convergence guarantees in restricted cases.On the practical side, when used for 1-2 bit vector quantization, PV-Tuning outperforms prior techniques for highly-performant models such as Llama and Mistral. Using PV-Tuning, we achieve the first Pareto-optimal quantization for Llama-2 family models at 2 bits per parameter.

Spinal neuron diversity scales exponentially with swim-to-limb transformation during frog metamorphosis - Vijatovic, David; Toma, Florina Alexandra; Harrington, Zoe P; Sommer, Christoph M; Hauschild, Robert; Trevisan, Alexandra J.; Chapman, Phillip; Julseth, Mara; Brenner-Morton, Susan; Gabitto, Mariano I.; Dasen, Jeremy S.; Bikoff, Jay B.; et al. (bioRxiv)
Vertebrates exhibit a wide range of motor behaviors, ranging from swimming to complex limb-based movements. Here we take advantage of frog metamorphosis, which captures a swim-to-limb-based movement transformation during the development of a single organism, to explore changes in the underlying spinal circuits. We find that the tadpole spinal cord contains small and largely homogeneous populations of motor neurons (MNs) and V1 interneurons (V1s) at early escape swimming stages. These neuronal populations only modestly increase in number and subtype heterogeneity with the emergence of free swimming. In contrast, during frog metamorphosis and the emergence of limb movement, there is a dramatic expansion of MN and V1 interneuron number and transcriptional heterogeneity, culminating in cohorts of neurons that exhibit striking molecular similarity to mammalian motor circuits. CRISPR/Cas9-mediated gene disruption of the limb MN and V1 determinants FoxP1 and Engrailed-1, respectively, results in severe but selective deficits in tail and limb function. Our work thus demonstrates that neural diversity scales exponentially with increasing behavioral complexity and illustrates striking evolutionary conservation in the molecular organization and function of motor circuits across species.

Eigenstate thermalisation at the edge for Wigner matrices - Cipolloni, Giorgio; Erdös, László; Henheik, Sven Joscha (arXiv)
We prove the Eigenstate Thermalisation Hypothesis for Wigner matrices uniformly in the entire spectrum, in particular near the spectral edges, with a bound on the fluctuation that is optimal for any observable. This complements earlier works of Cipolloni et. al. (Comm. Math. Phys. 388, 2021; Forum Math., Sigma 10, 2022) and Benigni et. al. (Comm. Math. Phys. 391, 2022; arXiv: 2303.11142) that were restricted either to the bulk of the spectrum or to special observables. As a main ingredient, we prove a new multi-resolvent local law that optimally accounts for the edge scaling.

Cusp universality for correlated random matrices - Erdös, László; Henheik, Sven Joscha; Riabov, Volodymyr (arXiv)
For correlated real symmetric or complex Hermitian random matrices, we prove that the local eigenvalue statistics at any cusp singularity are universal. Since the density of states typically exhibits only square root edge or cubic root cusp singularities, our result completes the proof of the Wigner-Dyson-Mehta universality conjecture in all spectral regimes for a very general class of random matrices. Previously only the bulk and the edge universality were established in this generality [arXiv:1804.07744], while cusp universality was proven only for Wigner-type matrices with independent entries [arXiv:1809.03971, arXiv:1811.04055]. As our main technical input, we prove an optimal local law at the cusp using the Zigzag strategy, a recursive tandem of the characteristic flow method and a Green function comparison argument. Moreover, our proof of the optimal local law holds uniformly in the spectrum, thus also re-establishing universality of the local eigenvalue statistics in the previously studied bulk [arXiv:1705.10661] and edge [arXiv:1804.07744] regimes.

Multi-band superconductors have enhanced critical temperatures - Henheik, Sven Joscha; Langmann, Edwin; Lauritsen, Asbjørn Bækgaard (arXiv)
We introduce a multi-band BCS free energy functional and prove that for a multi-band superconductor the effect of inter-band coupling can only increase the critical temperature, irrespective of its attractive or repulsive nature and its strength. Further, for weak coupling and weaker inter-band coupling, we prove that the dependence of the increase in critical temperature on the inter-band coupling is (1) linear, if there are two or more equally strongly superconducting bands, or (2) quadratic, if there is only one dominating band.