1-Lipschitz neural networks are more expressive with N-activations

preprint 1-Lipschitz neural networks are more expressive with N-activations draft Bernd Prach author 2D561D42-C427-11E9-89B4-9C1AE6697425 Christoph Lampert author 40C20FD2-F248-11E8-B48F-1D18A9856A870000-0001-8622-7887 GradSch department ChLa department A crucial property for achieving secure, trustworthy and interpretable deep learning systems is their robustness: small changes to a system's inputs should not result in large changes to its outputs. Mathematically, this means one strives for networks with a small Lipschitz constant. Several recent works have focused on how to construct such Lipschitz networks, typically by imposing constraints on the weight matrices. In this work, we study an orthogonal aspect, namely the role of the activation function. We show that commonly used activation functions, such as MaxMin, as well as all piece-wise linear ones with two segments unnecessarily restrict the class of representable functions, even in the simplest one-dimensional setting. We furthermore introduce the new N-activation function that is provably more expressive than currently popular activation functions. We provide code at this https URL. https://creativecommons.org/licenses/by/4.0/ 2023 eng arXiv 2311.0610310.48550/ARXIV.2311.06103 https://research-explorer.ista.ac.at/record/19759 B. Prach and C. Lampert, “1-Lipschitz neural networks are more expressive with N-activations,” arXiv. . Prach, Bernd, and Christoph Lampert. “1-Lipschitz Neural Networks Are More Expressive with N-Activations.” ArXiv, n.d. <a href="https://doi.org/10.48550/ARXIV.2311.06103">https://doi.org/10.48550/ARXIV.2311.06103</a>. Prach, Bernd, and Christoph Lampert. “1-Lipschitz Neural Networks Are More Expressive with N-Activations.” ArXiv, 2311.06103, doi:<a href="https://doi.org/10.48550/ARXIV.2311.06103">10.48550/ARXIV.2311.06103</a>. B. Prach, C. Lampert, ArXiv (n.d.). Prach, B., & Lampert, C. (n.d.). 1-Lipschitz neural networks are more expressive with N-activations. arXiv. <a href="https://doi.org/10.48550/ARXIV.2311.06103">https://doi.org/10.48550/ARXIV.2311.06103</a> Prach B, Lampert C. 1-Lipschitz neural networks are more expressive with N-activations. arXiv, 2311.06103. Prach B, Lampert C. 1-Lipschitz neural networks are more expressive with N-activations. arXiv. doi:<a href="https://doi.org/10.48550/ARXIV.2311.06103">10.48550/ARXIV.2311.06103</a> 150392024-02-28T17:59:32Z2026-04-07T11:49:51Z