Backward-compatible prediction updates: A probabilistic approach

Träuble F, Kügelgen J von, Kleindessner M, Locatello F, Schölkopf B, Gehler P. 2021. Backward-compatible prediction updates: A probabilistic approach. 35th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems vol. 34, 116–128.

Download (ext.)
Conference Paper | Published | English
Author
Träuble, Frederik; Kügelgen, Julius von; Kleindessner, Matthäus; Locatello, FrancescoISTA ; Schölkopf, Bernhard; Gehler, Peter
Department
Abstract
When machine learning systems meet real world applications, accuracy is only one of several requirements. In this paper, we assay a complementary perspective originating from the increasing availability of pre-trained and regularly improving state-of-the-art models. While new improved models develop at a fast pace, downstream tasks vary more slowly or stay constant. Assume that we have a large unlabelled data set for which we want to maintain accurate predictions. Whenever a new and presumably better ML models becomes available, we encounter two problems: (i) given a limited budget, which data points should be re-evaluated using the new model?; and (ii) if the new predictions differ from the current ones, should we update? Problem (i) is about compute cost, which matters for very large data sets and models. Problem (ii) is about maintaining consistency of the predictions, which can be highly relevant for downstream applications; our demand is to avoid negative flips, i.e., changing correct to incorrect predictions. In this paper, we formalize the Prediction Update Problem and present an efficient probabilistic approach as answer to the above questions. In extensive experiments on standard classification benchmark data sets, we show that our method outperforms alternative strategies along key metrics for backward-compatible prediction updates.
Publishing Year
Date Published
2021-07-02
Proceedings Title
35th Conference on Neural Information Processing Systems
Volume
34
Page
116-128
Conference
NeurIPS: Neural Information Processing Systems
Conference Location
Virtual
Conference Date
2021-12-07 – 2021-12-10
IST-REx-ID

Cite this

Träuble F, Kügelgen J von, Kleindessner M, Locatello F, Schölkopf B, Gehler P. Backward-compatible prediction updates: A probabilistic approach. In: 35th Conference on Neural Information Processing Systems. Vol 34. ; 2021:116-128.
Träuble, F., Kügelgen, J. von, Kleindessner, M., Locatello, F., Schölkopf, B., & Gehler, P. (2021). Backward-compatible prediction updates: A probabilistic approach. In 35th Conference on Neural Information Processing Systems (Vol. 34, pp. 116–128). Virtual.
Träuble, Frederik, Julius von Kügelgen, Matthäus Kleindessner, Francesco Locatello, Bernhard Schölkopf, and Peter Gehler. “Backward-Compatible Prediction Updates: A Probabilistic Approach.” In 35th Conference on Neural Information Processing Systems, 34:116–28, 2021.
F. Träuble, J. von Kügelgen, M. Kleindessner, F. Locatello, B. Schölkopf, and P. Gehler, “Backward-compatible prediction updates: A probabilistic approach,” in 35th Conference on Neural Information Processing Systems, Virtual, 2021, vol. 34, pp. 116–128.
Träuble F, Kügelgen J von, Kleindessner M, Locatello F, Schölkopf B, Gehler P. 2021. Backward-compatible prediction updates: A probabilistic approach. 35th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems vol. 34, 116–128.
Träuble, Frederik, et al. “Backward-Compatible Prediction Updates: A Probabilistic Approach.” 35th Conference on Neural Information Processing Systems, vol. 34, 2021, pp. 116–28.
All files available under the following license(s):
Copyright Statement:
This Item is protected by copyright and/or related rights. [...]

Link(s) to Main File(s)
Access Level
OA Open Access

Export

Marked Publications

Open Data ISTA Research Explorer

Sources

arXiv 2107.01057

Search this title in

Google Scholar
ISBN Search