{"publication_identifier":{"eissn":["1573-1405"],"issn":["0920-5691"]},"month":"04","doi":"10.1007/s11263-019-01232-x","abstract":[{"text":"We study the problem of automatically detecting if a given multi-class classifier operates outside of its specifications (out-of-specs), i.e. on input data from a different distribution than what it was trained for. This is an important problem to solve on the road towards creating reliable computer vision systems for real-world applications, because the quality of a classifier’s predictions cannot be guaranteed if it operates out-of-specs. Previously proposed methods for out-of-specs detection make decisions on the level of single inputs. This, however, is insufficient to achieve low false positive rate and high false negative rates at the same time. In this work, we describe a new procedure named KS(conf), based on statistical reasoning. Its main component is a classical Kolmogorov–Smirnov test that is applied to the set of predicted confidence values for batches of samples. Working with batches instead of single samples allows increasing the true positive rate without negatively affecting the false positive rate, thereby overcoming a crucial limitation of single sample tests. We show by extensive experiments using a variety of convolutional network architectures and datasets that KS(conf) reliably detects out-of-specs situations even under conditions where other tests fail. It furthermore has a number of properties that make it an excellent candidate for practical deployment: it is easy to implement, adds almost no overhead to the system, works with any classifier that outputs confidence scores, and requires no a priori knowledge about how the data distribution could change.","lang":"eng"}],"ddc":["004"],"isi":1,"has_accepted_license":"1","_id":"6944","status":"public","oa_version":"Published Version","volume":128,"publication":"International Journal of Computer Vision","file":[{"content_type":"application/pdf","file_name":"2019_IJCV_Sun.pdf","date_updated":"2020-07-14T12:47:45Z","file_id":"7110","creator":"dernst","checksum":"155e63edf664dcacb3bdc1c2223e606f","date_created":"2019-11-26T10:30:02Z","relation":"main_file","access_level":"open_access","file_size":1715072}],"issue":"4","citation":{"mla":"Sun, Rémy, and Christoph Lampert. “KS(Conf): A Light-Weight Test If a Multiclass Classifier Operates Outside of Its Specifications.” <i>International Journal of Computer Vision</i>, vol. 128, no. 4, Springer Nature, 2020, pp. 970–95, doi:<a href=\"https://doi.org/10.1007/s11263-019-01232-x\">10.1007/s11263-019-01232-x</a>.","apa":"Sun, R., &#38; Lampert, C. (2020). KS(conf): A light-weight test if a multiclass classifier operates outside of its specifications. <i>International Journal of Computer Vision</i>. Springer Nature. <a href=\"https://doi.org/10.1007/s11263-019-01232-x\">https://doi.org/10.1007/s11263-019-01232-x</a>","ieee":"R. Sun and C. Lampert, “KS(conf): A light-weight test if a multiclass classifier operates outside of its specifications,” <i>International Journal of Computer Vision</i>, vol. 128, no. 4. Springer Nature, pp. 970–995, 2020.","ama":"Sun R, Lampert C. KS(conf): A light-weight test if a multiclass classifier operates outside of its specifications. <i>International Journal of Computer Vision</i>. 2020;128(4):970-995. doi:<a href=\"https://doi.org/10.1007/s11263-019-01232-x\">10.1007/s11263-019-01232-x</a>","chicago":"Sun, Rémy, and Christoph Lampert. “KS(Conf): A Light-Weight Test If a Multiclass Classifier Operates Outside of Its Specifications.” <i>International Journal of Computer Vision</i>. Springer Nature, 2020. <a href=\"https://doi.org/10.1007/s11263-019-01232-x\">https://doi.org/10.1007/s11263-019-01232-x</a>.","short":"R. Sun, C. Lampert, International Journal of Computer Vision 128 (2020) 970–995.","ista":"Sun R, Lampert C. 2020. KS(conf): A light-weight test if a multiclass classifier operates outside of its specifications. International Journal of Computer Vision. 128(4), 970–995."},"title":"KS(conf): A light-weight test if a multiclass classifier operates outside of its specifications","day":"01","intvolume":"       128","scopus_import":"1","date_created":"2019-10-14T09:14:28Z","author":[{"full_name":"Sun, Rémy","last_name":"Sun","first_name":"Rémy"},{"orcid":"0000-0001-8622-7887","last_name":"Lampert","id":"40C20FD2-F248-11E8-B48F-1D18A9856A87","full_name":"Lampert, Christoph","first_name":"Christoph"}],"ec_funded":1,"article_processing_charge":"Yes (via OA deal)","type":"journal_article","page":"970-995","publisher":"Springer Nature","department":[{"_id":"ChLa"}],"publication_status":"published","date_updated":"2024-02-22T14:57:30Z","year":"2020","tmp":{"image":"/images/cc_by.png","legal_code_url":"https://creativecommons.org/licenses/by/4.0/legalcode","name":"Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)","short":"CC BY (4.0)"},"file_date_updated":"2020-07-14T12:47:45Z","date_published":"2020-04-01T00:00:00Z","external_id":{"isi":["000494406800001"]},"project":[{"call_identifier":"FP7","_id":"2532554C-B435-11E9-9278-68D0E5697425","name":"Lifelong Learning of Visual Scene Understanding","grant_number":"308036"},{"name":"IST Austria Open Access Fund","_id":"B67AFEDC-15C9-11EA-A837-991A96BB2854"}],"quality_controlled":"1","language":[{"iso":"eng"}],"oa":1,"article_type":"original","related_material":{"link":[{"relation":"erratum","url":"https://doi.org/10.1007/s11263-019-01262-5"}],"record":[{"id":"6482","status":"public","relation":"earlier_version"}]},"user_id":"3E5EF7F0-F248-11E8-B48F-1D18A9856A87"}