---
_id: '6944'
abstract:
- lang: eng
  text: 'We study the problem of automatically detecting if a given multi-class classifier
    operates outside of its specifications (out-of-specs), i.e. on input data from
    a different distribution than what it was trained for. This is an important problem
    to solve on the road towards creating reliable computer vision systems for real-world
    applications, because the quality of a classifier’s predictions cannot be guaranteed
    if it operates out-of-specs. Previously proposed methods for out-of-specs detection
    make decisions on the level of single inputs. This, however, is insufficient to
    achieve low false positive rate and high false negative rates at the same time.
    In this work, we describe a new procedure named KS(conf), based on statistical
    reasoning. Its main component is a classical Kolmogorov–Smirnov test that is applied
    to the set of predicted confidence values for batches of samples. Working with
    batches instead of single samples allows increasing the true positive rate without
    negatively affecting the false positive rate, thereby overcoming a crucial limitation
    of single sample tests. We show by extensive experiments using a variety of convolutional
    network architectures and datasets that KS(conf) reliably detects out-of-specs
    situations even under conditions where other tests fail. It furthermore has a
    number of properties that make it an excellent candidate for practical deployment:
    it is easy to implement, adds almost no overhead to the system, works with any
    classifier that outputs confidence scores, and requires no a priori knowledge
    about how the data distribution could change.'
article_processing_charge: Yes (via OA deal)
article_type: original
author:
- first_name: Rémy
  full_name: Sun, Rémy
  last_name: Sun
- first_name: Christoph
  full_name: Lampert, Christoph
  id: 40C20FD2-F248-11E8-B48F-1D18A9856A87
  last_name: Lampert
  orcid: 0000-0001-8622-7887
citation:
  ama: 'Sun R, Lampert C. KS(conf): A light-weight test if a multiclass classifier
    operates outside of its specifications. <i>International Journal of Computer Vision</i>.
    2020;128(4):970-995. doi:<a href="https://doi.org/10.1007/s11263-019-01232-x">10.1007/s11263-019-01232-x</a>'
  apa: 'Sun, R., &#38; Lampert, C. (2020). KS(conf): A light-weight test if a multiclass
    classifier operates outside of its specifications. <i>International Journal of
    Computer Vision</i>. Springer Nature. <a href="https://doi.org/10.1007/s11263-019-01232-x">https://doi.org/10.1007/s11263-019-01232-x</a>'
  chicago: 'Sun, Rémy, and Christoph Lampert. “KS(Conf): A Light-Weight Test If a
    Multiclass Classifier Operates Outside of Its Specifications.” <i>International
    Journal of Computer Vision</i>. Springer Nature, 2020. <a href="https://doi.org/10.1007/s11263-019-01232-x">https://doi.org/10.1007/s11263-019-01232-x</a>.'
  ieee: 'R. Sun and C. Lampert, “KS(conf): A light-weight test if a multiclass classifier
    operates outside of its specifications,” <i>International Journal of Computer
    Vision</i>, vol. 128, no. 4. Springer Nature, pp. 970–995, 2020.'
  ista: 'Sun R, Lampert C. 2020. KS(conf): A light-weight test if a multiclass classifier
    operates outside of its specifications. International Journal of Computer Vision.
    128(4), 970–995.'
  mla: 'Sun, Rémy, and Christoph Lampert. “KS(Conf): A Light-Weight Test If a Multiclass
    Classifier Operates Outside of Its Specifications.” <i>International Journal of
    Computer Vision</i>, vol. 128, no. 4, Springer Nature, 2020, pp. 970–95, doi:<a
    href="https://doi.org/10.1007/s11263-019-01232-x">10.1007/s11263-019-01232-x</a>.'
  short: R. Sun, C. Lampert, International Journal of Computer Vision 128 (2020) 970–995.
corr_author: '1'
date_created: 2019-10-14T09:14:28Z
date_published: 2020-04-01T00:00:00Z
date_updated: 2025-04-15T07:10:25Z
day: '01'
ddc:
- '004'
department:
- _id: ChLa
doi: 10.1007/s11263-019-01232-x
ec_funded: 1
external_id:
  isi:
  - '000494406800001'
file:
- access_level: open_access
  checksum: 155e63edf664dcacb3bdc1c2223e606f
  content_type: application/pdf
  creator: dernst
  date_created: 2019-11-26T10:30:02Z
  date_updated: 2020-07-14T12:47:45Z
  file_id: '7110'
  file_name: 2019_IJCV_Sun.pdf
  file_size: 1715072
  relation: main_file
file_date_updated: 2020-07-14T12:47:45Z
has_accepted_license: '1'
intvolume: '       128'
isi: 1
issue: '4'
language:
- iso: eng
license: https://creativecommons.org/licenses/by/4.0/
month: '04'
oa: 1
oa_version: Published Version
page: 970-995
project:
- _id: 2532554C-B435-11E9-9278-68D0E5697425
  call_identifier: FP7
  grant_number: '308036'
  name: Lifelong Learning of Visual Scene Understanding
- _id: B67AFEDC-15C9-11EA-A837-991A96BB2854
  name: IST Austria Open Access Fund
publication: International Journal of Computer Vision
publication_identifier:
  eissn:
  - 1573-1405
  issn:
  - 0920-5691
publication_status: published
publisher: Springer Nature
quality_controlled: '1'
related_material:
  link:
  - relation: erratum
    url: https://doi.org/10.1007/s11263-019-01262-5
  record:
  - id: '6482'
    relation: earlier_version
    status: public
scopus_import: '1'
status: public
title: 'KS(conf): A light-weight test if a multiclass classifier operates outside
  of its specifications'
tmp:
  image: /images/cc_by.png
  legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
  name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
  short: CC BY (4.0)
type: journal_article
user_id: 3E5EF7F0-F248-11E8-B48F-1D18A9856A87
volume: 128
year: '2020'
...