Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Reliable Confidence Predictions Using Conformal Prediction
Stockholm University, Faculty of Social Sciences, Department of Computer and Systems Sciences.
2016 (English)In: Advances in Knowledge Discovery and Data Mining: 20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, 2016, Proceedings, Part I / [ed] James Bailey, Latifur Khan, Takashi Washio, Gill Dobbie, Joshua Zhexue Huang, Ruili Wang, Springer, 2016, p. 77-88Conference paper, Published paper (Refereed)
Abstract [en]

Conformal classifiers output confidence prediction regions, i.e., multi-valued predictions that are guaranteed to contain the true output value of each test pattern with some predefined probability. In order to fully utilize the predictions provided by a conformal classifier, it is essential that those predictions are reliable, i.e., that a user is able to assess the quality of the predictions made. Although conformal classifiers are statistically valid by default, the error probability of the prediction regions output are dependent on their size in such a way that smaller, and thus potentially more interesting, predictions are more likely to be incorrect. This paper proposes, and evaluates, a method for producing refined error probability estimates of prediction regions, that takes their size into account. The end result is a binary conformal confidence predictor that is able to provide accurate error probability estimates for those prediction regions containing only a single class label.

Place, publisher, year, edition, pages
Springer, 2016. p. 77-88
Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 9651
National Category
Information Systems
Research subject
Computer and Systems Sciences
Identifiers
URN: urn:nbn:se:su:diva-137489DOI: 10.1007/978-3-319-31753-3_7ISI: 000389019500007ISBN: 978-3-319-31752-6 (print)ISBN: 978-3-319-31753-3 (print)OAI: oai:DiVA.org:su-137489DiVA, id: diva2:1062764
Conference
20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, 2016
Available from: 2017-01-08 Created: 2017-01-08 Last updated: 2022-02-28Bibliographically approved
In thesis
1. Nonconformity Measures and Ensemble Strategies: An Analysis of Conformal Predictor Efficiency and Validity
Open this publication in new window or tab >>Nonconformity Measures and Ensemble Strategies: An Analysis of Conformal Predictor Efficiency and Validity
2021 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Conformal predictors are a family of predictive models that associate with each of their predictions a measure of confidence, enabling them to provide quantitative information about their own trustworthiness. In risk-laden machine learning applications, where bad predictions may lead to economic loss, personal injury, or worse, such inherent quality control appears highly beneficial, if not required. While the foundations of conformal prediction were initially published some twenty years ago, their use, and further development, is still (at the time of writing this thesis) not widespread in the machine learning community, and several open questions remain regarding the proper design and use of conformal prediction systems. In this thesis, we attempt to tackle some of these questions, focusing our attention on three specific characteristics of conformal predictors. First, conformal predictors rely on so-called nonconformity functions, which are mappings from the object space onto the real line, typically based on traditional classification or regression models; here, we investigate properties of the underlying learning algorithm and characteristics of the resulting conformal predictor. Second, conformal predictors output predictions on a form that is distinct from traditional prediction methods, by supplying multi-valued prediction regions with a statistically valid coverage probability; we propose two procedures for post-processing the output from conformal classification models that provide interpretations more closely related to traditional predictive models, while still retaining meaningful confidence information. Finally, we provide contributions relating to the construction of conformal predictor ensembles, illustrating potential issues with existing ensemble procedures, as well as proposing and evaluating an alternative ensemble method.

Abstract [sv]

Avhandlingen behandlar områdetconformal prediction, som beskriver en fa-milj prediktiva modeller vars prediktioner associeras med ett konfidensmått,som låter modellerna själva uttrycka sig om sin egen tillförlitlighet. I hög-riskapplikationer, där dåliga prediktioner kan få allvarliga ekonomiska konse-kvenser, eller leda till personskada, tycks en sådan inbyggd säkerhetskontrollhögst värdefull, om inte nödvändig. Medan den teoretiska grunden till confor-mal prediction lades för cirka 20 år sedan, är forskningsområdet fortfaranderelativt ungt, och många öppna frågor kvarstår gällande design och använd-ning av conformal prediction-system. I avhandlingen behandlas några av des-sa öppna frågor, och fokus läggs på tre specifika karakteristika hos conformal-prediktorer. Först behandlas de så kallade icke-konformitetsfunktionerna (non-conformity functions) som ligger till grund för conformal prediction, och sam-bandet utforskas mellan egenskaper hos icke-konformitetsfunktionerna och deresulterande prediktorerna. även egenskaper hos de prediktioner som produ-ceras i en conformal predictor undersöks, och två post-processeringsmetoderpresenteras i ett försök att bistå med en mer intuitivt begriplig tolkning av des-sa prediktioner. Slutligen utforskas strategier för konstruktion av ensemblerav conformal prediction-modeller, där svagheter illustreras i vedertagna stra-tegier, följt av en presentation av en ny ensemblestrategi som ämnar adresseradessa svagheter.

Place, publisher, year, edition, pages
Stockholm: Department of Computer and Systems Sciences, Stockholm University, 2021. p. 62
Series
Report Series / Department of Computer & Systems Sciences, ISSN 1101-8526 ; 21-001
Keywords
Data Science, Machine Learning, Conformal Prediction, Classification, Regression
National Category
Computer Sciences
Research subject
Computer and Systems Sciences
Identifiers
urn:nbn:se:su:diva-192613 (URN)978-91-7911-502-9 (ISBN)978-91-7911-503-6 (ISBN)
Public defence
2021-06-14, online via Zoom, public link is available at the department website, Stockholm, 13:00 (English)
Opponent
Supervisors
Available from: 2021-05-20 Created: 2021-04-25 Last updated: 2022-02-25Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full text

Authority records

Boström, Henrik

Search in DiVA

By author/editor
Boström, Henrik
By organisation
Department of Computer and Systems Sciences
Information Systems

Search outside of DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 152 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf