Topics on Conformal Prediction (2024)

This is the online page for the minicourse on Topics on Conformal Prediction. It will be updated with references, exercises and additional materials.

Classes

• 2024-10-28: Introduction and split conformal prediction

  • Set predictors their associated probabilistic guarantees; split conformal prediction
  • Marginal guarantee for split conformal prediction: Lemma 1 and Theorem 1 of [Tibshirani et al., 2020]
  • See also: exchangeability and operations that preserve exchangeability: Section 2 of [Kuchibhotla, 2020]

• 2024-10-30: More split conformal prediction: training-conditional guarantee

  • Training-conditional (PAC) guarantee for split conformal prediction: Theorem 1 of [Bian and Barber, 2022]
  • Exercises: see here.

• 2024-11-04: Conformity scores and conditional coverage

  • Examples of conformity scores:
    • Conformalized Bayes: see Section 2.4 of [Angelopoulos and Bates, 2021]
    • Classification:
      • Logits/conditional probabilities (special instance of conformalized Bayes)
      • Cumulative conditional probabilities (Section 2.1 of [Angelopoulos and Bates, 2021])
      • Further: e.g., [Huang et al, 2024]
    • Regression:
      • Gaussian/Laplace predictive posterior
        • Standard absolute error score
        • Absolute error scaled by an uncertainty estimate score
      • Error prediction on hold-out data (in Section 2.3 of [Angelopoulos and Bates, 2021])
      • Conformalized Quantile Regression ([Romano et al., 2019], also in Section 2.2 of [Angelopoulos and Bates, 2021])
      • Further: [Pouplin et al., 2024]
  • Conditional coverage:
    • Goal and impossibility results ([Barber et al., 2019])
    • Group-conditional and label-conditional conformal predition ([Vovk, 2012]; see also [Angelopoulos and Bates, 2021] sections 4.1 and 4.2)
    • See also: [Gibbs et al., 2023]
  • Exercises: TBA

• 2024-11-06: Beyond exchangeability — under distribution shift

• 2024-11-11: Generalizing conformal prediction, “data-efficient” methods

• 2024-11-13: Related ideas: prediction-powered inference and multi-calibration

Suggested Reading

• [Angelopoulos and Bates, 2021]: This ia nice introduction overall to Conformal Prediction, if a bit dated already.
• [Kuchibhotla, 2020]: Gives a good background on exchangeability, which is essential to most Conformal Prediction methods.
• [Bian and Barber, 2022]: Training-conditional coverage for distribution-free predictive inference
• DKW inequality
• PAC and generalization bounds: see books on learning theory, e.g., Mohri et al. (especially chapters 2 and 3) and Bach.

This list will be updated over time.