Exercises for Lecture 1

Step 1: Get some data

Choose a dataset for regression or classification. Feel free to choose whichever dataset you wish.

For dataset suggestions, try taking a look at the datasets in the UC Irvine ML Repository with 2000+ samples and easily importable in Python.

Split your data into three parts: a training split (where ML models will be trained), a validation/calibration split (where the conformal prediction calibration will happen) and a test split (where performance metrics will be estimated).

Step 2: Modelling and Conformal Prediction

The following is a list of exercises. There is one, ‘basic’ exercise, which we recommend doing completely, as well as additional exercises to further your understanding according to your interests.

Basic exercise:

  1. Train a ML model on the training set. Use your favorite model! Possibly one that gives an estimate of uncertainty (e.g., probabilistic classification, quantile regression).
  2. Choose one or more \(\alpha\) (e.g., \(\alpha = 0.05\)). Pick a conformity score and use it to do the split conformal calibration on the calibration set, finding the threshold.
  3. Implement a way to present the predictive sets \(C_\alpha(X)\) to the user. E.g., \(a \pm b\) in the case of regression.
  4. Calculate the coverage probability in the test set: \(\mathbb{P}[Y \in C_\alpha(X)] \approx n^{-1} \sum_{i=1}^n \mathbf{1}[Y_i \in C_\alpha(X_i)]\). How far is it from the specified \(\alpha\)?
  5. Calculate the average set size in the test set: \(\mathbb{E}[\mathrm{size}(C_\alpha(X))] \approx n^{-1} \sum_{i=1}^n \mathrm{size}(C_\alpha(X_i))\). Are your predictive sets informative enough?
  6. Typically there are different levels of uncertainty for different covariates \(X\). How do your intervals change as the \(X\) change?

Additional exercises:

  1. Implement the conformity score proposed by [Huang et al, 2024] for multi-class classification.
  2. Adapt the conformity score for conformalized quantile regression to leverage the relaxed quantile regression method of [Pouplin et al., 2024].
  3. The inclusion of \(+\infty\) for the quantile computation is a crucial part of split conformal prediction. What happens to the marginal coverage if it is not included? Feel free to assume that the conformity scores are almost surely unique. (Suggestion: try to prove an alternate version of the quantile lemma [i.e., Lemma 1 of [Tibshirani et al., 2020]].)
  4. Hard: Suppose \(Y = f(X)\) for some \(f\) and consider the conformity score \(s_{\tilde{f}}(x, y) = \lvert \tilde{f}(x) - y \rvert\). We know that if \(\tilde{f} = f\), then our predictive sets will be singletons, thus ensuring tightness. Can you prove something when \(\tilde{f} \neq f\)? (E.g., under some assumption of closeness of \(f\) to \(\tilde{f}\).)