Exercises for Lecture 1

Step 1: Get some data

Choose a dataset for regression or classification. Feel free to choose whichever dataset you wish.

For dataset suggestions, try taking a look at the datasets in the UC Irvine ML Repository with 2000+ samples and easily importable in Python.

Split your data into three parts: a training split (where ML models will be trained), a validation/calibration split (where the conformal prediction calibration will happen) and a test split (where performance metrics will be estimated).

Step 2: Modelling and Conformal Prediction

The following is a list of exercises. There is one, ‘basic’ exercise, which we recommend doing completely, as well as additional exercises to further your understanding according to your interests.

Basic exercise:

1. Train a ML model on the training set. Use your favorite model! Possibly one that gives an estimate of uncertainty (e.g., probabilistic classification, quantile regression).
2. Choose one or more $\alpha$ (e.g., $\alpha = 0.05$). Pick a conformity score and use it to do the split conformal calibration on the calibration set, finding the threshold.
3. Implement a way to present the predictive sets $C_\alpha(X)$ to the user. E.g., $a \pm b$ in the case of regression.
4. Calculate the coverage probability in the test set: $\mathbb{P}[Y \in C_\alpha(X)] \approx n^{-1} \sum_{i=1}^n \mathbf{1}[Y_i \in C_\alpha(X_i)]$. How far is it from the specified $\alpha$?
5. Calculate the average set size in the test set: $\mathbb{E}[\mathrm{size}(C_\alpha(X))] \approx n^{-1} \sum_{i=1}^n \mathrm{size}(C_\alpha(X_i))$. Are your predictive sets informative enough?
6. Typically there are different levels of uncertainty for different covariates $X$. How do your intervals change as the $X$ change?

Additional exercises:

1. Implement the conformity score proposed by [Huang et al, 2024] for multi-class classification.
2. Adapt the conformity score for conformalized quantile regression to leverage the relaxed quantile regression method of [Pouplin et al., 2024].
3. The inclusion of $+\infty$ for the quantile computation is a crucial part of split conformal prediction. What happens to the marginal coverage if it is not included? Feel free to assume that the conformity scores are almost surely unique. (Suggestion: try to prove an alternate version of the quantile lemma [i.e., Lemma 1 of [Tibshirani et al., 2020]].)
4. Hard: Suppose $Y = f(X)$ for some $f$ and consider the conformity score $s_{\tilde{f}}(x, y) = \lvert \tilde{f}(x) - y \rvert$. We know that if $\tilde{f} = f$, then our predictive sets will be singletons, thus ensuring tightness. Can you prove something when $\tilde{f} \neq f$? (E.g., under some assumption of closeness of $f$ to $\tilde{f}$.)