Docs

[pat-alt]

Some early work on regression docs (-> blog post). In regions of the domain that were not seen during training, predictive uncertainty should be really high (no?). See, for example, my results for a Bayesian Neural Network lifted from LaplaceRedux.jl:

When I run a similar experiment using CP (here split conformal for XGBoost), results look very different:

Considering how CP works, I don’t find this result too surprising, it’s just not great and I’m now a bit confused as to how exactly the coverage guarantees work.

The chart is produced in this notebook, where I’ve been trying to replicate this tutorial from MAPIE.

It would be useful to cross-check the results against the MAPIE results (notebook stored here). In there they don't really look at values of x outside of the training range at the prediction stage, but it should be straight-forward to make adjustments to the code in the notebook. (EDIT: done below)

[pat-alt]

I've now run the MAPIE tutorial with gradient boosting instead of polynomial reg and the results are the same (so this is not an implementation issue):

Quite puzzled now: looking at the chart, test coverage left and right of the training points is very low. I guess we don't have exchangeability in this case or I'm just misunderstanding the marginal coverage property. @aangelopoulos @valeman perhaps you can enlighten me? Sorry if trivial, I have serious brain fog right now.

[codecov-commenter]

Codecov Report

Merging #33 (0ad8296) into main (479b94f) will decrease coverage by 12.86%.
The diff coverage is 46.23%.

@@             Coverage Diff             @@
##             main      #33       +/-   ##
===========================================
- Coverage   99.32%   86.45%   -12.87%     
===========================================
  Files           6        9        +3     
  Lines         297      384       +87     
===========================================
+ Hits          295      332       +37     
- Misses          2       52       +50     

Impacted Files	Coverage Δ
src/ConformalModels/inductive_bayes.jl	0.00% <0.00%> (ø)
src/ConformalModels/plotting.jl	50.00% <50.00%> (ø)
src/ConformalModels/inductive_classification.jl	98.41% <100.00%> (ø)
src/ConformalModels/inductive_regression.jl	100.00% <100.00%> (ø)
src/ConformalModels/transductive_classification.jl	100.00% <100.00%> (ø)
src/ConformalModels/transductive_regression.jl	100.00% <100.00%> (ø)
src/ConformalModels/utils.jl	100.00% <100.00%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[pat-alt]

When using polynomial regression as opposed to gradient boosting, the results look better (in some cases). But this is driven by the fact that separate fits for subsets of the data (CV, Jackknife, ...) reflect the underlying stochasticity of the data.

Again, I suppose exchangeability is violated: I draw training data from $\mathcal{X}\sim\text{Unif}(-3,3)$ and then map to $\mathcal{Y}$. Test data left and right of the training data is not drawn from $\mathcal{X}$, so we have covariate shift (will be interesting to look at Section 4.5 in this context). Is that right @aangelopoulos? Thanks in advance and apologies if trivial (still fairly new territory for me and trying to wrap my head around things).

Edit: Also Section 3 of the MAPIE tutorial seems to be looking at pretty much the same thing.

[aangelopoulos]

Hey @pat-alt ! Happy to help!

I think you have the right idea --- the example that you've given comes from an extreme covariate shift.
If you run covariate shift conformal using Section 4.5 of the gentle intro, what will happen is that your intervals will explode outside the training data, since the likelihood ratio gets very large.

Without using the covariate shift technique, conformal will only give you the marginal guarantee if the new data is i.i.d. with your old data... so the regions with no training data don't really matter, since they occur with very low probability.

My suggestion would be this. If you like the way the Bayesian NN performs in this setting, use it as your heuristic notion of uncertainty. Then calibrate it with conformal. Best of both worlds!

[pat-alt]

Hey @pat-alt ! Happy to help!

I think you have the right idea --- the example that you've given comes from an extreme covariate shift. If you run covariate shift conformal using Section 4.5 of the gentle intro, what will happen is that your intervals will explode outside the training data, since the likelihood ratio gets very large.

Without using the covariate shift technique, conformal will only give you the marginal guarantee if the new data is i.i.d. with your old data... so the regions with no training data don't really matter, since they occur with very low probability.

My suggestion would be this. If you like the way the Bayesian NN performs in this setting, use it as your heuristic notion of uncertainty. Then calibrate it with conformal. Best of both worlds!

Got it - thank you very much!

I'm gradually working my way through your tutorial (which is fantastic by the way!). Both Conformalized Bayes #31 and covariate shift conformal #39 are on the to-do list 😄

Thanks again for your help - much appreciated!

[aangelopoulos]

Thanks Pat!! This package that you are building is great!!