Reasonable Expectations in Parametric GP vs Exact GP performance

[dangthatsright]

Hi, I'm looking at https://arxiv.org/pdf/1910.07123.pdf and I found the results very promising. Specifically the better nll values would be very helpful in my use case. However, when trying these out on my (2) datasets, I get around NLL=1.0 for Exact GPs and NLL=1.3 for Parametric GPs for both datasets. I would like to know if I am doing something wrong or these fall in cases where Exact GPs are better. Furthermore, one dataset has 5k+ datapoints and I'm using 2k inducing points so I thought that would account for the differences but my other dataset has 1.8k datapoints and I'm still seeing the discrepancies leading me to think that perhaps something is wrong?

Some other notes:

• exact gp implementation is basically like the docs
• parametric gp implemention is ApproximateGP + BatchDecoupledVariationalStrategy + PredictiveLogLikelihood
• I'm doing a hyperparameter optimization for both on kernel and learning rate.
• I'm measuring cross validation NLL. It gets computed as the following

with torch.no_grad(), gpytorch.settings.fast_pred_var():
            preds = self.model(X)
            y_preds = self.likelihood(preds)
            log_likelihood = self.likelihood.log_marginal(y, y_preds).mean()
            return -log_likelihood.data.cpu().numpy()

Any insights would be appreciated! I still have trouble following the details of the Variational GPs so my prior that I am misusing it is high :P

[gpleiss]

I would like to know if I am doing something wrong or these fall in cases where Exact GPs are better.

This is definitely dataset dependent. If you are getting such a high NLL with an exact GP, this suggests that there is not much signal in your data, or you are not using a good kernel for your data.

[dangthatsright]

Thanks for the insight, it is undoubtedly a hard dataset. I was hoping to get some better uncertainty predictions but seems like I need to focus on better featurizations first :P Any ideas on what is an acceptable NLL?

[gpleiss]

Again, that really depends. If your y values are z-scored (mean 0, stdv 1), then a "trivial" GP model (predictive posterior is N(0, 1) everywhere) will get a predictive NLL of 1. The actual NLL values that you can expect depend on the signal-to-noise ratio.

[dangthatsright]

Hey @gpleiss, I was thinking a little bit more about this and I was curious how you got this. I am getting something around 1.4? I feel like it'll be a really big red flag if I didn't understand this. Here is my code:

import torch
m = torch.distributions.normal.Normal(0,1)
samples = m.sample(sample_shape=torch.Size([1000]))
-m.log_prob(samples).numpy().mean()

[gpleiss]

Ah sorry my mental math was off! 1.4 is actually the more reasonable value. I forgot to think about the normalizing constant of the Normal log probability.