No meaningful predictive variance

[asanFel]

Hi everybody,
I am fairly new to gpytorch, but I wanted to use it for some experiments. I trained a CNN as a feature extractor and afterwards created a model which takes images as input, puts it through the feature extractor CNN and uses the features as an input to a Gaussian Process layer. I was hoping to get kind of a correlation between the MSE of a sample and the variance of the prediction for the sample. Unfortunately, there isn´t.

Maybe you guys have a hint for me, why I don´t get meaningful variances for my predictions - it seems like they are kind of random as the following picture shows.

I will also post a code snippet - maybe I did some mistakes in the implementation which cause this behaviour, or maybe it is just not possible to get "calibrated" variances.

class DeepKernelModel(nn.Module):
    def __init__(self, gp_layer, feature_extr):
        super(DeepKernelModel, self).__init__()
        self.gp_layer = gp_layer
        self.feature_extr = feature_extr

    def forward(self, x):
        extracted_features = self.feature_extr(x)
        gp_x = self.gp_layer(extracted_features)
        return gp_x

class GPModel_layer(ApproximateGP):
    def __init__(self, inducing_points):
        variational_distribution = CholeskyVariationalDistribution(inducing_points.size(0))
        variational_strategy = VariationalStrategy(self, inducing_points, variational_distribution, learn_inducing_locations=True)

        super(GPModel_layer, self).__init__(variational_strategy)
        self.mean_module = gpytorch.means.ConstantMean()

        self.covar_module = gpytorch.kernels.ScaleKernel(gpytorch.kernels.RBFKernel(
            lengthscale_prior=gpytorch.priors.SmoothedBoxPrior(
                math.exp(-1), math.exp(1), sigma=0.5, transform=torch.exp
            )
        ))

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)

Training Code:

gp_layer = GPModel_layer(inducing_points)
model = DeepKernelModel(gp_layer, feature_extractor)
likelihood = gpytorch.likelihoods.GaussianLikelihood()
mse = torch.nn.MSELoss()
model.gp_layer.train()
likelihood.train()
optimizer_gp = torch.optim.SGD([
        {'params': model.gp_layer.variational_parameters(), 'lr': 0.1},
        {'params': model.gp_layer.hyperparameters(), 'lr': 0.1},
        {'params': likelihood.parameters(), 'lr': 0.01},
    ], lr=0.1, momentum=0.9)
mll = gpytorch.mlls.VariationalELBO(likelihood, model.gp_layer, num_data=train_y.size(0))
for i in range(opt["EPOCHSGP"]):
        model.train()
        likelihood.train()
        running_mae = 0.0
        running_mse = 0.0
        for x_batch, y_batch in train_loader:
            optimizer_gp.zero_grad()
            output = model(x_batch)
            running_mae += torch.mean(torch.abs(output.mean - y_batch.squeeze(1))) * len(y_batch)
            running_mse += mse(output.mean, y_batch.squeeze(1)) * len(y_batch)
            loss = -mll(output, y_batch.squeeze(1))  # + mse(output.mean, y_batch.squeeze(1))
            loss.backward()
            optimizer_gp.step()

I would appreciate every hint/help a lot!
Thanks in advance,
Felix

[gpleiss]

I was hoping to get kind of a correlation between the MSE of a sample and the variance of the prediction for the sample. Unfortunately, there isn´t.

This isn't totally unexpected behavior. See, for example: https://arxiv.org/abs/1805.10522

Intuitively, CNNs aren't great at producing meaningful uncertainties (especially those that correlate with actual model error). While a CNN+GP might improve the variances of a normal CNN, I'd still expect the variances not to correlate with model error.

This is a very active area of research - many research groups are proposing different models to address this.

[asanFel]

@gpleiss Thanks for answering so quickly! Makes sense, I already thought, that it wouldn´t be as easy to just extract features and use them with a GP to get variances which are correlated to the model error.