One GP for all features.

[jilsamia]

Hi everyone,

I am replicating this tutorial https://docs.gpytorch.ai/en/latest/examples/06_PyTorch_NN_Integration_DKL/Deep_Kernel_Learning_DenseNet_CIFAR_Tutorial.html where they learn different GP for every features (10 GP since we have 10 features). In my case, I would like to have one common GP for all the features. Has anyone done this before ? Any help will be much appreciated.

Thanks

[gpleiss]

I would like to have one common GP for all the features.

A GP (at least as we define it in this package) is a scalar output function. Therefore, if you learn a single GP for all features, the output of your model will be a scalar. This is appropriate for regression or binary classification, but you typically need a vector-valued output for multi class classification. This is why we have multiple GPs (one for each feature).

[gpleiss]

Yes - that is totally possible. See this LMC example

[leontegral]

Thanks @gpleiss .
How about using an IndexKernel or MultitaskKernel for the covar_module in the GP layer, like in the Hadamard Multitask example?
Or can those kernels only be used for exact inference?

[ackbar03]

Hi @gpleiss,

How would I adapt the code from https://docs.gpytorch.ai/en/latest/examples/06_PyTorch_NN_Integration_DKL/Deep_Kernel_Learning_DenseNet_CIFAR_Tutorial.html to use one common GP for all the features though i.e. the original question?

I am trying to conduct deep regression on image inputs. I tried to remove the IndependentMultitaskVariationalStrategy wrapper but then it leads to problems with the grid size, giving me the following error:

File "/home/wdaiaj/anaconda3/envs/videotorch/lib/python3.9/site-packages/gpytorch/module.py", line 28, in __call__
  outputs = self.forward(*inputs, **kwargs)
File "/home/wdaiaj/.local/lib/python3.9/site-packages/echonet/models/resnet50dkl.py", line 522, in forward
  res = self.gp_layer(features)
File "/home/wdaiaj/anaconda3/envs/videotorch/lib/python3.9/site-packages/gpytorch/models/approximate_gp.py", line 81, in __call__
  return self.variational_strategy(inputs, prior=prior, **kwargs)
File "/home/wdaiaj/anaconda3/envs/videotorch/lib/python3.9/site-packages/gpytorch/variational/_variational_strategy.py", line 124, in __call__
  return super().__call__(
File "/home/wdaiaj/anaconda3/envs/videotorch/lib/python3.9/site-packages/gpytorch/module.py", line 28, in __call__
  outputs = self.forward(*inputs, **kwargs)
File "/home/wdaiaj/anaconda3/envs/videotorch/lib/python3.9/site-packages/gpytorch/variational/grid_interpolation_variational_strategy.py", line 89, in forward
  interp_indices, interp_values = self._compute_grid(x)
File "/home/wdaiaj/anaconda3/envs/videotorch/lib/python3.9/site-packages/gpytorch/variational/grid_interpolation_variational_strategy.py", line 62, in _compute_grid
  interp_indices, interp_values = Interpolation().interpolate(self.grid, inputs)
File "/home/wdaiaj/anaconda3/envs/videotorch/lib/python3.9/site-packages/gpytorch/utils/interpolation.py", line 47, in interpolate
  assert num_dim == len(x_grid)
AssertionError

Currently my model is defined as follows. I don't think changing grid_size to 24 will solve the problem however since it leads to way too much memory being used:

class GaussianProcessLayer(gpytorch.models.ApproximateGP):
    def __init__(self, num_dim, grid_bounds=(-10., 10.), grid_size=64):
        variational_distribution = gpytorch.variational.CholeskyVariationalDistribution(
            num_inducing_points=grid_size, batch_shape=torch.Size([num_dim])
        )

        # Our base variational strategy is a GridInterpolationVariationalStrategy,
        # which places variational inducing points on a Grid
        # We wrap it with a IndependentMultitaskVariationalStrategy so that our output is a vector-valued GP
        # variational_strategy = gpytorch.variational.IndependentMultitaskVariationalStrategy(
        #     gpytorch.variational.GridInterpolationVariationalStrategy(
        #         self, grid_size=grid_size, grid_bounds=[grid_bounds],
        #         variational_distribution=variational_distribution,
        #     ), num_tasks=num_dim,
        # )

        variational_strategy = gpytorch.variational.GridInterpolationVariationalStrategy(
                self, grid_size=grid_size, grid_bounds=[grid_bounds],
                variational_distribution=variational_distribution,
            )
        super().__init__(variational_strategy)

        self.covar_module = gpytorch.kernels.ScaleKernel(
            gpytorch.kernels.RBFKernel(
                lengthscale_prior=gpytorch.priors.SmoothedBoxPrior(
                    math.exp(-1), math.exp(1), sigma=0.1, transform=torch.exp
                )
            )
        )
        self.mean_module = gpytorch.means.ConstantMean()
        self.grid_bounds = grid_bounds

    def forward(self, x):
        mean = self.mean_module(x)
        covar = self.covar_module(x)
        return gpytorch.distributions.MultivariateNormal(mean, covar)



class DKLModel(gpytorch.Module):
    def __init__(self, grid_bounds=(-10., 10.)):
        super(DKLModel, self).__init__()
        self.feature_extractor = resnet50_dkl(pretrained=True)
        self.gp_layer = GaussianProcessLayer(num_dim=2048, grid_bounds=grid_bounds)
        self.grid_bounds = grid_bounds
        self.num_dim = 2048

        # This module will scale the NN features so that they're nice values
        self.scale_to_bounds = gpytorch.utils.grid.ScaleToBounds(self.grid_bounds[0], self.grid_bounds[1])

    def forward(self, x):
        features = self.feature_extractor(x)
        features = self.scale_to_bounds(features)
        # This next line makes it so that we learn a GP for each feature
        # features = features.transpose(-1, -2).unsqueeze(-1)
        res = self.gp_layer(features)
        return res

I'm using the Gaussian Process likelihood

[tkelestemur]

Hey @ackbar03, were you able to solve this? I have a similar case where I'm trying to convert SVDKL example to a regression task. I'm getting the exact same error.

[ackbar03]

Hi @tkelestemur

From what I've sort of gathered, it seems that its not actually possible to train a deep feature extractor end to end with a DKL regressor that uses all the extracted features (e.g. 2048 from resnet 50) on top. This leads to memory overload (someone correct me if i'm wrong).

This is why SVDKL example given in the tutorials uses some sort of average of DKL classifiers, each of which operates on 1 output feature.

I guess it should be possible to change the classifiers in the tutorial to output a regression value through combining / averaging the gaussian outputs by changing the code somehow ? but I didn't manage to figure it out and gave up.

Also, I read the original image DKL regression implementation in the paper I was referencing more carefully, and it seemed they were using a pre-trained frozen feature extractor to extract feature embedding, which sort of solved my confusion and was enough for my purposes.

It would be good to know if you manage to figure it out though, good luck