SVGP Classification reference

[ZhiliangWu]

Hi,

For a multi-class classification task, if I follow the tutorial of Variational GPs w/ Multiple Outputs and replace the MultitaskGaussianLikelihood with SoftmaxLikelihood and setting mixing_weights=False in the SoftmaxLikelihood. Am I essentially using the method proposed in Scalable Variational Gaussian Process Classification from Hensman et al. (cf. Equation 19 in page 5)? Following is a toy example of the implementation.

X, y = make_blobs(n_samples=30, centers=num_groups, n_features=2, random_state=0)
X = torch.tensor(X, dtype=torch.float32)
y = torch.tensor(y, dtype=torch.int64)

class GPClassifier(ApproximateGP):
    def __init__(self, train_x, num_classes):
        variational_distribution = CholeskyVariationalDistribution(
            train_x.size(0),
            batch_shape=torch.Size([num_classes]))

        variational_strategy = VariationalStrategy(
            self, train_x, variational_distribution,
            learn_inducing_locations=True
        )
        variational_strategy = IndependentMultitaskVariationalStrategy(
            variational_strategy, num_tasks=num_classes
        )

        super(GPClassifier, self).__init__(variational_strategy)
        self.mean_module = gpytorch.means.ConstantMean(
            batch_shape=torch.Size([num_classes]))
        self.covar_module = gpytorch.kernels.ScaleKernel(
            gpytorch.kernels.RBFKernel(batch_shape=torch.Size([num_classes])),
            batch_shape=torch.Size([num_classes]
                                   ))

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        latent_pred = gpytorch.distributions.MultivariateNormal(mean_x, covar_x)
        return latent_pred


model = GPClassifier(X, num_groups)
likelihood = gpytorch.likelihoods.SoftmaxLikelihood(
    num_classes=num_groups, mixing_weights=False)
mll = VariationalELBO(likelihood, model, y.numel())

However, from some other issues, the link to the SVDKL paper (cf. Equation 1 at the bottom of page 3) has been mentioned several times, which makes me a bit confused of the correct reference of the method. Some elaborations on these two would really be helpful.

[gpleiss]

The Hensman paper mostly focuses on binary classification, though the multi-class method is a simple extension of Hensman's method. To extend Hensman's method to multi class, we need to have multiple GPs and we need to replace the quadrature in the likelihood with Monte Carlo sampling. However, the Hensman paper would be an appropriate reference.

[ZhiliangWu]

Hi @gpleiss. Thanks for the reply.
I have one follow-up question: If one sets mixing_weights=True, would the reference be the SVDKL paper (cf. Equation 1 at the bottom of page 3)? If that's the case, then can we understand the method in SVDKL as a generalization of Hensman's?

[gpleiss]

SVDKL builds upon the DKL work (Wilson et al, 2016) and the SVGP classification work of Hensman et al, 2015. If you're looking for a citation, I think citing both of those papers should be sufficient.