Using active dimension argument so that different kernels use different part of training and test data

[GHU2021]

Hi,

I implemented independent multioutput Gaussian processes and multitask Gaussian processes. Now, I want to use active dimension argument (i.e., active_dims) so that different kernels use different part of training and test data.

The structure of training and test data are as follows:

1. Input part of both training data (train_x1 and train_x2) is a 2-dimensional NxM matrix.
2. Output part of training data (train_y) is a 2-dimensional NxL matrix.
3. Input part of both test data (test_x1 and test_x2) is a 2-dimensional KxM matrix.
4. Output part of test data (test_y) is a 2-dimensional KxL matrix.

If we concatenate train_x1 and train_x2 in a way that the resulting matrix train_x be a 2-dimensional (Nx2M) matrix, then what value/s should we assign to active_dims argument so that a kernel (e.g., RBF kernel or in general, a combination of some kernels) uses one part of train_x (i.e., train_x1) and another kernel (linear kernel, RBF, etc.) uses another part of train_x (i.e., train_x2 )?

For example, in independent multioutput Gaussian processes, we may want:

class BatchIndependentMultitaskGPModel(gpytorch.models.ExactGP):
    def __init__(self, train_x, train_y, likelihood):
        super().__init__(train_x, train_y, likelihood)
        self.mean_module = gpytorch.means.ConstantMean(batch_shape=torch.Size([L]))
        self.covar_module = gpytorch.kernels.ScaleKernel(
            gpytorch.kernels.RBFKernel(batch_shape=torch.Size([L]),active_dims=....),
            batch_shape=torch.Size([L])) + 
            gpytorch.kernels.ScaleKernel(
            gpytorch.kernels.RBFKernel(batch_shape=torch.Size([L]),active_dims=....),
            batch_shape=torch.Size([L]))
    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultitaskMultivariateNormal.from_batch_mvn(
            gpytorch.distributions.MultivariateNormal(mean_x, covar_x))

or in multitask Gaussian processes, we may need:

class MultitaskGPModel(gpytorch.models.ExactGP):
    def __init__(self, train_x, train_y, likelihood):
        super(MultitaskGPModel, self).__init__(train_x, train_y, likelihood)
        self.mean_module = gpytorch.means.MultitaskMean(
            gpytorch.means.ConstantMean(), num_tasks=L)
        self.covar_module = gpytorch.kernels.MultitaskKernel(
            gpytorch.kernels.RBFKernel(active_dims=....)+gpytorch.kernels.RBFKernel(active_dims=....), num_tasks=L, rank=1)
    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultitaskMultivariateNormal(mean_x, covar_x)

I appreciate your help.

[Balandat]

If I understand correctly, what you'd want is

        self.covar_module = gpytorch.kernels.ScaleKernel(
            gpytorch.kernels.RBFKernel(batch_shape=torch.Size([L]),active_dims=[0, 1, ..., M-1]),
            batch_shape=torch.Size([L])) + 
            gpytorch.kernels.ScaleKernel(
            gpytorch.kernels.RBFKernel(batch_shape=torch.Size([L]),active_dims=[M, M+1, ..., 2M-1]),
            batch_shape=torch.Size([L]))

[GHU2021]

Yes. Exactly. I tested it and it worked. Thank you very much.
Since the values of L and M can change, I revised your suggested commands for multioutput Gaussian processes and multitask Gaussian processes as follows:

1. In case of multioutput Gaussian processes, we can have:
   class BatchIndependentMultitaskGPModel(gpytorch.models.ExactGP):
   def init(self, train_x, train_y, likelihood, L, M):
   super().init(train_x, train_y, likelihood)
   self.mean_module = gpytorch.means.ConstantMean(batch_shape=torch.Size([L]))
   self.covar_module = gpytorch.kernels.ScaleKernel(
   gpytorch.kernels.RBFKernel(batch_shape=torch.Size([L]), active_dims=list(range(0,M)))+gpytorch.kernels.RBFKernel(batch_shape=torch.Size([L]), active_dims=list(range(M,2*M))),
   batch_shape=torch.Size([L]))
   def forward(self, x):
   mean_x = self.mean_module(x)
   covar_x = self.covar_module(x)
   return gpytorch.distributions.MultitaskMultivariateNormal.from_batch_mvn(
   gpytorch.distributions.MultivariateNormal(mean_x, covar_x))
   and
   model = BatchIndependentMultitaskGPModel(train_x, train_y, likelihood, L, M)

2. In case of multitask Gaussian processes, we can have:
   class MultitaskGPModel(gpytorch.models.ExactGP):
   def init(self, train_x, train_y, likelihood, L, M):
   super(MultitaskGPModel, self).init(train_x, train_y, likelihood)
   self.mean_module = gpytorch.means.MultitaskMean(
   gpytorch.means.ConstantMean(), num_tasks=L)
   self.covar_module = gpytorch.kernels.MultitaskKernel(
   gpytorch.kernels.RBFKernel(active_dims=list(range(0,M)))+gpytorch.kernels.RBFKernel(active_dims=list(range(M,2*M))), num_tasks=L, rank=1)
   def forward(self, x):
   mean_x = self.mean_module(x)
   covar_x = self.covar_module(x)
   return gpytorch.distributions.MultitaskMultivariateNormal(mean_x, covar_x)
   and
   model = MultitaskGPModel(train_x, train_y, likelihood, L, M)