Multi-output GP with IndexKernel for task-specific kernel: what does v stand for

[2mLi]

Hello there,

I'm recently trying to implement the GPyTorch's illustration of multi-output GP (MOGP) by using the IndexKernel class to model task-specific kernel. I notice on the documentation that the IndexKernel is actually composed of a matrix $B$ and vector $v$ such that:

$k(i, j) = {BB^{T} + diag(v)}_{i, j}$

And I notice this expression is the same as what is called "linear model of coregionalisation" in other literatures regarding MOGP, where matrix $B$ is the representation of the mapping from observed output data to the k-dimensional latent GP space, and k would be the column number of matrix B.

My question is, does the vector $v$ have any connection to the concept of linear model coregionalisation MOGP? I notice in many literatures that they introduce a component to the diagonal of the overall covariance matrix to model an overall noise. However, the vector v looks more alike a term for task-specific noise that I did not see in other literatures before. If v is simply a term added to the diagonal of task-specific kernel to circumvent the issue of positive-definiteness, would the value of v be too large as it is usually distributed N(0, 1)?

[Balandat]

The vector v is essentially a learnable noise level that is task-specific. In the most basic LMC model as you suggest the noise is the same for all tasks. This formulation here is more expressive and would allow you to model cases in which the noise levels of the different tasks are different.

If v is simply a term added to the diagonal of task-specific kernel to circumvent the issue of positive-definiteness, would the value of v be too large as it is usually distributed N(0, 1)?

I am not sure why v would be distributed as N(0,1); the idea is more that task j would be be subject to observation noise distributed according to N(0,v_j), where v_j is a learnable parameter.

[2mLi]

Dear Max,

Thank you for your reply! I think this explanation makes perfect sense, thank you a lot for the explanation. I review my outcome again and maybe they do not distribute N(0, 1), apologise for the confusion.

May I ask if you could also kindly help me with my other question in this discussion about (sparse) variational inference of this Hadamard multi-task GP in GPyTorch?
I have a

[partev]

The vector v is essentially a learnable noise level that is task-specific. In the most basic LMC model as you suggest the noise is the same for all tasks. This formulation here is more expressive and would allow you to model cases in which the noise levels of the different tasks are different.

dear @Balandat ,
Is it possible to enforce the same v_ii for all "i"? to make it less susceptible to overfitting? In other words the enforce the same (but learnable) noise level for all tasks.

[Balandat]

I don't think that's straighforwardly possible with the IndexKernel implementation as is, but one could make a simple version of it that uses a single noise variance (assuming you have a good reason to believe the noise level is the same). You'd have to change this to not register num_tasks parameters but just a single one: https://github.com/cornellius-gp/gpytorch/blob/develop/gpytorch/kernels/index_kernel.py#L71

And then you'd have to make sure that the variance is still returned with the right shape - easiest would be to change this to expand the parameter: https://github.com/cornellius-gp/gpytorch/blob/develop/gpytorch/kernels/index_kernel.py#L80-L81