variance in the training points not equal to likelihood noise (FixedNoise likelihood)

[ppp0l]

Hello everyone, this might be a bit of a naive question but I'm struggling with this and I cannot find any information.

We can say, I have some observations with known heteroskedatic noise as a training set and thus I am using an ExactGP with FixedNoiseGaussianLikelihood.
In my theoretical understanding of GPR, if I fix the variance of the likelihood (which is the noise), then in training points the variance of the GP should be "almost equal" to the likelihood noise. This is a very relevant property for me.

This seems not to be the case when using an ExactGP with FixedNoiseGaussianLikelihood. Is there something I'm missing about how GPyTorch handles variance prediction? Am I doing something wrong at the implementation level? What do you think?

Here is what I get for one of my training points (standard deviation):
my measurement tolerance: 0.1
likelihood noise ( by model.likelihood.noise.sqrt() ): 0.1
predicted std ( by likelihood(model(point)).variance.sqrt() ) : 0.001

I am using a basic RBF anisotropic kernel + constant mean if that is useful.

[gpleiss]

In my theoretical understanding of GPR, if I fix the variance of the likelihood (which is the noise), then in training points the variance of the GP should be "almost equal" to the likelihood noise. This is a very relevant property for me.

I assume you are talking about the posterior variance? It is approximately true when the likelihood noise is much smaller than the GP covariance, but in general it does not hold.

To see this, note that the posterior predictive variance on the training data is given by $\boldsymbol K - \boldsymbol K^T (\boldsymbol K + \boldsymbol D)^{-1} \boldsymbol K + D$, where $\boldsymbol K$ is the training covariance matrix and $\boldsymbol D$ is a diagonal matrix with observational noise. $\boldsymbol K - \boldsymbol K^T (\boldsymbol K + \boldsymbol D)^{-1} \boldsymbol K$ is only equal to 0 when $\boldsymbol D$ is zero.

[ppp0l]

Thank you for the answer.
I see what you say, but I still do not understand my results. I would expect my posterior predictive variance to be in the order of or greater than the variance of the likelihood, independently by the actual size of $\boldsymbol D$, instead I get a smaller posterior variance, which makes no sense to me.
My argument is the following (I'm not sure this actually works, I'm just justifying my expectation):

• if $\boldsymbol D$ is small enough, ${\sum_0}^\infty (-\boldsymbol K ^{-1} \boldsymbol D) ^ i$ converges, so I am guaranteed $\boldsymbol K - \boldsymbol K^T (\boldsymbol K + \boldsymbol D)^{-1} \boldsymbol K = \boldsymbol K - \boldsymbol K ( I +\boldsymbol K ^{-1} \boldsymbol D)^{-1} = \boldsymbol K - \boldsymbol K {\sum_0 }^\infty (-\boldsymbol K ^{-1} \boldsymbol D) ^ i = \boldsymbol D {\sum_0 }^\infty (-\boldsymbol K ^{-1} \boldsymbol D) ^ i $, which means that for $\boldsymbol D$ close to 0, I have $\boldsymbol D + \boldsymbol K - \boldsymbol K^T (\boldsymbol K + \boldsymbol D)^{-1} \boldsymbol K \approx 2 \boldsymbol D $. (this reasoning was behind the statement in my question, but I was assuming that having a very small $\boldsymbol D$ w.r.t. $\boldsymbol K$ wasn't a strong assumption)
• if $\boldsymbol D$ is not small enough, anyways $\boldsymbol K - \boldsymbol K^T (\boldsymbol K + \boldsymbol D)^{-1} \boldsymbol K$ is a covariance matrix (posterior non predictive covariance), thus is SPD and in particular has positive diagonal, so the diagonal of $\boldsymbol D + \boldsymbol K - \boldsymbol K^T (\boldsymbol K + \boldsymbol D)^{-1} \boldsymbol K$ (predictive posterior variance) cannot be smaller than the one from $\boldsymbol D$ (likelihood variance/noise)

In my results the predictive variance is much smaller than the likelihood variance, instead. I have done multiple trials with different noises, not just the one example above, and the results are similar.

[gpleiss]

Ah sorry I misunderstood the question. You're right, the predicted variance should not be smaller than the likelihood noise. That being said, model.likelihood.noise might not be the right way to access the model noise.

Please post a complete code example reproducing the issue.

[ppp0l]

I wrote an example code to post here, but I cannot reproduce the issue: I must have been doing something wrong in the initial code. Thank you for your time and responses