ELBO scaling

[InfProbSciX]

I noticed that for a gplvm, the numerical value of the elbo (VariationalELBO) doesn't change much at all w.r.t. the number of data points. Is the ELBO scaled internally so that it is the mean across data?

[wjmaddox]

Yes, in general it's meaned across the data. See the type of scaling in

gpytorch/gpytorch/mlls/_approximate_mll.py

Line 57 in fc2053b

log_likelihood = self._log_likelihood_term(approximate_dist_f, target, **kwargs).div(num_batch)

[InfProbSciX]

Thank you - why is this the case? Also, would you mind a PR from me adding this detail to the documentation at https://docs.gpytorch.ai/en/latest/_modules/gpytorch/mlls/variational_elbo.html?

[gpleiss]

A PR would be great!

[wjmaddox]

Sure. Averaging across data points is also done for the MLL and I believe it's to keep everything on the same scale as NN losses in pytorch (which are also traditionally averaged across the batch) with the end goal being easy interpretability of torch.optim learning rates, etc.