ExactMarginalLogLikelihood and MLE estimation

[spectraldani]

Hi everyone, I was recently reading the source code for ExactMarginalLogLikelihood and noticed this part of the code:

gpytorch/gpytorch/mlls/exact_marginal_log_likelihood.py

Lines 41 to 43 in 405f0bd

	# Add log probs of priors on the (functions of) parameters
	for name, module, prior, closure, _ in self.named_priors():
	res.add_(prior.log_prob(closure(module)).sum())

I am most likely misunderstanding this, but shouldn't the marginal log-likelihood not contain the prior contribution into account? It seems like the ExactMarginalLogLikelihood shouldn't be used for maximum likelihood estimation; is this correct?

Another question that I had was that shouldn't this prior term also have a Jacobian contribution since one can place prior on arbitrary transformations of the hyperparameters?

Sorry if the answer is obvious! I'm still exploring how the library works :)

[gpleiss]

But shouldn't the marginal log-likelihood not contain the prior contribution into account?

Sorry, I'm not sure exactly what you mean by this. The marginal log likelihood objects are primarily used for hyperparameter optimization via Type II MLE. You can also add a prior distribution over the hyperparameters, in which case the hyperparameters will be optimized w.r.t the MAP objective. If you do not register any priors on any of the hyperparameters, this line does nothing.

shouldn't this prior term also have a Jacobian contribution since one can place prior on arbitrary transformations of the hyperparameters?

We're assuming that a reasonable prior is put on the hyperparameters. You can also put the prior on a deterministic transform of the priors using the param_or_closure option in register_prior. See https://docs.gpytorch.ai/en/stable/module.html#gpytorch.Module.register_prior

[spectraldani]

If you do not register any priors on any of the hyperparameters, this line does nothing.

I understand. I guess the name of the object misled me; at first, I wasn't expecting a log-likelihood object to also compute log-priors. In my case, I was registering priors on the hyperparameters to use BoTorch's optimization restart but not to obtain MAP estimates. Would it be appropriate to add another class in GPyTorch or a flag to avoid adding the log-prior terms even if priors are registered?

You can also put the prior on a deterministic transform of the priors using the param_or_closure option in register_prior.

Right, I was using the param_or_closure option to set priors on the logarithms of the hyperparameters. But, by doing that, I would expect that the MLL object would compute:

log p(y | theta) + log p(log theta) + log abs(d(log theta)/d(theta))

instead of just

log p(y | theta) + log p(log theta)

Does that make sense?

[gpleiss]

Would it be appropriate to add another class in GPyTorch or a flag to avoid adding the log-prior terms even if priors are registered?

This seems like a reasonable idea. We can add a context manager that disables adding the prior terms to the MLL computation (for examples of our current context managers, see gpytorch.settings). We'd probably want some help implementing this though.

I would expect that the MLL object would compute log p(y | theta) + log p(log theta) + log abs(d(log theta)/d(theta))

I'm not sure why you'd expect the derivative term?