Interface with Distributions

[simsurace]

Following issue #231 in AbstractGPs.jl, I would like to propose an interface that allows to automatically use Distributions.jl's large library of distributions. Something like (I haven't tested this yet):

struct GPLikelihood{T <: Distribution, Tl <: AbstractLink} <: AbstractLikelihood
    d::T
    invlink::Tl # takes a value of the latent variable and outputs a parameter vector for `d`
end

(l::GPLikelihood)(f::Real) = l.d(l.invlink(f)...)

function (l::GPLikelihood)(fs::AbstractVector{<:Real})
    return Product((l.d).(l.invlink.(fs)...)) # might need different notation to broadcast correctly
end

and then have convenience structs for special cases:

const BernoulliLikelihood(invlink) =  GPLikelihood(Bernoulli, invlink)
const CategoricalLikelihood(invlink) = GPLikelihood(Categorical, invlink)
....

Would this be useful?

[theogf]

I think that for quick and dirty likelihood creations, it would be more practical to have a CustomLikelihood:

struct CustomLikelihood{F} <: AbstractLikelihood
   f_to_dist::F
end
# For example to create a Binomial distribution with logistic invlink
l = CustomLikelihood(f->Binomial(10, f))

And for other likelihoods, users can always create their own type

struct MyLikelihood{L} <: GPLikelihoods.AbstractLikelihood
   invlink::L
end

and do whatever they want with it.

The interest of having different subtypes is that one can easily multiple-dispatch on them. What you proposed with the aliases would work but be a bit lengthy ::GPLikelihood{<:Bernoulli,<:LogisticLink} instead of ::Bernoulli{<:LogisticLink}.
See for example here: https://github.com/theogf/AugmentedGaussianProcesses.jl/blob/eb5eb7f886c209a80e596ea6e0f678dd2e5f0a7b/src/likelihood/logistic.jl#L60

[simsurace]

Ah, I screwed up the const definitions, they should be on the level of types. To do what you want with dispatch using the GPLikelihood type, one could define special cases as follows:

const BernoulliLikelihood{T} = GPLikelihood{<:Bernoulli, T} where T

etc.

Apart from the question of what is the best way to construct likelihoods, the issue here (I think) is whether it would be possible to have fallbacks for some or most of the methods that are used by the other packages in the ecosystem. Is there already a consensus on what would be the standard API for subtypes of AbstractLikelihoods? Things like loglikelihood could be directly taken from Distributions, and gradients could be calculated with some autodiff package, right? In case that custom methods provide better performance, they can always be defined later.

[theogf]

That's another good point. We never really set on a proper API. This issue could be a good start for a discussion.
The general idea so far is that any AbstractLikelihood applied on f (or fs) will return a Distribution.
We could provide some nice wrapping like

Distributions.loglikelihood(l::AbstractLikelihood, y, f) = loglikelihood(l(f), y)

but it would be nice to generally not be too opinionated about the choices we make.

Another thing planned is to move some expectation computation tools from ApproximateGPs here.

[st--]

To get a better understanding of all the moving pieces, it'd be really good to have some concrete examples (in code) of likelihoods both with additional parameters and multiple latent functions (e.g. Normal with trainable but scalar variance, Normal with both mean and variance parameterised by GPs...)