Warped GP

[torabin]

Hi friends, wanted to get your advised on warped GP which uses a monotonic transformation to bring GP samples to the actual interval that observations fall in. Wanted to see whether this is sth that GPytroch could support? In my case, observations are real positive values--I tried using log transform of the observations and does training on them--However, interpretation of the predictions in the original space gives me very off results and consequently super high MSE.

[Balandat]

However, interpretation of the predictions in the original space gives me very off results and consequently super high MSE.

How exactly are you doing the un-transformation?

We have some functionality for learnable outcome transforms in BoTorch:
https://github.com/pytorch/botorch/blob/master/botorch/models/transforms/outcome.py

Re warping: If that is useful, we also have learnable input warping (of the features), e.g. https://github.com/pytorch/botorch/blob/master/botorch/models/transforms/input.py#L570
This is helpful for using a stationary kernel on a non-stationary problem.

[torabin]

How exactly are you doing the un-transformation?

I get the mean and var from GPytorch (trained over the log of observed values) and calculate the mean and var in the original space using the mapping between normal and lognormal

I guess one possible thing that contributes to a super high MSE is that I replace observed_y = 0 to observed_y = 1e-5 to be able to take the log of that--and I have a good chunk of items with observed_y = 0--

Re warping: If that is useful, we also have learnable input warping (of the features), e.g. https://github.com/pytorch/botorch/blob/master/botorch/models/transforms/input.py#L570
This is helpful for using a stationary kernel on a non-stationary problem.

Thanks but I think it is not relevant to my current use case--my use case is pretty simple, I am looking for (x, y) relation when y belongs to [0, +inf) and right now I simply train GPytorch using (x, log(y))

[gpleiss]

One idea is an inverse softplus transform of the observations y' = y + log( 1 - e^{-y} ). This function is R+ -> R. When y is >2, this function is essentially linear. When y is <10-^2, this function is essentially logarithmic.

[torabin]

One idea is an inverse softplus transform of the observations y' = y + log( 1 - e^{-y} ). This function is R+ -> R. When y is >2, this function is essentially linear. When y is <10-^2, this function is essentially logarithmic.

what about 'y' = 0? do u have any suggestion for that?

[gpleiss]

I think it's hard to find a deterministic transform that would have support for y=0... In this case I think @Balandat 's solution s may make more sense.

[Balandat]

@dme65 has also implemented a Yeo-Johnson power transform for Ax that seems to work quite well. The MSE is still not great on the original scale, but on the transformed scale this can really help with the modeling. Since "relative goodness" is usually good enough for our BayesOpt approaches, this works well.