BetaLikelihood Example #1573

Luckick · 2021-04-14T18:06:13Z

Luckick
Apr 14, 2021

Hi,
I tried to use BetaLikelihood to predict a percentage number. My X is an (N, M) matrix, and y is an (N,) vector with numbers between [0,1].

However, I get the singular U error:
RuntimeError: cholesky_cpu: U(1,1) is zero, singular U.
gpytorch.utils.errors.NanError: cholesky_cpu: 116964 of 116964 elements of the torch.Size([342, 342]) tensor are NaN.

Is there any tutorial for the BetaLikelihood?
Or anybody can help with the following code?
Thanks a lot!

import gpytorch
from gpytorch.models import ApproximateGP
from gpytorch.likelihoods import BetaLikelihood
from gpytorch.variational import VariationalStrategy
from gpytorch.variational import NaturalVariationalDistribution
from gpytorch.priors import SmoothedBoxPrior
from gpytorch.distributions import base_distributions
import torch
from gpytorch.models import ExactGP
from gpytorch.kernels import MaternKernel, ScaleKernel
from gpytorch.means import ConstantMean
from  gpytorch.distributions import MultivariateNormal

class base_model(ApproximateGP):
  def __init__(self, train_x):
    distribution = CholeskyVariationalDistribution(train_x.shape[0])
    variational_strategy = VariationalStrategy(self, train_x, distribution, learn_inducing_locations=True)
    super(base_model, self).__init__(variational_strategy)
    self.mean_module = gpytorch.means.ConstantMean(prior=SmoothedBoxPrior(0.5, 1))
    num_dims = len(train_x) if len(train_x) == 0 else len(train_x[0])
    kernel = MaternKernel(nu=2.5, ard_num_dims=num_dims)
    self.covar_module = ScaleKernel(kernel)

  def forward(self, x):
    mean_x = self.mean_module(x)
    covar_x = self.covar_module(x)
    latent_pred = gpytorch.distributions.MultivariateNormal(mean_x, covar_x)
    return latent_pred

class BetaGP_Model:
  def __init__(self, X, y, training_iters=100, learning_rate=0.1,  gpu=False, nu=2.5,n_restarts=0):
    self.X = X
    self.y = y
    self.training_iters = training_iters
    self.learning_rate = learning_rate
    self.gpu = gpu
    self.n_restarts = n_restarts
    self.likelihood = BetaLikelihood()
    self.model = base_model(self.X)
    if torch.cuda.is_available() and gpu == True:
      self.model = self.model.cuda()

  def fit(self):
    self.model.train()
    self.likelihood.train()
    optimizer = torch.optim.Adam([{'params': self.model.parameters()}, ], lr=self.learning_rate)
    mll = gpytorch.mlls.VariationalELBO(self.likelihood, self.model, num_data=y.size(0))
    for restart in range(0 + 1):
      for i in range(self.training_iters):
        optimizer.zero_grad()
        output = self.model(self.X)
        loss = -mll(output, self.y).sum()
        loss.backward()
        optimizer.step()

model = BetaGP_Model(X, y, training_iters=100, learning_rate=0.1, gpu=False, nu=2.5, n_restarts=0)
model.fit()

Answered by Luckick

Apr 18, 2021

It turns out we get inf loss.
The output is a MultivariateNormal that means are close to 0, and variance are 0.69, y values are between 0-1.

How could this happen? How can I fix it?
Thanks.

output
Out[2]: MultivariateNormal(loc: torch.Size([342]))
output.mean
Out[3]: 
tensor([ 3.3964e-04,  3.6003e-04,  3.9577e-04,  4.5128e-04,  4.8628e-04,
         5.2528e-04, -1.2303e-03, -1.2058e-03, -1.1802e-03, -1.1609e-03,
        -1.1544e-03, -1.1494e-03, -3.9062e-04, -4.0379e-04, -4.0096e-04,
        -3.7526e-04, -3.5229e-04, -3.2224e-04, -6.8367e-04, -6.4440e-04,
        -6.0505e-04, -5.6604e-04, -5.2772e-04, -4.9046e-04, -4.5460e-04,
        -4.2049e-04, -3.8840e-04, -3.5857e-04, -3.0662e-04, -2.…

View full answer

Luckick · 2021-04-18T00:22:58Z

Luckick
Apr 18, 2021
Author

It turns out we get inf loss.
The output is a MultivariateNormal that means are close to 0, and variance are 0.69, y values are between 0-1.

How could this happen? How can I fix it?
Thanks.

output
Out[2]: MultivariateNormal(loc: torch.Size([342]))
output.mean
Out[3]: 
tensor([ 3.3964e-04,  3.6003e-04,  3.9577e-04,  4.5128e-04,  4.8628e-04,
         5.2528e-04, -1.2303e-03, -1.2058e-03, -1.1802e-03, -1.1609e-03,
        -1.1544e-03, -1.1494e-03, -3.9062e-04, -4.0379e-04, -4.0096e-04,
        -3.7526e-04, -3.5229e-04, -3.2224e-04, -6.8367e-04, -6.4440e-04,
        -6.0505e-04, -5.6604e-04, -5.2772e-04, -4.9046e-04, -4.5460e-04,
        -4.2049e-04, -3.8840e-04, -3.5857e-04, -3.0662e-04, -2.8481e-04,
        -2.6593e-04, -2.5005e-04, -2.3716e-04, -2.2725e-04,  1.0417e-04,
         9.3208e-05,  3.1796e-05, -7.4514e-05, -1.4011e-04, -2.1069e-04,
        -6.4871e-06,  2.2320e-05,  3.2229e-05,  3.8753e-05, -1.1583e-03,
        -1.1134e-03, -1.0520e-03, -9.4299e-04,  9.1210e-04,  1.2399e-03,
         1.5442e-03,  1.7897e-03,  1.8811e-03,  1.9479e-03, -1.5637e-04,
        -7.6131e-05, -7.9378e-05, -1.7867e-04, -2.6264e-04, -3.6604e-04,
        -4.0399e-04, -4.4589e-04, -4.0145e-04, -2.5942e-04, -2.4696e-05,
         2.8100e-04,  6.2216e-04,  3.8367e-04,  4.5919e-04,  4.2073e-04,
         2.7095e-04,  3.5066e-05, -2.4518e-04, -5.2156e-04,  4.3136e-05,
         2.0494e-04,  2.7228e-04,  2.2699e-04,  7.3242e-05, -1.6296e-04,
        -4.3996e-04, -2.5653e-04, -2.9731e-04, -3.1768e-04, -3.1316e-04,
        -2.8287e-04, -2.3000e-04, -1.6117e-04,  1.8378e-03,  1.7195e-03,
         1.5085e-03,  1.2243e-03,  8.9573e-04,  5.5636e-04,  1.1910e-03,
         1.1524e-03,  1.0583e-03,  9.1883e-04,  7.4922e-04,  5.6744e-04,
         7.7342e-04,  7.2234e-04,  6.3960e-04,  5.2754e-04,  3.9147e-04,
         2.4001e-04, -1.1911e-03, -1.2223e-03, -1.2294e-03, -1.2086e-03,
        -1.1868e-03, -1.1574e-03,  1.0912e-03,  1.0722e-03,  9.8986e-04,
         8.4413e-04,  7.5012e-04,  6.4461e-04,  1.5635e-04,  2.2897e-04,
         3.1094e-04,  3.9216e-04,  4.2883e-04,  4.6092e-04, -4.9923e-04,
        -3.9555e-04, -2.8252e-04, -1.8024e-04, -1.3871e-04, -1.0575e-04,
         1.0051e-04,  1.4590e-05, -9.0270e-05, -1.4634e-04, -2.0262e-04,
         1.0585e-03,  8.0432e-04,  4.8791e-04,  1.3804e-04, -3.8947e-05,
        -2.1152e-04, -1.0712e-03, -1.1633e-03, -1.2532e-03, -1.3312e-03,
        -1.3851e-03, -1.4033e-03, -1.3777e-03, -1.9658e-04, -3.3373e-04,
        -4.0585e-04, -4.0517e-04, -3.3859e-04, -2.2523e-04, -1.0857e-03,
        -1.3642e-03, -1.6509e-03, -1.9274e-03, -2.1721e-03, -2.3630e-03,
        -2.6433e-04, -2.3046e-04, -2.6072e-04, -3.5380e-04, -4.1954e-04,
        -4.9428e-04, -7.2414e-04, -1.0980e-03, -1.4023e-03, -1.6082e-03,
        -1.6693e-03, -1.7020e-03, -1.4109e-04, -5.0886e-04, -8.1746e-04,
        -1.0350e-03, -1.1041e-03, -1.1458e-03, -1.0578e-03, -1.0482e-03,
        -1.0255e-03, -9.9199e-04, -9.7145e-04, -9.4832e-04, -3.1534e-04,
        -2.9827e-04, -2.6900e-04, -2.3224e-04, -1.9318e-04, -2.3303e-06,
         1.6623e-04,  3.5126e-04,  5.3741e-04,  7.0785e-04,  4.0959e-04,
         2.8525e-04,  3.9136e-06, -7.0306e-05, -2.1957e-04, -2.9350e-04,
        -4.3712e-04, -5.7158e-04, -1.1843e-03, -1.2618e-03, -1.2788e-03,
        -1.2397e-03, -1.1555e-03, -7.1839e-04, -9.4170e-04, -1.0383e-03,
        -1.1213e-03,  3.7563e-04,  4.3042e-04,  4.9597e-04,  5.6184e-04,
         6.1583e-04,  6.4713e-04,  6.4883e-04,  7.5684e-05, -4.8712e-06,
        -7.3614e-05, -1.0170e-04, -1.2498e-04, -1.5647e-04, -1.6867e-04,
        -2.6309e-04, -3.2076e-04, -3.7657e-04, -4.2473e-04, -4.5956e-04,
        -4.7688e-04, -6.0815e-04, -8.5201e-04, -1.0447e-03, -1.1602e-03,
        -1.1842e-03, -1.1850e-03, -3.1850e-04, -4.9687e-04, -6.2440e-04,
        -6.9344e-04, -7.0731e-04, -6.7777e-04, -1.3234e-03, -1.5377e-03,
        -1.6384e-03, -1.6230e-03, -1.5076e-03, -1.3208e-03, -1.4987e-03,
        -1.2270e-03, -9.3704e-04, -6.5019e-04, -3.8197e-04, -1.4174e-04,
         6.5811e-05, -1.4078e-04, -1.1089e-04, -7.0508e-05, -2.5776e-05,
         1.6859e-05,  5.2139e-05,  5.8802e-04,  6.0715e-04,  6.5884e-04,
         7.2338e-04,  7.7483e-04,  7.8889e-04, -3.8227e-04, -4.2082e-04,
        -3.9434e-04, -3.5770e-04, -3.0746e-04,  6.2044e-05, -2.6513e-04,
        -5.8365e-04, -7.2844e-04, -8.5827e-04, -3.6362e-05,  3.8748e-05,
        -7.9671e-05, -1.0767e-04, -1.4316e-04, -1.8189e-04,  2.8963e-05,
        -1.1489e-04, -2.2186e-04, -2.8193e-04, -2.9431e-04, -5.8076e-04,
        -2.9600e-04,  6.9713e-05,  4.7608e-04,  6.7928e-04,  8.7404e-04,
         1.1436e-03,  1.5282e-03,  1.8666e-03,  2.1158e-03,  2.2447e-03,
         1.7437e-04,  1.7254e-04,  1.6055e-04,  1.4329e-04,  1.2620e-04,
         7.7677e-05,  2.3712e-04,  4.0560e-04,  6.1764e-04,  8.5655e-04,
         1.0975e-03,  1.1669e-03,  1.4186e-03,  1.5641e-03,  1.5895e-03,
         1.5005e-03, -1.9466e-04, -1.4713e-04, -4.6302e-05,  9.8218e-05,
         2.6807e-04,  4.3995e-04,  5.9053e-04,  9.9850e-04,  9.5558e-04,
         9.0641e-04,  8.4705e-04,  7.7003e-04,  6.6783e-04,  5.3694e-04,
        -1.0764e-03, -8.6471e-04, -6.0558e-04, -3.2744e-04, -6.0599e-05,
        -5.1306e-04, -2.2468e-04,  7.9476e-05,  3.6150e-04,  5.8694e-04,
         7.3180e-04,  7.8659e-04], grad_fn=<AddBackward0>)
output.variance
Out[4]: 
tensor([0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932,
        0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932, 0.6932],
       grad_fn=<ExpandBackward>)
y
Out[5]: 
tensor([0.2964, 0.3251, 0.3865, 0.4646, 0.4872, 0.5261, 0.1555, 0.2944, 0.3601,
        0.4616, 0.5020, 0.5552, 0.2685, 0.3884, 0.4672, 0.5256, 0.5769, 0.6096,
        0.2773, 0.3062, 0.3266, 0.3443, 0.3669, 0.3744, 0.3864, 0.4193, 0.4366,
        0.4564, 0.5053, 0.5320, 0.5502, 0.5680, 0.5826, 0.5960, 0.3811, 0.4480,
        0.5447, 0.6280, 0.6678, 0.7102, 0.3797, 0.5449, 0.5752, 0.6384, 0.0611,
        0.1679, 0.3522, 0.4787, 0.0870, 0.1640, 0.2680, 0.4210, 0.4750, 0.5500,
        0.0581, 0.1044, 0.1964, 0.3360, 0.3900, 0.4499, 0.1910, 0.3552, 0.4994,
        0.6157, 0.6413, 0.6388, 0.6229, 0.6080, 0.6030, 0.6216, 0.6560, 0.6330,
        0.6220, 0.6120, 0.0990, 0.1998, 0.3222, 0.4254, 0.4956, 0.5327, 0.5365,
        0.6933, 0.7416, 0.7507, 0.7467, 0.7226, 0.6892, 0.6518, 0.0951, 0.1399,
        0.2172, 0.2513, 0.3098, 0.4180, 0.8470, 0.8946, 0.8761, 0.8823, 0.8496,
        0.8237, 0.9554, 0.9333, 0.9260, 0.8966, 0.8761, 0.8536, 0.1426, 0.3035,
        0.4211, 0.5273, 0.5558, 0.5951, 0.1074, 0.2188, 0.3449, 0.4400, 0.4786,
        0.5095, 0.1928, 0.2965, 0.4100, 0.4641, 0.4929, 0.5137, 0.3610, 0.4030,
        0.4560, 0.5020, 0.5470, 0.5720, 0.3855, 0.4209, 0.4563, 0.5022, 0.5280,
        0.6263, 0.7271, 0.7055, 0.7135, 0.7082, 0.7072, 0.1030, 0.1470, 0.2700,
        0.4150, 0.5730, 0.6650, 0.6890, 0.4363, 0.5590, 0.6705, 0.7202, 0.7595,
        0.7776, 0.1610, 0.2205, 0.2494, 0.2908, 0.3459, 0.3860, 0.5058, 0.5414,
        0.5741, 0.5956, 0.6412, 0.6470, 0.4790, 0.5311, 0.5651, 0.5938, 0.5953,
        0.5926, 0.4994, 0.5311, 0.5651, 0.5938, 0.5953, 0.6493, 0.5534, 0.5758,
        0.6154, 0.6606, 0.7493, 0.9460, 0.5500, 0.5737, 0.5779, 0.5922, 0.6131,
        0.5750, 0.6021, 0.6121, 0.6327, 0.6357, 0.5547, 0.5510, 0.6229, 0.7141,
        0.9369, 0.9769, 0.9733, 0.9761, 0.5480, 0.5699, 0.5702, 0.6169, 0.6226,
        0.5376, 0.5697, 0.5747, 0.5798, 0.6123, 0.6381, 0.6665, 0.7023, 0.7443,
        0.7849, 0.8713, 0.6494, 0.6586, 0.7440, 0.9129, 0.9714, 0.9856, 0.9820,
        0.7195, 0.7513, 0.7640, 0.7957, 0.8161, 0.8392, 1.0000, 1.0000, 1.0000,
        1.0000, 1.0000, 1.0000, 0.6540, 0.5820, 0.6160, 0.6550, 0.6910, 0.7020,
        0.5779, 0.6133, 0.6448, 0.6632, 0.6623, 0.4907, 0.1740, 0.2590, 0.3960,
        0.4720, 0.5400, 0.5820, 0.6010, 0.3740, 0.4750, 0.5810, 0.6430, 0.7020,
        0.7290, 0.3757, 0.4618, 0.5457, 0.6286, 0.6727, 0.7300, 0.3130, 0.4470,
        0.5320, 0.5790, 0.6150, 0.3430, 0.4150, 0.4730, 0.5010, 0.5270, 0.6390,
        0.6210, 0.1990, 0.3000, 0.3360, 0.3970, 0.4890, 0.5650, 0.6150, 0.6570,
        0.6970, 0.4240, 0.5190, 0.6260, 0.6710, 0.6920, 0.7190, 0.1849, 0.2618,
        0.3259, 0.4097, 0.4711, 0.3670, 0.4280, 0.4820, 0.6440, 0.9800, 0.3170,
        0.5180, 0.6030, 0.6190, 0.6440, 0.6330, 0.6793, 0.7134, 0.7306, 0.7392,
        0.7424, 0.3140, 0.3706, 0.4671, 0.5470, 0.6078, 0.6059, 0.6320, 0.7960,
        0.7860, 0.7660, 0.7410, 0.7120, 0.6850, 0.6510, 0.2338, 0.3242, 0.3972,
        0.4718, 0.5505, 0.3303, 0.3432, 0.3993, 0.4729, 0.5632, 0.6271, 0.6861])
mll(output, y)
Out[6]: tensor(-inf, grad_fn=<SubBackward0>)

4 replies

gpleiss Apr 26, 2021
Maintainer

Are the y values strictly less than 1 and strictly greater than 0?

Luckick Apr 27, 2021
Author

Are the y values strictly less than 1 and strictly greater than 0?

Good point. There are some values that are exactly 1.
By adding a small perturbation, it seems good. Thanks!

plucaci Jul 25, 2023

What if I my ground truth is categorical data, such as classes?

Instead of the BetaLikelihood function, SoftmaxLikelihood would be the way to go.

But when calling an instantiated object of type SoftmaxLikelihood, it requires a MultitaskMultivariateNormal object as input which, in turn, is the model's output. So, the model has to be of DeepGP type instead of ApproximateGP, and so a DeepGPLayer is needed. Isn't 1x DeepGPLayer still an overkill just for a classification?

gpleiss Jul 26, 2023
Maintainer

@plucaci you don't need a deep gp; you just need a multitask gp model. Follow this example, but replace the MultitaskGaussianLikelihood with a SoftmaxLikelihood: https://docs.gpytorch.ai/en/stable/examples/04_Variational_and_Approximate_GPs/SVGP_Multitask_GP_Regression.html

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BetaLikelihood Example #1573

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 4 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

BetaLikelihood Example #1573

Luckick Apr 14, 2021

Replies: 1 comment · 4 replies

Luckick Apr 18, 2021 Author

gpleiss Apr 26, 2021 Maintainer

Luckick Apr 27, 2021 Author

plucaci Jul 25, 2023

gpleiss Jul 26, 2023 Maintainer

Luckick
Apr 14, 2021

Replies: 1 comment 4 replies

Luckick
Apr 18, 2021
Author

gpleiss Apr 26, 2021
Maintainer

Luckick Apr 27, 2021
Author

gpleiss Jul 26, 2023
Maintainer