Skip to content

Fitting a recognition model (VAE) to do approximate inference on intractable posteriors of probabilistic models using an ELBO estimator.

Notifications You must be signed in to change notification settings

xwinxu/variational-mnist

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 

Repository files navigation

Variational Autoencoder and Stochastic Variational Inference

Intro

An implementation of the Variational Autoencoder based on Auto-Encoding Variational Bayes (Kingma and Welling 2013.).

VAE was trained and performed inference on binarized MNIST (handwritten digits) dataset.

Model Definition

For an i.i.d. dataset with continuous latent variables per data point, the Variational Bayes algorithm optimizes a recognition network (encoder) that performs approximate posterior inference using ancestral sampling.

Prior

Latent variable z is sampled from prior distribution on z usuing true parameters theta*.

Likelihood

Likelihood of data x (i.e. all 784 pixels of image) is from a conditional distribution on z using true parameters theta*. Here the distribution is a product of independent Bernoulli's whose means are outputted by the generator network (decoder) parameterized by theta.

Variational Objective

Using an closed form expression for two Gaussians.

KL divergence

Like a regularization term, calculates the expected log ratio of approximate posterior from prior.

Negative expectated reconstruction error

Negative log likelihood of datapoints.

Visualizations

Post training, let's explore the properties of our trained approximate posterior.

Generative model

Sampling latent from prior then using the decoder (generative model) to compute the bernoulli means over the pixels of image given latent z: binary image sample

Latent Posterior

Inferred latent posterior means from the encoder (recognition model): interpolate between latent rep of two points

Linear Interpolation

Generated samples from latent representations interpolated between the posterior means of two different training examples: sampled pairs 1-2, 3-8, 4-5

Joint distribution

Isocountours from joint distribution over latent z (note: 2D in this case, but can be higher for more expressivity) and trained top half of image x: true and variational latent log posteriors

Creating Frankenstein images with optimized approximate posterior

Sample a latent z, feed into our probabilistic decoder, and infer the Bernouilli means of all bottom half of the image's pixels: Predicting the bottom half of an image from the top

About

Fitting a recognition model (VAE) to do approximate inference on intractable posteriors of probabilistic models using an ELBO estimator.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages