mcmc_recover_intervals extensions - bias + coverage

[wds15]

Hi!

I just discovered the intervals function which looks great! I know that I should put all of my parameters one the unit-scale, but in practice I sometimes don't do that (even thought I should, I know). For these circumstances it would be nice to plot things as bias. So instead of showing the true values along with the intervals I would like to see an option which would allow me plot the bias.

Of course, the concept of bias is shaky in a Bayesian world, but as long as I can be sure that my prior is weakly-informative, I would like to be able to do that.

Another very useful extensions (I am happy to open another issue) would be a plot of the coverage when I replicate things a lot of times.

BTW, these tools look awesome to me!

[bob-carpenter]

If you mean the bias of an estimator, that makes sense even with "Bayesian" estimators like the posterior mean: https://en.wikipedia.org/wiki/Bias_of_an_estimator - Bob

…

On Jan 3, 2017, at 4:45 AM, wds15 ***@***.***> wrote: Hi! I just discovered the intervals function which looks great! I know that I should put all of my parameters one the unit-scale, but in practice I sometimes don't do that (even thought I should, I know). For these circumstances it would be nice to plot things as bias. So instead of showing the true values along with the intervals I would like to see an option which would allow me plot the bias. Of course, the concept of bias is shaky in a Bayesian world, but as long as I can be sure that my prior is weakly-informative, I would like to be able to do that. Another very useful extensions (I am happy to open another issue) would be a plot of the coverage when I replicate things a lot of times. BTW, these tools look awesome to me! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[betanalpha]

Sort of — the problem is that it’s not a well-defined concept until you define a particular loss function. Which is why, for example, theta might be unbiased but theta^{2} isn’t.

…

On Jan 3, 2017, at 1:57 PM, Bob Carpenter ***@***.***> wrote: If you mean the bias of an estimator, that makes sense even with "Bayesian" estimators like the posterior mean: https://en.wikipedia.org/wiki/Bias_of_an_estimator - Bob > On Jan 3, 2017, at 4:45 AM, wds15 ***@***.***> wrote: > > Hi! > > I just discovered the intervals function which looks great! I know that I should put all of my parameters one the unit-scale, but in practice I sometimes don't do that (even thought I should, I know). For these circumstances it would be nice to plot things as bias. So instead of showing the true values along with the intervals I would like to see an option which would allow me plot the bias. > > Of course, the concept of bias is shaky in a Bayesian world, but as long as I can be sure that my prior is weakly-informative, I would like to be able to do that. > > Another very useful extensions (I am happy to open another issue) would be a plot of the coverage when I replicate things a lot of times. > > BTW, these tools look awesome to me! > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub, or mute the thread. > — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[wds15]

Hi!

Ok, getting the bias is simple using what is there like this:

library(rstanarm)
alpha <- 1; beta <- c(-.5, .5); sigma <- 2
X <- matrix(rnorm(200), 100, 2)
y <- rnorm(100, mean = c(alpha + X %*% beta), sd = sigma)
fit <- stan_glm(y ~ X)
draws <- as.matrix(fit)
print(colnames(draws))
true <- c(alpha, beta, sigma)
mcmc_recover_intervals(draws, true)

draws_bias <- sweep(draws, 2, true)
mcmc_recover_intervals(draws_bias, rep(0, 4))

that gives me what I want for the bias. Let's see if I figure out the coverage. BTW, the package is awesome!

[jgabry]

@wds15 it would be pretty simple to add either an option or separate function to do what you did manually there. I'll try to get that into the next release, or at least up on a branch for testing soon.

…

On Tue, Jan 24, 2017 at 3:10 AM wds15 ***@***.***> wrote: Hi! Ok, getting the bias is simple using what is there like this: library(rstanarm) alpha <- 1; beta <- c(-.5, .5); sigma <- 2 X <- matrix(rnorm(200), 100, 2) y <- rnorm(100, mean = c(alpha + X %*% beta), sd = sigma) fit <- stan_glm(y ~ X) draws <- as.matrix(fit) print(colnames(draws)) true <- c(alpha, beta, sigma) mcmc_recover_intervals(draws, true) draws_bias <- sweep(draws, 2, true) mcmc_recover_intervals(draws_bias, rep(0, 4)) that gives me what I want for the bias. Let's see if I figure out the coverage. BTW, the package is awesome! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#63 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHb4Q2ovq_5fk0KXG8HYrY9-IltVBii7ks5rVbHqgaJpZM4LZdmj> .

[wds15]

This is so simple that we could just include it as an example in the documentation maybe? What I have done is just a good use of what is there, so I am not sure if you really want to make this part of bayesplot as you need to maintain it. The gain of including it explicitly is that you encourage people to look at the their problems in this viewangle more likely as the option is more prominent. Up to you to decide.

[jgabry]

That's true. Adding an example is probably sufficient.

…

On Wed, Jan 25, 2017 at 2:42 AM wds15 ***@***.***> wrote: This is so simple that we could just include it as an example in the documentation maybe? What I have done is just a good use of what is there, so I am not sure if you really want to make this part of bayesplot as you need to maintain it. The gain of including it explicitly is that you encourage people to look at the their problems in this viewangle more likely as the option is more prominent. Up to you to decide. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#63 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHb4Q3yNPLrZ3oODQMS5cdXp0aqht6btks5rVvzpgaJpZM4LZdmj> .

[wds15]

So we got the bias in now which leaves the coverage. For that one would need multiple posteriors each coming from an independent fit of (usually) fake data. I am not sure of the "batch" concept is exactly that or is it?

[jgabry]

I think the batch argument is different from what you're describing. It was intended to allow grouping parameters together that make sense to look at together or that make it easier to get the axis limits right. So for example you could put coefficients that vary by group into group-specific batches. Or put all parameters you expect to be on certain scales together in the same batch. It just affects how they are displayed in facets. Mon, Feb 6, 2017 at 3:54 PM wds15 <[email protected]> wrote:

…

So we got the bias in now which leaves the coverage. For that one would need multiple posteriors each coming from an independent fit of (usually) fake data. I am not sure of the "batch" concept is exactly that or is it? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#63 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHb4Q80797jLOSM3C-xhXBzY62xV0bFYks5rZ4hpgaJpZM4LZdmj> .

[wds15]

I thought so. Should we then close this issue as it seems out of scope?

However, I do have to say that I have done such experiments, i.e. repeated fake-data simulations and then look at the coverage and correlation in the bias. Both of which were very interesting to learn about in the model as it did tell me a new aspect of my model how it performs under repetition. A correlated bias and the coverage is something one wants to know, I think; would be curious what others think. Possibly worth to discuss in a Stan meeting?