You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am exploring working out a workflow for using add_epred_draws() and subsequent calculations when the number of potential predictors is very large. Appreciate the answer might be "just use a smaller prediction matrix" 🤣, but interested to see what is possible.
Following along with this example from Andrew Heiss, I have constructed a brms regression model, that fits well:
Of course, when we try to add_epred_draws() using nd3_matrix in the newdata= argument, we very quickly run out of memory.
So instead, I wondered if it would be possible split nd3_matrix into more manageable chunks, and write to a database to allow more efficient post-processing, like this:
# Create a temporary duckdb database
con <- dbConnect(duckdb::duckdb(dbdir = "preds_m3.duckdb"))
#make a function to write a table to database for each `nest_id`
write_preds <- function(nest_value, ndraws){
temp <- nd3_matrix %>%
group_nest(sex, reading, tb_contact, age_z) %>%
mutate(nest_id = row_number()) %>%
unnest() %>%
ungroup() %>%
filter(nest_id==nest_value) %>%
add_epred_draws(object=m3, re_formula = NA, ndraws = ndraws) %>%
filter(.category == 0) %>%
mutate(.epred = 1 - .epred)
dbWriteTable(con, paste0("processed_data", "_", nest_value), temp, append = TRUE)
}
#Now run for all groups, and save results to database
nest_ids <- nd3_matrix %>%
group_nest(sex, reading, tb_contact, age_z) %>%
mutate(nest_id = row_number()) %>%
unnest() %>%
ungroup() %>%
distinct(nest_id) %>%
pull(nest_id)
walk(nest_ids, .f = \(x) write_preds(nest_value = x, ndraws = 100)) #still only 100 draws... but
# Disconnect from the DuckDB database
dbDisconnect(con)
This seems to work (within a reasonable hour or so), and allows me to work with predictions using dbplyr that might not otherwise be possible. But wondering if there are any ways to future optimise this, and potentially increase the ndraws possible?
Thanks!
The text was updated successfully, but these errors were encountered:
Yeah, we've had to do similar things on large numbers of predictions. It probably would be helpful to have some infrastructure to make this kind of workflow easier, e.g. by providing columns to use to define batches to generate predictions from or something like that.
Re: ndraws specifically, can you make a predictions from one row of this model with all draws fine? I guess I'm not clear on exactly what the problem is...
I am exploring working out a workflow for using
add_epred_draws()
and subsequent calculations when the number of potential predictors is very large. Appreciate the answer might be "just use a smaller prediction matrix" 🤣, but interested to see what is possible.Following along with this example from Andrew Heiss, I have constructed a
brms
regression model, that fits well:We make a very large prediction matrix, comprising 5 million rows, with all combinations of predictors (just for this example).
Of course, when we try to
add_epred_draws()
usingnd3_matrix
in thenewdata=
argument, we very quickly run out of memory.So instead, I wondered if it would be possible split
nd3_matrix
into more manageable chunks, and write to a database to allow more efficient post-processing, like this:This seems to work (within a reasonable hour or so), and allows me to work with predictions using
dbplyr
that might not otherwise be possible. But wondering if there are any ways to future optimise this, and potentially increase thendraws
possible?Thanks!
The text was updated successfully, but these errors were encountered: