For users who are comfortable with writing their own posterior
samplers, dbarts makes it easy to incorporate a linear BART
component in a Gibbs sampler. In the example below, we fit a functional
mixture model, where observations may have come from one of two
underlying functions and class membership is unobserved.
Performing inference conditional on \(x\), we assume that:
\[Y\mid Z = z \sim \mathrm{N}\left(f(x, z), \sigma^2 \right),\] \[Z \sim \mathrm{Bern}(p),\]
and that \(Z\) is unobserved. To make the model fully Bayesian, we impose a \(\mathrm{Beta}(1, 1)\) prior on \(p\). \(f\) and \(\sigma^2\) have implicit priors defined by BART.
To implement a Gibbs sampler, we will have to produce updates for \(Z\) and \(p\). After writing down the joint distribution of \(Z\), \(Y\), \(p\), \(f\), and \(\sigma^2\), we have:
where \(\phi\) is the density of a normal distribution. This yields an overall strategy of:
To illustrate, we create some toy data. The simulated data is visualized later, together with the fitted model.
# Underlying functions
f0 <- function(x) 90 + exp(0.06 * x)
f1 <- function(x) 72 + 3 * sqrt(x)
set.seed(2793)
# Generate true values.
n <- 120
p.0 <- 0.5
z.0 <- rbinom(n, 1, p.0)
n1 <- sum(z.0); n0 <- n - n1
# In order to make the problem more interesting, x is confounded with both
# y and z.
x <- numeric(n)
x[z.0 == 0] <- rnorm(n0, 20, 10)
x[z.0 == 1] <- rnorm(n1, 40, 10)
y <- numeric(n)
y[z.0 == 0] <- f0(x[z.0 == 0])
y[z.0 == 1] <- f1(x[z.0 == 1])
y <- y + rnorm(n)
data_train <- data.frame(y = y, x = x, z = z.0)
data_test <- data.frame( x = x, z = 1 - z.0)For this specific model, the sampler needs to be updated with new
predictor variables/covariates. This can be done with a
dbartsSampler reference class object by calling the
sampler$setPredictor function. As, given any \(Z = z\), the sampler also needs to evaluate
\(f(x, 1 - z)\), we utilize the test
slot of the sampler and keep it up-to-date with the “counterfactual”
predictor using the sampler$setTestPredictor function.
Models that instead modify the response variable can use the
sampler$setResponse or sampler$setOffset
functions.
One complication in updating the predictor matrix while the sampler
runs is that new values of \(z\) must
leave the sampler in an internally consistent state. During warmup this
constraint is ignored; after warmup is complete, rejection sampling is
used to guarantee that no leaf nodes are empty. This is accomplished by
using the forceUpdate argument to setPredictor
and checking that the logical response is TRUE.
n_warmup <- 100
n_samples <- 500
n_total <- n_warmup + n_samples
# Allocate storage for result.
samples_p <- rep.int(NA_real_, n_samples)
samples_z <- matrix(NA_real_, n_samples, n)
samples_mu0 <- matrix(NA_real_, n_samples, n)
samples_mu1 <- matrix(NA_real_, n_samples, n)
library(dbarts, quietly = TRUE)
# We only need to draw one sample at a time, although for illustrative purposes
# a small degree of thinning is done to the BART component.
control <- dbartsControl(updateState = FALSE, verbose = FALSE,
n.burn = 0L, n.samples = 1L, n.thin = 3L,
n.chains = 1L)
# We create the sampler with a z vector that contains at least one 1 and one 0,
# so that all of the cut points are set correctly.
sampler <- dbarts(y ~ x + z, data_train, data_test, control = control)
# Sample from prior.
p <- rbeta(1, 1, 1)
z <- rbinom(n, 1, p)
# Ignore result of this sampler call
invisible(sampler$setPredictor(x = z, column = 2, forceUpdate = TRUE))
sampler$setTestPredictor(x = 1 - z, column = 2)
sampler$sampleTreesFromPrior()
for (i in seq_len(n_total)) {
# Draw a single sample from the posterior of f and sigma^2.
samples <- sampler$run()
# Recover f(x, 1) and f(x, 0).
mu0 <- ifelse(z == 0, samples$train[,1], samples$test[,1])
mu1 <- ifelse(z == 1, samples$train[,1], samples$test[,1])
p0 <- dnorm(y, mu0, samples$sigma[1]) * (1 - p)
p1 <- dnorm(y, mu1, samples$sigma[1]) * p
p.z <- p1 / (p0 + p1)
z <- rbinom(n, 1, p.z)
if (i <= n_warmup) {
sampler$setPredictor(x = z, column = 2, forceUpdate = TRUE)
} else while (sampler$setPredictor(x = z, column = 2) == FALSE) {
z <- rbinom(n, 1, p.z)
}
sampler$setTestPredictor(x = 1 - z, column = 2)
n1 <- sum(z); n0 <- n - n1
p <- rbeta(1, 1 + n1, 1 + n0)
# Store samples if no longer warming up.
if (i > n_warmup) {
offset <- i - n_warmup
samples_p[offset] <- p
samples_z[offset,] <- z
samples_mu0[offset,] <- mu0
samples_mu1[offset,] <- mu1
}
}Saveing the samplerTo use save and load on the sampler, it
must first be told to write its state out as an R
object:
Finally, we visualize the results. In the graph below, the estimated label of observations encircles the true labels, both of which are represented by point color. We see that, beyond the range of the observed data, the estimated functions regress towards each other and points start to be mislabeled.
mean_mu0 <- apply(samples_mu0, 2, mean)
mean_mu1 <- apply(samples_mu1, 2, mean)
ub_mu0 <- apply(samples_mu0, 2, quantile, 0.975)
lb_mu0 <- apply(samples_mu0, 2, quantile, 0.025)
ub_mu1 <- apply(samples_mu1, 2, quantile, 0.975)
lb_mu1 <- apply(samples_mu1, 2, quantile, 0.025)
curve(f0(x), 0, 80, ylim = c(80, 110), ylab = expression(f(x)),
main = "Mixture Model")
curve(f1(x), add = TRUE)
points(x, y, pch = 20, col = ifelse(z.0 == 0, "black", "gray"))
lines(sort(x), mean_mu0[order(x)], col = "red")
lines(sort(x), mean_mu1[order(x)], col = "red")
# Add point-wise confidence intervals.
lines(sort(x), ub_mu0[order(x)], col = "gray", lty = 2)
lines(sort(x), lb_mu0[order(x)], col = "gray", lty = 2)
lines(sort(x), ub_mu1[order(x)], col = "gray", lty = 3)
lines(sort(x), lb_mu1[order(x)], col = "gray", lty = 3)
# Without constraining z for an observation to 0/1, its interpretation may be
# flipped from that which generated the data.
mean_z <- ifelse(apply(samples_z, 2, mean) <= 0.5, 0L, 1L)
if (mean(mean_z != z.0) > 0.5) mean_z <- 1 - mean_z
points(x, y, pch = 1, col = ifelse(mean_z == 0, "black", "gray"))
legend("topright", c("true func", "est func", "group 0", "group 1",
"est group 0", "est group 1"),
lty = c(1, 1, NA, NA, NA, NA), pch = c(NA, NA, 20, 20, 1, 1),
col = c("black", "red", "black", "gray", "black", "gray"),
cex = 0.8, box.col = "white", bg = "white")The sampler above holds one function. A model may want several, each entering the response through a multiplier the outer loop knows:
\[Y_i \sim \mathrm{N}\left(\sum_{f=1}^{K} m_{f,i}\, g_f(x_i),\ \sigma^2\right),\]
where \(g_1, \ldots, g_K\) are separate sums of trees and \(m_{f,i}\) is a known per-observation scalar - an exposure, a dose, a treatment indicator, a group loading. Bayesian causal forests is the \(K = 2\) case with \(m_1 \equiv 1\) and \(m_2 = z\).
One dbartsSampler per forest composes this in R, and the
composition is ordinary backfitting once the multiplier is folded in.
Given the residual \(r_i = y_i - \sum_{g \neq
f} m_{g,i} g_g(x_i)\), forest \(f\)’s leaf conditionals want response \(r_i / m_{f,i}\) and precision weight \(w_i m_{f,i}^2\); those are the two channels
the loop writes, with setSigma for the third and
run(0L, 1L) to advance one sweep.
updateScale = FALSE (the default) is what makes the swap
coherent: every sweep’s residual is read on the scale creation fixed, so
the leaf prior keeps meaning what it meant.
A multiplier at or near zero says the row carries no information about that forest. Divide anyway and the response is amplified without bound, so the recipe snaps such a row to an exactly zero response paired with its exactly zero weight - which is what the engine does internally, at \(\sqrt{\texttt{.Machine\$double.eps}}\).
set.seed(919)
n.mf <- 200L
x.mf <- matrix(runif(n.mf * 3L), n.mf, 3L)
z.mf <- rbinom(n.mf, 1L, 0.5)
g1.true <- 2 * sin(pi * x.mf[, 1L])
g2.true <- 1 + 2 * x.mf[, 2L]
y.mf <- g1.true + z.mf * g2.true + rnorm(n.mf, 0, 0.5)
# the known per-observation multipliers, one vector per forest
m <- list(rep(1, n.mf), as.double(z.mf))
K <- length(m)
control.mf <- dbartsControl(n.chains = 1L, n.threads = 1L,
updateState = FALSE)
# every sampler is built on the SAME response, with k fixed and sigma owned
# by the outer loop
samplers <- lapply(seq_len(K), function(f)
dbarts(x.mf, y.mf, control = control.mf, leaf.prior = normal(k = 1),
family = gaussian(sigma = fixed(1)), n.trees = 25L))
# one prior budget divided among the forests, rather than K copies of the
# budget for the whole of the sum
base.mf <- samplers[[1L]]$getLeafPrior()$k.scale / samplers[[1L]]$getK()[1L]
for (s in samplers) s$setLeafPrior(normal(sd = base.mf / sqrt(K)))
g <- rep(list(rep(0, n.mf)), K)
sigma.mf <- 1
n.keep <- 200L
keep.g2 <- matrix(0, n.mf, n.keep)
for (i in seq_len(100L + n.keep)) {
for (f in seq_len(K)) {
r <- y.mf - Reduce(`+`, Map(`*`, m[-f], g[-f]))
zero <- abs(m[[f]]) < sqrt(.Machine$double.eps)
samplers[[f]]$setResponse(ifelse(zero, 0, r / ifelse(zero, 1, m[[f]])),
updateScale = FALSE)
samplers[[f]]$setWeights(m[[f]]^2)
samplers[[f]]$setSigma(sigma.mf)
samplers[[f]]$run(0L, 1L)
g[[f]] <- as.numeric(samplers[[f]]$getFitsWithoutOffset())
}
# sigma is drawn in R, against the COMBINED residual
resid.mf <- y.mf - Reduce(`+`, Map(`*`, m, g))
sigma.mf <- sqrt(sum(resid.mf^2) / rchisq(1L, n.mf))
if (i > 100L) keep.g2[, i - 100L] <- g[[2L]]
}
c(recovery = cor(rowMeans(keep.g2), g2.true), sigma = sigma.mf, truth = 0.5)## recovery sigma truth
## 0.9056196 0.4290352 0.5000000
Measured on one machine against a single batched engine sampler at matched total trees - the denominator matters, and a price quoted without one says nothing - K samplers driven a sweep at a time cost about 1.9-2.1 times that sampler at \(K = 2\) and 4.6-5.2 times at \(K = 8\), growing roughly linearly in \(K\): each sampler pays its own per-sweep response install, weight install and result allocation over the full data. Driving one sampler a sweep at a time rather than in a batch is a separate and much smaller charge, about 1.1x.
dbarts(..., forests = list(forest(), forest(basis = factor(z))))
fits this model in one sampler, and the question a prototype should be
able to ask is whether moving to it changes the posterior being
targeted.
The additive decomposition ports exactly. What does not port by itself is prior shape. The two routes agree only if the caller
setOffset: each
single-forest sampler’s reported fit carries that shift, so the
composition attributes it once per forest where the sampler attributes
it once;leaf.prior = normal(k) with \(k\) fixed, every sampler built on a common
\(y\), and
updateScale = FALSE throughout;Without (1) and (2) the two routes are different priors that agree closely. With them the remaining difference is the leaf calibration alone. Measured on a pinned two-forest sampler against the recipe above, at \(n = 400\) over eight chains: with the remedies the posterior mean conditional effect agreed to 3.4% pointwise, inside Monte Carlo error in root-mean-square; without them the difference was 11.9% pointwise and several times Monte Carlo error. Both recovered the truth about equally well, which is the point - the difference is a prior, not a defect.
State equally plainly what the recipe does not reach and never will:
a response family, link or latent structure the sampler does not
provide, and a leaf or hyperprior it does not provide. Those are
selected by argument, not composed in R. forest() carries
the same scale knobs the composition sets by hand, and
$setForestBasis, $setForestWeights and
$getForestAmplitudes are the post-creation routes into a
multi-forest sampler, so a prototype ports without changing the model it
targets.
Everything above fits a continuous response, dbarts’s
default (family = "auto", which chooses
"gaussian" for a continuous \(y\) and "probit" for one coded
0/1). family can also be set to "gaussian",
"probit", or "logistic" explicitly. The two
binary families model \(P(Y = 1 \mid x) =
F(f(x))\) for a link \(F\) and
the sum-of-trees \(f\), and report fits
and predictions on that latent scale rather than as probabilities
(pnorm/plogis recovers the probability from
it). Probit takes \(F\) to be the
standard normal CDF; logistic uses Polya-Gamma augmentation and takes
\(F\) to be the logistic CDF, whose
latent variable has heavier tails than probit’s, so the leaf-value prior
widens to match (its scale becomes pi * sqrt(3) in place of probit’s 3),
without naming the slot.
When only a handful of predictors out of many are relevant, the CGM
tree prior’s uniform choice of split variable wastes proposals on the
rest. split.probs = dart(...) replaces it with a Dirichlet
prior over split-variable probabilities (Linero 2018): the sampler
learns which variables matter and increasingly proposes splits on them,
which doubles as a variable-importance diagnostic.
set.seed(12)
n.dart <- 60
x.dart <- matrix(runif(n.dart * 8), n.dart)
y.dart <- 4 * x.dart[, 1] + rnorm(n.dart, 0, 0.2)
fit.dart <- dbarts(y.dart ~ x.dart, split.probs = dart(),
control = dbartsControl(n.burn = 100L,
n.chains = 1L, n.threads = 1L),
n.trees = 10L)
samples.dart <- fit.dart$run(100L, 30L)
# One row per predictor, averaged over the retained samples: only the first
# column of x.dart drives y.dart, and its split probability dominates.
round(rowMeans(samples.dart$varprobs), 3)## [1] 0.991 0.000 0.005 0.001 0.000 0.000 0.001 0.002
The Dirichlet concentration alpha is itself sampled by
default (update.alpha = TRUE); a smaller alpha
concentrates the split probabilities more aggressively.
?dbartsPriors documents the rest of dart’s
arguments.
dbarts and bart take a
data.frame directly, and
(factors = "categorical", the default for both) an
unordered factor predictor becomes a single categorical column: its
splits send a subset of levels down each branch, rather than the
one-indicator-per-level expansion bartBT still uses.
factors = "indicators" recovers that expansion on either
function, useful for comparing the two. Ordered factors are their own
predictor kind regardless of factors: coded on their
integer levels and split at a threshold, with one candidate threshold
per level boundary rather than a fixed count of cuts, so
n.cuts does not reach them.
Missing predictor values need not be removed beforehand. By default
every split rule learns a direction for NA: an observation
whose split variable is missing follows that rule’s direction (“Missing
Incorporated in Attributes”, Twala et al. 2008), estimated jointly with
the rest of the tree. na.action = na.fail instead refuses
any missing value, the response’s too.
set.seed(13)
n.miss <- 60
g <- factor(sample(c("a", "b", "c"), n.miss, replace = TRUE))
x.miss <- rnorm(n.miss)
x.miss[sample(n.miss, 10)] <- NA
y.miss <- ifelse(g == "a", 2, -1) +
ifelse(is.na(x.miss), 0, x.miss) + rnorm(n.miss, 0, 0.2)
df.miss <- data.frame(y = y.miss, x = x.miss, g = g)
fit.miss <- bart(y ~ x + g, df.miss, n.trees = 10L, n.chains = 1L,
n.threads = 1L, n.samples = 20L, n.burn = 20L,
verbose = FALSE)g above ingests as one categorical predictor with three
levels rather than two indicator columns, and the ten missing values of
x are incorporated rather than dropped.
setOffsetThe response-side conduit carries a whole outer block just as
setPredictor carries the predictor side. A random-intercept
model \[
y_i = f(x_i) + b_{g(i)} + \epsilon_i, \quad
\epsilon_i \sim N(0, \sigma^2), \quad
b_j \sim N(0, \tau^2)
\] is the smallest worked example: \(b\) and \(\tau\) are two Gibbs blocks the outer loop
owns, and BART sees them only as an offset. Each sweep draws \(b\) from its conjugate normal against the
current residual, installs \(f\)’s new
offset with setOffset, runs one sweep, subtracts the offset
back off the reported fit, and then draws \(\tau\).
setOffset(offset, updateScale) takes
updateScale = FALSE after warmup: the response transform
BART calibrated at creation must stay put, or \(b\) and \(\tau\) - which the outer loop holds in
response units - would come to mean something else from one sweep to the
next.
set.seed(24)
n_ri <- 200L
n_groups <- 8L
g <- rep_len(seq_len(n_groups), n_ri)
x_ri <- matrix(runif(n_ri * 3L), n_ri, 3L)
b_true <- rnorm(n_groups, 0, 0.8)
y_ri <- sin(pi * x_ri[, 1L]) + 2 * x_ri[, 2L] + b_true[g] + rnorm(n_ri, 0, 0.5)
n_warmup_ri <- 200L
n_samples_ri <- 500L
control_ri <- dbartsControl(updateState = FALSE, verbose = FALSE,
n.burn = 0L, n.samples = 1L, n.chains = 1L,
n.threads = 1L)
sampler_ri <- dbarts(x_ri, y_ri, control = control_ri, n.trees = 50L)
# rows of each group, and its size; the conjugate draw needs both
rows_of <- split(seq_len(n_ri), g)
size_of <- lengths(rows_of)
# Half-Cauchy(0, 2.5 sd(y)) on tau, sampled by a random-walk Metropolis step
tau_scale <- 2.5 * sd(y_ri)
log_tau_posterior <- function(tau, b) {
if (tau <= 0) return(-Inf)
-n_groups * log(tau) - 0.5 * sum(b^2) / tau^2 +
dcauchy(tau, 0, tau_scale, log = TRUE)
}
b <- rnorm(n_groups, 0, tau_scale / 5)
tau <- tau_scale / 5
sigma <- sd(y_ri)
f_hat <- rep_len(0, n_ri)
samples_b <- matrix(NA_real_, n_samples_ri, n_groups)
samples_tau <- rep_len(NA_real_, n_samples_ri)
for (i in seq_len(n_warmup_ri + n_samples_ri)) {
# b | f, sigma, tau: independent normals, one per group
resid <- y_ri - f_hat
post_var <- 1 / (size_of / sigma^2 + 1 / tau^2)
post_mean <- post_var * (vapply(rows_of, function(rows) sum(resid[rows]), 0) /
sigma^2)
b <- rnorm(n_groups, post_mean, sqrt(post_var))
# f, sigma | b: one BART sweep against the intercepts as an offset. The
# scale re-anchors during warmup only, and is pinned for every kept draw.
sampler_ri$setOffset(b[g], updateScale = i <= n_warmup_ri)
draw <- sampler_ri$run()
f_hat <- as.vector(draw$train) - b[g]
sigma <- draw$sigma[1L]
# tau | b: a random-walk Metropolis step on the log scale
proposal <- tau * exp(rnorm(1, 0, 0.2))
logRatio <- log_tau_posterior(proposal, b) - log_tau_posterior(tau, b) +
log(proposal) - log(tau)
if (log(runif(1)) < logRatio) tau <- proposal
if (i > n_warmup_ri) {
samples_b[i - n_warmup_ri, ] <- b
samples_tau[i - n_warmup_ri] <- tau
}
}
# the group intercepts and their scale are only identified up to a shared
# level, which the forest absorbs, so compare them centered
cor(colMeans(samples_b) - mean(colMeans(samples_b)), b_true - mean(b_true))## [1] 0.99589
The same shape carries any outer block whose contribution to the
linear predictor is a per-observation vector: a parametric term, a
spatial effect, a frailty. setResponse is the companion for
a block that changes \(y\) itself
rather than shifting it.
When implementing a Gibbs sampler using dbarts in R, it
is most often the case that multiple threads will need to be handled by
creating separate copies of the sampler and data. While
dbartsSamplers are natively multithreaded, few of their
slots are stored independently across chains, so an outer loop that owns
per-chain state runs one single-chain sampler per chain and threads them
itself.
This custom-loop bookkeeping is only needed when embedding BART in a
larger sampler. A plain multi-chain bart fit already tracks
its chain dimension: summary() reports split-R-hat and
effective sample size for the scalar parameters the fit sampled
(sigma, and k when it is drawn) and names any
it held fixed on a line under the table, and
dbarts::extract(fit, "sigma", combineChains = FALSE)
returns a sampled parameter’s draws chain-separated, as a
chains-by-samples matrix; a parameter held fixed comes back as its one
number.