Stephen Thomas

Robust Statistics

2025-04-14T00:00:00Z

Even if you’ve forgotten everything else about statistics, mean and standard deviation are probably familiar. Both measures describe important characteristics of a numerical data set. The mean gives us a sense of the “average” or “typical” value of the data, and the standard deviation tells us how widely values vary from that average. Sometimes, however, those two measures don’t work as well as we would like. If a data set has been contaminated with even a few outliers, for example, both the mean and the standard deviation can be misleading. And in those cases, robust statistics provide alternatives that may better describe the data. In this article, we’ll look at a concrete example of the problems with traditional measures, we’ll see how robust statistics solve those problems, and we’ll consider ways to relate robust measures with traditional ones.

Mean and Standard Deviation are Sensitive to Outliers

To see how mean and standard deviation can fall short, consider this simple example: Imagine a population of people whose net worth follows a normal distribution with a mean of $200,000 and a standard deviation of $25,000 Although the distribution of net worth in the United States is far from a normal distribution, its median value in 2023 was about $200,000 according to the Federal Reserve's Survey of Consumer Finance. So this imaginary population at least bears a superficial resemblence to the US population.. Pick fifty random inviduals from this population, and describe that sample using statistics. We can simulate that experiment in software.

import numpy as np
np.random.seed(42)

population_mean = 200000
population_stdev = 25000

group = np.random.normal(loc=population_mean, scale=25000, size=50)

mean = np.mean(group)
stdev = np.std(group)

The mean net worth of this sample is about $194,000, and its standard deviation is about $23,000. Both of these values are consistent with the whole population. If those measures were the only information you had about a group of fifty people, you would have no reason to doubt that the group’s members were typical folks from the population.

Now imagine that a single billionaire joins the group. How does that addition affect these measures?

newgroup = np.append(group, 1000000000)
mean = np.mean(newgroup)
stdev = np.std(newgroup)

The group’s mean net worth has now grown to nearly $20 million! This new group looks like a gathering of multi-millionaires, not at all typical of the general population. The group’s standard deviation, now over $138 million, also differs substantially from the population’s $23 thousand. Based solely on the mean and standard deviation, we would never suspect that there is nothing special about 50 of the group’s 51 members. The traditional measures could mislead us to the wrong conclusion.

Robust Measures Limit the Effects of Outliers

If we suspect that a data set contains outliers that don’t represent the population, we may want to limit the effects of those extreme values in our analysis. Robust statistics are one way to do that. The robust alternatives to the mean and standard deviation are the median and the median absolute deviation (MAD). Here is what those measures say about our example. First, consider the original 50 folks without the billionaire.

from scipy import stats

median = np.median(group)
mad = stats.median_abs_deviation(group, scale="normal")

The median net worth of our original group is $194,146 and its MAD More precisely, the code example calculates the normalized MAD, which corresponds to the standard deviation for normally-distributed populations. We’ll look at that in more detail below. is $22,009. Those measures reasonably reflect the characteristics of the entire population with its mean of $200,000 and standard deviation of $25,000. But what happens when the billionaire arrives?

median = np.median(newgroup)
mad = stats.median_abs_deviation(newgroup)

The median net worth only increases to $194,147, and the new MAD is $22,604. One outlier doesn’t change the measures much at all, and, by those values, the new group still reflects the general population.

Calculating these robust measures is straightforward. For the median, arrange the data in order and pick the value exactly in the middle. Once the median is known, determine the MAD by first calculating the absolute values of the differences between each data value and the median, and then find the median of the result. That’s probably easier to understand with an example:

Data set: [3, 1, 10, 5, 7]
Data set in order: [1, 3, 5, 7, 10]
Median is central (third) value: 5
Deviations from median in original data set: [-2, -4, 5, 0, 2]
Absolute deviations: [2, 4, 5, 0, 2]
Absolute deviations in order: [0, 2, 2, 4, 5]
Median absolute deviation is central (third) value: 2

Sometimes Robust Measures can Match Traditional Measures

In the example above, we were able to compare the sample’s median and MAD (194,000 and 22,600) to the population’s mean and standard deviation (200,000 and 25,000). The result suggested that the sample was indeed from the population. To make this comparison, however, we had to rely on an extra calculation that note 2 hints at: we had to normalize the MAD. It’s not shown in the procedure shown above, so let’s add it now.

Optionally, and if possible, scale the median and/or the MAD

To scale a robust measure, we multiply its raw value by a scaling factor. This additional multiplication makes the robust measure directly comparable with its non-robust equivalent. In our example, the raw MAD for the original group is 14,845. That’s not very close to the population’s standard deviation of 25,000 or the sample’s standard deviation of 23,107. It’s when we multiply that raw value by 1.4826 that we get the normalized MAD of 22,009 which we can compare to both.

There are two important qualifiers in step 8. The first is “optionally.” It’s not always necessary that robust measures correspond to traditional ones. Unscaled robust measures are a perfectly legitimate way to describe data. Nonetheless, mean and standard deviation are more common, so scaling robust measures to match traditional ones can make an analysis easier to understand and appreciate, especially by folks unfamiliar with robust statistics. The second qualifier adds a warning though: that’s not always possible. You can scale robust measures for some data, but nor for all data. It depends on the probability distribution of the underlying population. Our example follows a normal distribution, and the normal distribution supports scaling. That’s all we’ll consider in this post, but a follow up will look at other distributions where things get tricky.

The mathematics behind the scaling of robust statistics are not super complicated. Even so, I find the process easier to understand visually, so that’s the approach I’ll use here. In a few graphs we can see how scaling works and, in particular, how to derive the magic value of 1.4826.

In many cases, including our example, the mean/median relationship is trivial: they’re the same. This is true whenever the distribution is symmetric about the mean, including for the normal distribution we’ve been using in our example. You can see that graphically in figure 1, which plots the probability density for our example population. Half of the density lies below the mean and half lies above it, so the mean must be exactly in the middle. Of course, the “middle” is, by definition, the median. So in this case we can consider the median to be the “robust mean.”

The probability density function (PDF) of the normal distribution shows that the mean lies exactly in the middle; that is exactly the definition of the median, and for this distribution, the two measures have the same expected value.

The situation is more complicated for the MAD and the standard deviation. They’re clearly not equal, even in our example, as the population standard deviation was $25,000 while the sample MAD of the original group was only $14,845. You can’t directly compare those numbers. It turns out, however, that all we need is a constant scaling factor to make the two values comparable. Let’s find that scaling factor. Remember that the MAD is defined as the median of absolute deviations. We’ve already seen that the median equals the mean, $μ$ .

$MAD (X) = median (| X_{i} - median (X) |) = median (| X_{i} - μ) |)$

The symmetry of the normal distribution helps here as well. Notice that the absolute deviations can be split into two groups: (1) the deviation for values greater than the mean, $X_{i} - μ$ , and (2) the deviation for values less than the mean, $μ - X_{i}$ . But, as figure 2 shows, those two groups have the same absolute deviations; for every point that’s greater than the mean, there is an equivalent point less than the mean with the same absolute deviation. And since each group has the same absolute deviations, each group will also have the same median value for those absolute deviations. Deriving the median for one of the groups is enough to tell us the MAD of the entire distribution.

For the normal distribution, every point below the mean/median has a corresponding point above the mean/median with the same deviation.

Just like the median is the point exactly in the middle of the probability density, the median deviation from that median is the point exactly in the middle of the upper half of the probability density. It is point at which half of the probability lies to its left and half lies to its right. Figure 3 highlights these divisions. As the figure shows, we first divide the probability density in half to find the median, and then we divide one of those halves in half again to find the median deviation.

The area of the PDF above the mean/median represents half of the total probability. Dividing that region in half again locates the median deviation from the mean/median.

One we see where the median deviation is located, it’s easy to calculate. The value, in this case net worth, is the 75th percentile value for our distribution. And the median absolute deviation is the difference between that value and the median value or the population mean. In our example the expected value for the MAD is $16,862,

percentile75 = population.ppf(0.75)
mad = percentile75 - population_mean

The same approach works for any normal distribution. The expected value of the median is the population mean, and the expected value of the MAD is the difference between the 75th percentile and the mean.

$\begin{aligned} median [X \sim N (μ, σ)] & = μ \\ MAD [X \sim N (μ, σ)] & = P_{0.75} (X) - μ \end{aligned}$

To make the relationship between MAD and $σ$ more concrete, use the formula for percentile values of the normal distribution. It depends on both $μ$ and $σ$ as well as values of the probit function $Φ^{- 1}$ . Equation 3 shows the results: the expected value of MAD is linearly proportional to the standard deviation, and the scaling constant is $Φ^{- 1} (0.75)$ .

$\begin{aligned} MAD [X \sim N (μ, σ)] & = P_{0.75} (X) - μ \\ = [μ + σ Φ^{- 1} (0.75)] - μ \\ = σ Φ^{- 1} (0.75) \\ or \\ σ & = \frac{MAD}{Φ^{- 1} (0.75)} \end{aligned}$

No simple expression exists for the probit funtion $Φ^{- 1} (p)$ in general, but its value at 0.75 can be computed analytically and is approximately 0.67449. If we measure the MAD of a sample from a normally distributed population, we can expect the population’s standard deviation to equal that measurement divided by 0.67449. This value is sometimes called the normalized median absolute deviation, and it serves as a robust version of the standard deviation for normal distributions. Some programming libraries (e.g. R stats) even employ this scaling by default when computing the median absolute deviation; they’re assuming, by default, that the population is normal.

Now we have a way to provide robust descriptive statistics that are directly comparable to to traditional measures, at least for normally distributed populations:

$\begin{aligned} μ & \sim median \\ σ & \sim 1.4826 \cdot MAD \end{aligned}$

Robust Measures can be Effective with Many Outliers

Now that we have a way to calculate robust versions of the mean and standard deviation, we can see how well those measures resist outliers. Recall that our original group was sampled from a normal distribution with $μ = 200, 000$ and $σ = 25, 000$ , and we added one outlier to that group. The mean and standard deviation of the new group were nearly 20,000,000 and 138,000,000, respectively. Those measures aren’t even close to the distribution parameters. The robust measures, however, reflect the population quite nicely: 194,000 and 22,600 for the “robust mean” and “robust standard deviation”. At least for our initial example, robust statistics provide a better description of the underlying population when the sample includes outliers.

To verify that this result isn’t a fluke, we can generate a series of samples from normal distributions with a range of parameters, introduce outliers into those samples, and compare traditional and robust measures.

np.random.seed(42)
sample_stats = []
for mu in [1, 10, 100]:
    for sigma in [x * mu for x in [0.1, 0.2, 0.5, 1, 2, 5, 10]]:
        sample = np.random.normal(loc=mu, scale=sigma, size=500000)
        sample[0:500] = [mu + 5 * sigma] * 500 # replace some samples with outliers
        sample_stats.append({
            'mu': mu,
            'mean': np.mean(sample),
            'median': np.median(sample),
            'sigma': sigma,
            'std': np.std(sample),
            'mad': stats.median_abs_deviation(sample, scale="normal")
        })

Table 1 lists the results. As it makes clear, the robust measures consistently provide substantially more accurate estimates of the true population parameters.

Comparison of traditional and robust measures for multiple samples from various normal distributions. Each row summarizes 499,500 values randomly sampled from a distribution, combined with 500 values set to μ + 5σ. Note that MAD is the *normalized* median absolute deviation.
$μ$	mean	median	$σ$	std	MAD
1	1.00038	1.00003	0.1	0.101241	0.100256
1	1.0006	1.00006	0.2	0.202366	0.200352
1	1.00184	0.999959	0.5	0.507097	0.501442
1	1.00575	1.00302	1	1.01217	0.999266
1	1.01241	1.00631	2	2.02585	2.00451
1	1.02055	1.00661	5	5.0526	5.00356
1	1.05392	1.00601	10	10.1104	9.99765
10	10.0058	9.9996	1	1.01116	1.00149
10	10.0107	10.0073	2	2.02504	2.00467
10	10.0243	10.0064	5	5.05805	5.01062
10	10.0504	9.99848	10	10.1241	10.0024
10	10.055	10.0042	20	20.2442	20.0376
10	10.3353	10.1176	50	50.5643	50.0471
10	10.6621	10.1211	100	101.178	100.086
100	100.048	100.012	10	10.1221	10.0404
100	100.117	100.019	20	20.2282	20.0061
100	100.172	99.9882	50	50.6538	50.147
100	100.61	100.39	100	101.193	100.27
100	100.709	100.164	200	202.234	200.081
100	102.727	101.292	500	505.394	499.636
100	102.913	99.9615	1000	1012.84	1001.87

We can also visualize the performance differences between estimators. Figure 4 looks at 500,000 random samples from a standard normal population ( $μ$ = 0, $σ$ = 1). The data set in each panel is contaminated with an increasing number of outliers. The plots compare three probability density functions:

The true PDF of the population.
The PDF based on traditional estimates of the population parameters.
The PDF based on robust estimates of the population.

One again, the robust measures more accurately reflect the true population.

The true PDF of a normal distribution compared with estimated PDFs based on traditional and robust measures. Plots derived from sampled data sets consisting of 500,000 random samples corrupted with an increasing number of outliers.

These results are pretty impressive, but they do come with an important caveat: we’re considering a population represented by the normal distribution. The next post will examine the complications that different distributions introduce.

Colophon

Figures 1-3 were created by the following code:

from matplotlib import font_manager, pyplot as plt

# Define the range for the x-axis
x = np.linspace(
    population_mean - 4 * population_stdev, population_mean + 4 * population_stdev, 1000
)

# Compute the PDF of the normal distribution
population = stats.norm(population_mean, population_stdev)
pdf = population.pdf(x)

# Plot styles
%config InlineBackend.figure_format = 'retina'
font_dir = ["../Fonts"]
for font in font_manager.findSystemFonts(font_dir):
    font_manager.fontManager.addfont(font)

darkmode = True
plottype = "topmedian" # median, deviation, topmedian

plt.style.use('../Styles/darkmode.mplstyle' if darkmode else '../Styles/lightmode.mplstyle')

color1 = plt.rcParams["axes.prop_cycle"].by_key()["color"][0]
color2 = plt.rcParams["axes.prop_cycle"].by_key()["color"][1]
color3 = plt.rcParams["axes.prop_cycle"].by_key()["color"][2]
color4 = plt.rcParams["axes.prop_cycle"].by_key()["color"][3]
color5 = plt.rcParams["axes.prop_cycle"].by_key()["color"][4]
color6 = plt.rcParams["axes.prop_cycle"].by_key()["color"][5]
textcolor = plt.rcParams["text.color"]

#--- Figure created in layers
fig, ax = plt.subplots(figsize=(7.88, 5), tight_layout=True)

#--- shading of region divided by mean
if plottype in ("median", "deviation"):
    if darkmode:
        ax.add_patch(plt.Rectangle(
            (0, 0), 0.5, 1, facecolor=textcolor, alpha=0.20, transform=ax.transAxes
        ))
    else:
        ax.add_patch(plt.Rectangle(
            (0.5, 0), 0.5, 1, facecolor=textcolor, alpha=0.05, transform=ax.transAxes
        ))

#--- PDF itself
ax.plot(x, pdf)
ax.set_xlabel("Net Worth ($1000s)")
ax.xaxis.set_major_formatter(lambda x, pos: f'{x/1000:.0f}')
ax.set_ylabel("Probability Density")

# disable y ticks and tick labels
ax.tick_params(
    axis="y",
    which="both",
    left=False,
    right=False,
    labelleft=False,
)

# remove the bottom margin but retain the top margin
ax.set_ymargin(0)
ax.autoscale_view()
lim = ax.get_ylim()
delta = np.diff(lim)
top = lim[1] + delta * 0.05
bottom = lim[0]
ax.set_ylim(bottom, top)

#--- fill the area beneath the curve
if plottype in ("median", "deviation"):
    plt.fill_between(x[: len(x) // 2], pdf[: len(pdf) // 2], color=color1, alpha=0.33)
    plt.fill_between(x[len(x) // 2 :], pdf[len(pdf) // 2 :], color=color1)
elif plottype == "topmedian":
    x0 = population_mean
    x1 = population.ppf(0.75)
    i50plus = [i for i in range(len(x)) if x0 <= x[i] < x1]
    i75plus = [i for i in range(len(x)) if x[i] >= x1]
    plt.fill_between(x[: len(x) // 2], pdf[: len(pdf) // 2], color=color1, alpha=0.33)
    plt.fill_between(x[i50plus], pdf[i50plus], color=color1, alpha=0.67)
    plt.fill_between(x[i75plus], pdf[i75plus], color=color1)

#--- label the regions

if plottype == "median":
    ax.text(
        0.525,
        0.98,
        "Half of density above mean",
        transform=ax.transAxes,
        verticalalignment="top",
        horizontalalignment="left",
    )
    ax.text(
        0.475,
        0.98,
        "Half of density below mean",
        transform=ax.transAxes,
        verticalalignment="top",
        horizontalalignment="right",
    )

#--- highlight the mean value
if plottype in ("median", "deviation"):
    ax.axvline(
        population_mean,
        linewidth=4,
        color=color2,
    )
if plottype == "median":
    ax.annotate(
        "mean",
        xy=(0.5, 0.4),
        xytext=(0.45, 0.4),
        arrowprops=dict(facecolor=textcolor, arrowstyle='-|>'),
        horizontalalignment="right",
        verticalalignment="center",
        xycoords=ax.transAxes
    )

#--- add example deviations
if plottype == "deviation":
    x0 = population_mean
    y0 = population.pdf(x0)
    x1 = population_mean - 1.75 * population_stdev
    y1 = population.pdf(x1)
    x2 = population_mean + 1.75 * population_stdev
    y2 = population.pdf(x2)
    plt.scatter(x1, y1, color=color4, zorder=2.5)
    plt.scatter(x2, y2, color=color4, zorder=2.5)
    ax.annotate(
        "deviation",
        xy=(x0,y1),
        xytext=(x1 + (x0-x1)/2, y1),
        horizontalalignment="center",
        verticalalignment="bottom"
    )
    ax.annotate(
        "",
        xy=(x0,y1),
        xytext=(x1,y1),
        arrowprops=dict(facecolor=textcolor, arrowstyle="<->")
    )
    ax.annotate(
        "deviation",
        xy=(x0,y2),
        xytext=(x2 - (x2-x0)/2, y2),
        horizontalalignment="center",
        verticalalignment="bottom"
    )
    ax.annotate(
        "",
        xy=(x0,y2),
        xytext=(x2,y2),
        arrowprops=dict(facecolor=textcolor, arrowstyle="<->")
    )

#--- label regions of probability
if plottype == "topmedian":
    ax.annotate(
        "50% of total probability",
        horizontalalignment="center",
        verticalalignment="center",
        xy=(0.44,0.4),
        xytext=(0.2,0.75),
        xycoords=ax.transAxes,
        arrowprops=dict(facecolor=textcolor, arrowstyle='-|>', relpos=(0.55,0.5))
    )
    ax.annotate(
        "25% of total probability",
        horizontalalignment="left",
        verticalalignment="center",
        xy=(0.53,0.75),
        xytext=(0.6,0.9),
        xycoords=ax.transAxes,
        arrowprops=dict(facecolor=textcolor, arrowstyle='-|>', relpos=(0.05,0.5))
    )
    ax.annotate(
        "25% of total probability",
        horizontalalignment="left",
        verticalalignment="center",
        xy=(0.62,0.3),
        xytext=(0.65,0.65),
        xycoords=ax.transAxes,
        arrowprops=dict(facecolor=textcolor, arrowstyle='-|>', relpos=(0.15,0.5))
    )
    x0 = population_mean
    y = 0.25 * population.pdf(x0)
    x1 = population.ppf(0.75)
    ax.annotate(
        "median deviation\nfrom median",
        xy=(x0,y),
        xytext=(x1 + (x0-x1)/2, y*0.9),
        horizontalalignment="center",
        verticalalignment="top"
    )
    ax.annotate(
        "",
        xy=(x0,y),
        xytext=(x1,y),
        arrowprops=dict(facecolor=textcolor, arrowstyle="<->")
    )

plt.show()

The code below was used to create figure 4.

np.random.seed(42)

x = np.linspace(-4, 4, 1000)

fig, axs = plt.subplots(3, 3, figsize=(7.88, 5), tight_layout=True)

for index, num_outliers in enumerate([0, 500, 1000, 2000, 5000, 10000, 20000, 50000, 100000]):

    sample = np.random.normal(loc=0, scale=1, size=500000)
    sample[0:num_outliers] = [5] * num_outliers # replace some samples with outliers
    mean = np.mean(sample)
    median = np.median(sample)
    std = np.std(sample)
    mad = stats.median_abs_deviation(sample, scale="normal")
    
    pdf = stats.norm(0, 1).pdf
    est_pdf = stats.norm(mean, std).pdf
    robust_pdf = stats.norm(median, mad).pdf

    ax = axs[index // 3, index % 3]

    ax.plot(x, pdf(x), label="Population")
    ax.plot(x, robust_pdf(x), label="Robust Estimate")
    ax.plot(x, est_pdf(x), label="Traditional Estimate")
    ax.tick_params(
        axis="x",
        which="both",
        top=False,
        bottom=False,
        labelbottom=False,
    )
    ax.tick_params(
        axis="y",
        which="both",
        left=False,
        right=False,
        labelleft=False,
    )
    ax.text(
        0.0125,
        0.975,
        f'{num_outliers:,} Outliers',
        transform=ax.transAxes,
        verticalalignment="top",
        horizontalalignment="left",
        fontsize=10
    )
    if (index == 0):
        ax.legend(fontsize=5)

plt.show()

Arbitrary Statistical Models

2025-01-20T00:00:00Z

This post might be the only time Stata code appears on the blog. Most of the software I share here will be written in a freely available language such as Python. But I’ve had these notes tucked away for awhile, and someone else may find them useful. Even long-time Stata users might be unaware of its versatility for developing arbitrary statistical models. We can see that flexibility by analyzing the same data set using non-linear regression, maximum likelihood estimation, and Bayesian analysis.

Here is the initialization code that I typically use for all Stata scripts Experienced Stata users will note that I do not follow the normal convention of abbreviating commands and options. I find code much less confusing when they are spelled out in full.. The version command is one of my favorites. It directs Stata to execute the remainder of the script as if it were the specified Stata version. In this case I’m using version 18. In the future when newer Stata versions are available, this command will tell the application not to use updated versions of alogrithms and equations. This feature is a big help for reproducibility.

version 18          // Stata version used to create/execute this script
set more off        // disable pause of output display every full screen
clear all           // ensure clean slate
capture log close   // close any pending logs
set seed 123456789  // consistent random number generation

For this analysis I’ll reuse a commonly cited example from the now-retired OpenBUGS software. The data are length and age measurements for 27 captured dugongs (sea cows) Ratkowsky, David A.. Nonlinear Regression Modeling: A Unified Practical Approach. Switzerland, M. Dekker, 1983.. Carlin and Gelfand (1991)Carlin, B. P., and A. E. Gelfand. “An iterative Monte Carlo method for nonconjugate Bayesian analysis.” Statistics and Computing, vol. 1, no. 2, Dec. 1991, pp. 119–28. Scopus, doi:10.1007/BF01889986. model the data from this example using a simple nonlinear growth curve that increases rapidly at early ages and tends to an asymptote as the age increases. In simple form, $y = α - β \cdot λ^{x}$ , where $y$ starts at $α - β$ when $x = 0$ and, provided $λ < 1$ , asymptotically approaches $α$ . Equation 1 describes the model more formally; in particular we assume that any errors in the data are normally distributed with zero mean and an unknown standard deviation.

$y \sim α - β \cdot λ^{x} + N (0, σ^{2})$

We’ll use that same model for all the examples that follow. As a check for the results that we get from Stata, a simple. model in Stan provides what I consider the “gold standard” for estimates of model parameters. Table 1 details the parameter estimates from this model.

data {
        int<lower=0> N;
        array[N] real<lower=0> x;
        array[N] real<lower=0> y;
    }
    parameters {
        real alpha;
        real beta;
        real<lower=.5, upper= 1> lamda;
        real<lower=0> sigma;
    }
    transformed parameters {
        array[N] real m;
        for (i in 1:N)
            m[i] = alpha - beta * pow(lamda, x[i]);
    }
    model {
        // priors
        alpha ~ normal(0.0, 1000);
        beta  ~ normal(0.0, 1000);
        lamda ~ uniform(0.5, 1);
        sigma ~ inv_gamma(0.01, 0.01);

        // likelihood
        y ~ normal(m, sigma);
    }

Parameter estimates from Stan
	Mean	MCSE	StdDev	5%	50%	95%
alpha	2.7	3.5×10^-3	0.069	2.6	2.6	2.8
beta	0.97	3.2×10^-3	0.072	0.85	0.97	1.1
lambda	0.86	1.6×10^-3	0.031	0.81	0.87	0.91
Sigma	0.099	7.2×10^-4	0.015	0.078	0.097	0.12

Now we can turn to Stata. For reproducibility I’ll show the full data set as it’s loaded into Stata.

The Stata algorithms need initial guesses for the model parameters, so here’s what we’ll use. We could theoretically pick any values here, but the code will run much quicker if the guesses are at least in the ball park.

local alpha = 1
local beta  = 1
local lamda = 0.9

Non-Linear Least Squares

A simple approach for finding parameter values is non-linear least squares with Stata’s nl command. It makes assumptions that are not always appropriate (e.g. errors are normally distributed) but even when those assumptions don’t hold, the approach may be helpful as a first step. The nl results can provide good initial values and, potential estimates for the error variance. The simplest way to use nl is with expressions.

nl (y = {alpha} - {beta} * {lamda} ^ x), ///
    initial(alpha `alpha' beta `beta' lamda `lamda') ///
    nolog

Although not necessary for this example, more complicated models may be implemented as “programs.”

Stata requires programs used for non-linear least squares to have a name that starts with “nl.” That prefix is then omitted when the program is referenced in the actual nl command.

capture program drop nlgrowth
program define nlgrowth
    syntax varlist(min=2 max=2) if, at(name)
    local y: word 1 of `varlist'
    local x: word 2 of `varlist'
    tempname alpha beta lamda
    scalar `alpha' = `at'[1, 1]
    scalar `beta'  = `at'[1, 2]
    scalar `lamda' = `at'[1, 3]
    replace `y' = (`alpha' - `beta' * `lamda' ^ `x') `if'
end

nl growth @ y x, ///
    parameters(alpha beta lamda) ///
    initial(alpha `alpha' beta `beta' lamda `lamda') ///
    nolog

local alpha_nl = _b[/alpha]
local beta_nl  = _b[/beta]
local lamda_nl = _b[/lamda]

Least squares provides reasonable estimate of the error variance. This estimate can be helpful in other methods (e.g. maximum likelihood and bayes), so we’ll save it.

// σ² ≈ MSE = RSS / (n - p)
local sigma = sqrt(e(rss) / (_N - 2))

One benefit of nl is the ability to use predictnl to generate new variables based on the parameters. Note, though, that we can only do this if the model can be written as an expression in addition to or instead of a program. That's the case here. As an option, we can also ask for confidence intervals.

quietly predictnl pred_nl = (_b[/alpha] - _b[/beta] * _b[/lamda] ^ x), ///
    ci(low_nl high_nl)

Bayesian (MCMC)

For a completely different approach, we can use Bayesian analysis to parameterize our model. As with non-linear regression, Stata can do that easily; all it needs is the expression corresponding to our model as well as our prior probabilities. We’ll use standard non-informative priors and the same initial guesses as above.

bayesmh y = ({alpha} - {beta} * {lamda} ^ x), ///
    likelihood(normal({var})) ///
    prior({alpha beta}, normal(0, 1000)) ///
    prior({lamda}, uniform(0.5, 1)) ///
    prior({var}, igamma(0.1, 0.1)) ///
    initial({alpha} `alpha' {beta} `beta' {lamda} `lamda' {var} `sigma'^2)

For more complicated models, we can use a custom “program evaluator.” A log-likelihood evaluator is simpler than a log-posterior evaluator because we're okay using built-in distributions as priors for the parameters.

Our model has no linear combinations for the independent variable $x$ , so ideally we would just define a constraint that fixed the slope. Currently the bayesmh command doesn't allow constraints with program evaluators, though. To work around that and avoid any spurious warnings for variables that we don't use, we pass x as an extra variable instead. Here’s a program evaluator for our example.

capture program drop growth_lleval
program define growth_lleval
    // Arguments:
    //   lnden: the name of a temporary scalar to be filled in with an overall
    //          log-likelihood value
    //   alpha, beta, lamda, sigma: model parameters (not linear equations)
    args lnden alpha beta lamda sigma

    // Make the code a little bit more readable with clearer references
    local x "$MH_extravars"
    local y "$MH_y"

    tempvar lnfj
    quietly generate double `lnfj' = ///
        lnnormalden(`y', (`alpha' - `beta' * `lamda' ^ `x'), `sigma') ///
        if $MH_touse

    summarize `lnfj', meanonly

    // If there was a problem, don't report an actual result but return a
    // missing value instead. This is the Stata convention. To detect problems
    // we ensure that the number of generated values is the same as the number
    // of observations.
    if r(N) < $MH_n {
        scalar `lnden' = .
        exit
    }

    scalar `lnden' = r(sum)
end

And here’s how we use that program to estimate posterior parameter values.

// Bayes results include the MCMC posteriors and require significantly more
// storage space. Use a temporary file to hold them.
tempfile bayes_file

bayesmh y, noconstant ///
    llevaluator( ///
        growth_lleval, ///
        extravars(x) ///
        parameters({alpha} {beta} {lamda} {sigma}) ///
    ) ///
    prior({alpha beta}, flat) ///
    prior({lamda}, uniform(0.5, 1)) ///
    prior({sigma}, igamma(0.01, 0.01)) ///
    initial({alpha} `alpha' {beta} `beta' {lamda} `lamda' {sigma} `sigma') ///
    saving(`bayes_file')

// For a simple comparison, save the mean values from the MCMC posterior
local alpha_bs = e(mean)[1, "alpha"]
local beta_bs  = e(mean)[1, "beta"]
local lamda_bs = e(mean)[1, "lamda"]
local sigma_bs = e(mean)[1, "sigma"]
local mcmc_samples = e(mcmcsize)

With the Bayesian approach, Stata lets us perform more extensive analyses. To do that, we can load the posterior data in a separate data frame. The resulting dataset will contain the parameter values realized in the MCMC chains. With a small bit of housekeeping, we can randomly sample from those posterior values. The final results include these samples as a way to visualize our model’s uncertainty as a Bayesian alternative to confidence intervals.

tempname posterior
frame create `posterior'
frame `posterior': use `bayes_file'
quietly frame `posterior' {
    expand _frequency
    keep eq*
    rename eq0_p1 alpha
    rename eq0_p2 beta
    rename eq0_p3 lamda
    rename eq0_p4 sigma
    generate double randu = runiform()
    isid randu
    sort randu
}

Maximum Likelihood (ml) Estimation

Finally, Stata also support maximum likelihood estimates. As with our previous approaches, we can do that simply with a substitutable expression.

mlexp (lnnormalden(y, {alpha} - {beta} * {lamda} ^ x, {sigma})), ///
    from(alpha = `alpha' beta = `beta' lamda = `lamda' sigma = `sigma') ///
    difficult nolog

// The "predictnl" command also works after "mlexp"
quietly predictnl pred_ml = (_b[/alpha] - _b[/beta] * _b[/lamda] ^ x), ///
    ci(low_ml high_ml)

More complex problems may benefit from a complete “program.” To demonstrate that, we use the linear form for the maximum likelihood estimation, so this program is a “method-lf evaluator.” Method-lf evaluators are required to evaluate the observation-by-observation log likelihood ln(Lj), j = 1, …, N where the subscript j indexes the observations. Because the model doesn't have any parameters that create linear functions of $x$ (i.e. no scale or location) we constrain the linear coefficient to be 1. We'll also force the constant term to be zero when we execute the ml command.

constraint define 1 [xb]:x = 1

capture program drop growth_lfeval
program define growth_lfeval
    // Arguments:
    //   lnfj: variable to be filled in with observation-by-observation
    //         values of ln(Lj)
    //   x: independent variable obtained via a linear equation where the
    //      slope is constrained to be 1 (above) and no constant is allowed
    //      (below)
    //   alpha, beta, lamda, sigma: model parameters (not linear equations)
    args lnfj x alpha beta lamda sigma

    // Get access to the dependent variable (y). This isn't strictly
    // necessary (and possibly degrades performance), but it makes the
    // expressions easier to read.
    local y $ML_y1

    quietly replace `lnfj' = ///
        lnnormalden(`y', (`alpha' - `beta' * `lamda' ^ `x'), `sigma') ///
        if $ML_samp == 1
end

This is non-interactive mode, so the maximize option is specified. We also specify nopreserve as good practice even though it has no effect for linearform (lf) methods. In general, it tells Stata to assume that the evaluator is correctly using $ML_samp; without that assumption Stata takes extra precautions that degrade performance.

ml model linearform growth_lfeval ///
    (xb: y = x, noconstant) ///
    (alpha:, freeparm) ///
    (beta:,  freeparm) ///
    (lamda:, freeparm) ///
    (sigma:, freeparm) ///
    , constraint(1) ///
    init(xb:x = 1 alpha = `alpha' beta = `beta' lamda = `lamda' sigma = `sigma') ///
    maximize search(off) difficult nolog nopreserve

local alpha_ml = _b[/alpha]
local beta_ml  = _b[/beta]
local lamda_ml = _b[/lamda]
local sigma_ml = _b[/sigma]

The Results

Thankfully, all of the different modeling approaches yield similar results, as table 2 indicates. We can also compare the models by visualizing models based on the estimated parameters. Figure 1 shows that comparison. The final panel uses mean estimates from each approach to show all of them on a single plot.

Stata results for multiple models compared to Stan estimates (with 95% confidence intervals or 95% credible sets)
	$α$	$β$	$λ$
Stan	2.7 [2.6, 2.8]	0.97 [0.85, 1.1]	0.86 [0.81, 0.91]
`nl` expression	2.66 [2.53, 2.79]	0.96 [0.82, 1.11]	0.87 [0.82 0.92]
`nl` program	2.66 [2.53, 2.79]	0.96 [0.82, 1.11]	0.87 [0.82 0.92]
`bayesmh` expression	2.65 [2.49, 2.87]	0.98 [0.79, 1.19]	0.85 [0.75, 0.93]
`bayesmh` program	2.65 [2.54, 2.80]	0.97 [0.80, 1.11]	0.86 [0.80, 0.91]
`mlexp`	2.65 [2.54, 2.78]	0.96 [0.83, 1.09]	0.87 [0.82, 0.92]
`ml` program	2.65	0.96	0.87

gr_setscheme
local color1 `.__SCHEME.color.p1'
local color2 `.__SCHEME.color.p2'
local color3 `.__SCHEME.color.p3'
local color4 `.__SCHEME.color.p4'
local color5 `.__SCHEME.color.p5'
local color6 `.__SCHEME.color.p6'
local ciarea `.__SCHEME.color.ci_area'
local ciline `.__SCHEME.color.ci_line'

// One plot for each method separately to highlight that method's unique
// features, and then a combined plot to allow comparison of the results.

// Non-linear Least Squares - include confidence interval
twoway ///
    (rarea low_nl high_nl x, fcolor("`ciarea'") lwidth(none)) ///
    (function y = (`alpha_nl' - `beta_nl' * `lamda_nl' ^ x),  ///
        range(x) lcolor("`color3'") lwidth(1)) ///
    (scatter y x, mcolor("`color1'")), ///
    legend(off) ///
    title("Non-linear Least Squares", size(11pt) position(12) ring(0)) ///
    subtitle("w/ 95% CI", size(10pt) position(5) ring(0)) ///
    xtitle("") xlabel(none) ytitle("") ylabel(none) ///
    name(nl, replace) nodraw

// Maximum Likelihood - also with confidence interval
twoway ///
    (rarea low_ml high_ml x, fcolor("`ciarea'") lwidth(none)) ///
    (function y = (`alpha_ml' - `beta_ml' * `lamda_ml' ^ x),  ///
        range(x) lcolor("`color4'") lwidth(1)) ///
    (scatter y x, mcolor("`color1'")), ///
    legend(off) ///
    title("Maximum Likelihood", size(11pt) position(12) ring(0)) ///
    subtitle("w/ 95% CI", size(10pt) position(5) ring(0)) ///
    xtitle("") xlabel(none) ytitle("") ylabel(none) ///
    name(ml, replace) nodraw

// Bayesian - show plausible trajectories by sampling from the posterior
forvalues sample = 1/500 {
    local i = runiformint(1, `mcmc_samples')
    local a = _frval(`posterior', alpha, `i')
    local b = _frval(`posterior', beta,  `i')
    local l = _frval(`posterior', lamda, `i')
    local graph = "`graph' (function y = (`a' - `b' * `l' ^ x),"
    local graph = "`graph' range(x) lwidth(thin) lcolor(`ciline'%05))"
}

twoway ///
    `graph' ///
    (function y = (`alpha_bs' - `beta_bs' * `lamda_bs' ^ x),  ///
        range(x) lcolor("`color5'") lwidth(1)) ///
    (scatter y x, mcolor("`color1'")), ///
    legend(off) ///
    title("Bayesian (MCMC)", size(11pt) position(12) ring(0)) ///
    subtitle("w/ 500 posterior samples", size(10pt) position(5) ring(0)) ///
    xtitle("") xlabel(none) ytitle("") ylabel(none) ///
    name(bayes, replace) nodraw

// Summarized results on single plot - using mean values for parameters
twoway ///
    (function y = (`alpha_nl' - `beta_nl' * `lamda_nl' ^ x),  ///
        range(x) lcolor("`color3'") lwidth(1)) ///
    (function y = (`alpha_ml' - `beta_ml' * `lamda_ml' ^ x),  ///
        range(x) lcolor("`color4'") lwidth(1)) ///
    (function y = (`alpha_bs' - `beta_bs' * `lamda_bs' ^ x),  ///
        range(x) lcolor("`color5'") lwidth(1)) ///
    (scatter y x, mcolor("`color1'")), ///
    subtitle("Mean Estimates", size(11pt) position(12) ring(0)) ///
    legend(order( ///
        4 "Observations" ///
        1 "Non-linear Least Squares" ///
        2 "Maximum Likelihood" ///
        3 "Bayesian (MCMC)" ///
    ) ///
    rowgap(0) keygap(1) symxsize(9) size(8pt) region(margin(small)) ///
    cols(1) position(5) ring(0)) ///
    xtitle("") xlabel(none) ytitle("") ylabel(none) ///
    name(all, replace) nodraw

graph combine nl ml bayes all, cols(2) name(models, replace)

Stata Model Fitting Approaches

All of the approaches yield very similar results, though each has its own strengths and weaknesses. I tend to use non-linear least squares for “quick and dirty” analysis and follow up with Bayesian estimates if I want more thorough or more detailed results. Maximum likelihood is an alternative for those cases where Bayesian analysis may not be appropriate.

Dark Mode Support

2025-01-13T00:00:00Z

Those of you who read at night may have noticed that this site has a dark mode. Pages follow the lead of your web browser. When your browser switches to dark mode, they change with it. Or, at least, most pages do. This particular page works differently. There is a button in its upper right corner that manually sets light or dark mode. That button gives you direct control over the mode. This page is an exception because it discusses the site’s dark mode implementation, and I thought it would be helpful to let you directly compare the two styles.

The code that governs the mode is pretty simple, though it does rely on relatively recent features in the relevant web standards. The main control is a hidden checkbox that’s toggled on and off by the button:

<input id="dark-mode" type="checkbox" name="dark-mode" hidden="true" />
<label for="dark-mode">
    <!-- svg icon here -->
</label>

The CSS styles change based on the state of that checkbox.

html:has(#dark-mode:checked) {
    /* style changes for dark mode */
}

Most of the changes are straightforward: the background becomes dark gray and the text color changes to off-white, for example. To reduce the blinding effect of images, their brightness is reduced with a filter.

NCATS biologist Michele Ceribelli, Ph.D., prepares samples for a high-throughput flow cytometry analysis in a 384 well plate. Credit: Daniel Soñé Photography, LLC. (Public Domain.)

Charts and plots provide more control and flexibility, so they have optimized light and dark mode versions using different chart styles. The web page includes both images, but only one is visible at a time. Switching to and from dark mode changes which image becomes visible.

Sample chart to illustrate the differences between light and dark mode.

Since this page lets you manually switch modes, you might ask why every page doesn’t offer that option. Unfortunately, any practical implementation requires JavaScript or cookies, and this site has neither. This constraint not only keeps the site’s implementation simple, it also ensures that you, as a reader, have complete privacy. There is no way I can track your behavior on the site. The problem is persistence. It is possible to manually change the mode on any given page (such as this one), but there is no way to remember that preference and share it with other pages. So you could manually set dark mode on a particular page, but as soon as you navigated to a different page on the site, you would be confonted with light mode again. That would be frustrating and annoying, so I’ve stuck with what’s possible. Regular pages simply reflect the browser’s setting.

How the Site is Built

2025-01-06T00:00:00Z

I’m truly grateful to all the generous developers whose open source software makes this site possible. They certainly deserve acknowledgment as well as other forms of support. And since some readers might be curious to learn the reasons behind my choices, a brief post on the site’s construction seemed appropriate. When I began to contemplate a new blog, there were really only two requirements, and they both seemed fairly simple:

No JavaScript in the browser.
Nicely rendered mathematics.

It turns out that those requirements aren’t so trivial after all. The two major math rendering libraries are both designed to run as JavaScript in the browser. In theory, both can also execute in a node.js environment during the build process, but that approach introduces significant complexity and compromises. Furthermore, the output formats seem to be limited to unattractive SVG graphics or a Rube Goldberg combination of HTML, web components, and CSS.

Fortunately, modern browsers have pretty good support for MathML and there are tools that generate MathML from LaTeX input. So it’s possible to write mathematics in the format familiar to most mathematical authors:

\def\d{\mathrm{d}}

\oint_C \vec{B}\circ \d\vec{l} = \mu_0
\left(
  I_{\text{enc}} +
    \varepsilon_0
    \frac{\d}{\d t}
    \int_S {\vec{E} \circ \hat{n}}\; \d a
\right)

And have it appear as a nice equation whose styling perfectly matches other content on the site:

$\oint_{C} \vec{B} \circ d \vec{l} = μ_{0} (I_{enc} + ε_{0} \frac{d}{d t} \int_{S} \vec{E} \circ \hat{n} d a)$

With a solution for mathematical content, putting everything together was pretty straightforward. There are plenty of static site generating applications, but Eleventy looked like a good fit for me. It provides easy control over the HTML and CSS that ultimately make up the site, and it’s implemented in language (JavaScript) and environment (node.js) with which I have a lot of familiarity. I’ve tried to keep other dependencies to a minimum, so most of the functionality is built into Eleventy. The site does rely on a few extensions, however, As of this writing, the applications, libraries, and tools that produce this site include:

Static site generator: Eleventy.
Math rendering: Ron Kok’s TEMML via Trae Blain’s plugin.
Additional markdown plugins for subscripts, superscripts, and, from Martin Ring, table captions.
Theme: Adopted from Vincent Dörig’s awesome LATEX.css.
Fonts: Latin Modern families from GUST.
Icons: Material Icons, brand icons, and custom.

For the writing process itself the typora editor from appmakes deserves a special acknowledgment. I’ve been using it since the early beta releases, and it was the primary writing tool for my last book, my PhD thesis, and various academic papers.

Hello again, world

2025-01-01T00:00:00Z

It's been way more than a minute, long enough to wonder if I can still write a decent blog post. After all, a lot can change in nearly a decade. Still, I have been thinking about writing again, and a blog post from Cory Doctorow finally helped me overcome my intertia. Cory says that “Writing is a cognitive prosthesis for me. Working things out on the page helps me work things out in my life.” That’s close to how I feel. For me, it’s only when I can write about a topic that I have confidence that I understand it. And since I love trying to figure things out, I’m going to have to write. To me the most relatable sentence in Cory’s piece is “[T]he very act of writing is so important to me that even if no one read me, I would still write.”

So I’m going to start again. And if you, dear reader, find the articles interesting, I hope you’ll join me. At this point I can’t commit to a specific schedule, but I’m sure it won’t be frequent enough to feel like spam. On the other hand, I do have enough ideas in mind to establish some momentum. So check back when you can; see if you find something you like, and feel free to let me know how I’m doing.