Is bootstrap problematic in small samples?

Bootstrap estimates of standard errors are based on the assumption that the observed sample is the same as the true distribution, which is OK asymptotically. But a sample of size $n$ implies a distribution with $n$ mass points, which is quite unlike the true distribution if $n$ is small. For what sample sizes and what parent populations are the bootstrap estimates OK?

I had an impression that one of the main uses of bootstrap in statistics and econometrics is precisely in small samples. There, a bootstrap distribution is used when no analytical distribution is available and the sample is too small for the asymptotic distribution to be a good approximation of it. This makes Ed Leamer's criticism quite relevant and interesting. But perhaps my impression is wrong and I am misunderstanding things. Q: Is this a valid piece of criticism? If so, has the problem been studied in any detail? Have any solutions been proposed?

$\begingroup$ What's small? It's surely standard that -- in plain bootstrapping -- if your sample is say 1,2,3 on one variable then some of the time bootstrapping will yield samples that are useless for many purposes (notably that many calculations fail if there are too many ties). And no bootstrap sample will include 0 or 5 or 42 if those are possible values. You know this, and it's "no free lunch" again. Some researchers prefer to simulate from a plausible parent and they then have to worry what that is. $\endgroup$

$\begingroup$ Also, arguably, bootstrapping giving you poor results is sometimes to be written up as bootstrapping being a poor choice, but often the unwelcome lesson is that the dataset is not good enough for the analyst's (or client's) purposes. $\endgroup$

$\begingroup$ I see the bootstrap primarily as a tool for when the distribution of whatever it may be is difficult to derive analytically. $\endgroup$

$\begingroup$ @kjetilbhalvorsen, thank you! These were relevant threads. Consequently, my understanding is the following: we need the sample to be large enough so that it approximates the population well. Without that, bootstrap will not work well. I do not yet understand whether bootstrap will nevertheless work better than the asymptotic approximation (for certain pivotal quantities, it seemingly will) or not, and thus how universally it may be considered a cure for small sample size and preferred to the asymptotic distribution. $\endgroup$

1 Answer 1

My short answer would be: Yes, if samples are very small, this can definitely be a problem since the sample may not contain enough information to get a good estimate of the desired population parameter. This problem affects all statistical methods, not just the bootstrap.

The good news, however, is that ‘small’ may be smaller than most people (with knowledge about asymptotic behavior and the Central Limit Theorem) would intuitively assume. Here, of cause, I’m referring to the normal (naive) bootstrap without dependent data or other peculiarities. According to Michael Chernick, the author of ‘Bootstrap Methods: A guide for Practitioners and Researchers’, small may be as small as N=4.

But this number of distinct bootstrap samples gets large very quickly. So this is not an issue even for sample sizes as small as 8.

For reference, see Chernick's great answer to a very similar question: Determining sample size necessary for bootstrap method / Proposed Method

Of cause the suggested sample sizes are subject to uncertainty and no universal threshold for a minimum sample size can be specified. Chernick therefore suggests to increase the sample size and study the convergence behavior. I believe is a very reasonable approach.

Here’s another quote from the same answer, which somehow addresses the premise you quoted initially:

Whether or not the bootstrap principle holds does not depend on any individual sample "looking representative of the population". What it does depend on is what you are estimating and some properties of the population distribution (e.g., this works for sampling means with population distributions that have finite variances, but not when they have infinite variances). It will not work for estimating extremes regardless of the population distribution.