The Significance of Insignificance

Remarking on some recent poll results, Eugene says, "Repeat after me: I will not think that statistically insignificant changes in poll results are statistically significant -- even if I really, really like them."

In general, I agree with Eugene’s point (which he has made many times): people ascribe way too much import to small differences in polling results, without paying attention to the margin of error.

However, Eugene’s position may foster the false impression that all differences in poll results that fall within the margin of error are equally insignificant. Suppose that two candidates are separated by just 2 percentage points in the polls (say, 49% to 47%), and the margin of error is 3 percentage points for each figure. And suppose that two different candidates in another election -- or the same two candidates in a later poll -- are separated by 6 percentage points (say, 51% to 45%), again with a margin of error of 3 percentage points for each figure. While both differences are “insignificant” in the sense that the difference is within the combined margins of error, the latter result is clearly more significant than the former.

Indeed, the latter result would most likely have been deemed statistically significant had a very slightly lower level of confidence been applied. The margin of error is constructed using a conventional but essentially arbitrary confidence level. The typical convention is 95% confidence, but other levels of confidence could also be used; 90% and 99% are relatively common. These are the confidence levels employed by scientists, who don’t want to affirm a hypothesis unless they are very confident of it, and who are willing to remain agnostic in a wide range of cases. Lower levels of confidence might well be acceptable in other contexts, such as business, where some decisions have to be made without great confidence (e.g., should I plan to expand next year if I am 75% confident that consumer demand will pick up?). It’s not obvious what the appropriate level of confidence is for political prognostication, but I’ll put it this way: in the example given above, I would be willing to bet a larger amount of money on the second race than the first.

The broader point is that statistical significance is not an all-or-nothing proposition. Despite the way statistical significance is often taught, there is not a sharp discontinuity between significance and insignificance. Significance lies on a gradient.