Coherence vs. Consistency

Suppose you’ve just been hired as the new head of research for an organization that conducts public opinion surveys. The firm has conducted a survey for the last 30 years tracking answers to the question, “Do you consider yourself conservative or liberal?” Upon reflection, you realize this was probably a poor choice of question. Some people might consider themselves “moderate,” or even something exotic like “libertarian” or “communitarian.” The current survey is biased against such people, and depending on how these people categorize themselves when forced to pick from the two options, the survey might also be biased in favor of the conservative or the liberal camp.

To remedy the problem, you plan to change your survey to reflect these other options. But then your subordinate at the firm explains that the question has been in this form for 30 years, and for the sake of consistency it should be kept the same.

Is your subordinate necessarily wrong? Often the whole purpose of these surveys is to track changes in public opinion over time. If you change the question, the answers in subsequent years will no longer be strictly comparable to those in earlier years. You will have introduced a discontinuity in the data stream. Alternatively, you could keep using the same question in order to observe the real changes in the number of people who choose each category when faced with a binary choice. There’s a serious trade-off here. How should consistency over time be weighed against coherence at a moment in time?

To avoid the trade-off, you might decide to ask both questions. But which question you ask first will affect how the other question is answered. For instance, if you ask the question with more options first, survey respondents might refuse to answer when faced with the binary choice later. So you could run two different surveys, with two different pools of respondents. But now you’re looking at another kind of trade-off, because it takes more resources to conduct more surveys.

And sometimes you don’t have the option of conducting more surveys. The example best known to economists is the construction of a price index like the CPI. The characteristics and quality of the products change continually, so recording the price of a seemingly given product each year is a lot like asking a slightly different question every year. Yet the whole raison d’etre for the CPI is to allow comparisons over time.

And the CPI and the political survey turn out to be even more similar, because just as the characteristics of products change over time, so do the definitions of words and the perceptions of ideologies. This is a conundrum: Change is the primary reason we want to keep records over time. But change, by its very nature, persistently undermines our attempts to record it consistently.