As a result of Michael Moore’s SiCKO and the ensuing public debate, you’ve probably heard that the U.S. healthcare system United States ranks 37th in performance compared to other countries. Meanwhile, France and Canada’s systems both rank in the top 10, according to the World Health Organization. Here’s CNN.com’s story on the subject.
I was already skeptical of these numbers when I first heard them, because health performance statistics are affected by many things besides healthcare, such as crime, nutrition, and lifestyle choices. But I had assumed the statistics at least measured actual health outcomes. It took a post from David Masten at Distributed Republic to make me realize I had assumed incorrectly. Masten draws attention to this crucial little sentence (emphasis added):
The rankings are based on general health of the population, access, patient satisfaction and how the care’s paid for.Including the mode of payment when measuring the system’s performance is, as Masten astutely observes, assuming what you’re trying to prove. After all, the whole question is how healthcare ought to be financed – publicly or privately, with insurance or out-of-pocket payments, etc.
To see the illogic for myself, I downloaded the relevant WHO report and the study it was based on. But before I could verify the factors included in the health performance index, I had to figure out which index to look at. It turns out that the U.S. ranks 37th on the “overall performance index.” But on this index, while it’s true that France is #1, Canada does not rank in the top 10 – it’s only #30. There is another index, “overall health system attainment,” on which the U.S. ranks #15 (while France and Canada are #6 and #7, respectively). As far as I can tell, the two indices are based on the same underlying data, but with the “overall performance index” calibrated according to some measure of how well the country is theoretically capable of doing. I’m still trying to figure out exactly how this calibration works. In any case, it looks an awful lot like someone cherry-picked the results to make the U.S.’s relative performance look worse than it is. Contrary to CNN.com (and possibly Michael Moore – I haven’t seen the movie yet), there is no index that has both Canada and France in the top 10 and the U.S. at 37.
But back to Masten’s point. Both of these indices include “financial fairness” (FF) as a factor with 25% weight in measuring the system’s performance. FF is measured first by finding a household’s contribution to health expenditure as a percentage of household income (beyond subsistence), and then looking at the distribution of this percentage over all households. The wider is the distribution, the worse a nation will perform on the health performance index (other things equal). But it should not be surprising at all that a larger percentage of poor people’s income will be spend on health than would be spent by the rich. Insofar as healthcare is treated as a necessity, we should expect that people will spend a decreasing fraction (not a decreasing amount, but a decreasing fraction) of their income on healthcare as their income increases. Rich people tend to spend a larger percentage of their income on luxuries than do the poor.
More importantly, the distribution of household contributions will obviously decline when the government shoulders more of the health spending burden. In the extreme, if the government pays for all healthcare, every household will spend the same percentage of their income – zero – on healthcare. In other words, this measure of health outcomes necessarily makes countries that rely on private payment look inferior.
It gets worse. The ostensible reason for including FF in the healthcare performance index is to consider the possibility of people landing in dire financial straits because of their health needs. It’s debatable whether this factor deserves inclusion in a strict measure of actual health performance – but let’s suppose it does. FF does not actually measure exposure to risk of impoverishment. FF is based on cubing (!), for each household, the difference between that household’s contribution and the average household’s contribution to healthcare. Consequently, FF is negatively affected by households that spend a larger percentage of their income on healthcare than others. But FF is also negatively affected by households that spend a much smaller percentage of their income on healthcare than others! This is senseless, but it’s a natural result of focusing on distribution instead of the effectiveness of the healthcare people receive.
More later.
[UPDATE: See also Part 2 and Part 3.]
30 comments:
Glen,
Notwithstanding whatever problems exist in the WHO's methodology, are you contesting either of the following claims:
(1) the U.S. gets a relatively poor return on its health care spending;
and/or
(2) U.S. health outcomes -- measured by a variety of population health indices -- place it somewhat below average for industrialized nations.
Some version of the above two claims are generally accepted by the vast majority of health policy commentators, so I would be interested in knowing if you disagree, and why you would. If you do not disagree, I'm not sure what policy implications follow from pointing out flaws in the WHO ranking methodology.
www.medhumanities.org
Daniel -- I'm not convinced of those points. I'm not saying they're false, but part of the reason the "vast majority" of health policy commentators accept them is that they're looking at the WHO rankings. My point in this post, and the next one, is that these rankings are highly problematic.
It is undoubtedly true that the U.S. healthcare system has problems. No debate there, although we could have a debate about the causes. In my assessment, most of the causes relate to current tax policy, mandated benefits laws, and other government interventions. And, it should be observed, other countries' healthcare systems have problems, too. If we want a fair international comparison of problems, we need performance measures better than the WHO rankings.
Glen,
The WHO rankings are hardly the only ones indicating that the U.S. performs significantly worse compared to other industrialized nations, but I appreciate the clarification.
As to the causes of our health care problems, I am well aware of the prevailing views of the causes on this blog. Though I do not feel blog comments are an appropriate venue to air such discourse, I will say that I most assuredly disagree with the generally-stated views on this blog.
What about Daniels, Kawachi, and Kennedy's work on social epidemiology? The existence of a health gradient with a strong correlation to income inequality has been demonstrated for hundreds of years? Are you persuaded by the data presented by the social epidemiologists?
In any case, thanks for the clarification.
Daniel:
U.S. health outcomes -- measured by a variety of population health indices -- place it somewhat below average for industrialized nations.
This may be true, but to blame this on our not having socialized health care is fallacious. The United States leads the world in obesity, which negatively influences health outcomes but is almost certainly not a product of our health care system.
I remember hearing a year or two ago about a study that found that Americans were less healthy than Britons at all income levels, even in the highest income quintile. AFAIK, it's more or less universally agreed that well-off Americans get better health care than well-off Britons, so it's clear that health outcomes are not a simple function of consumption of health care.
Also, what's "a variety of population health indices"? Almost all the time I hear the claim about US health outcomes made, there are just two indices adduced as evidence: life expectancy at birth and infant mortality. This is a terrible pair of numbers to use to measure health system quality, as:
(a) it's really only 1.5 numbers (since LE at birth, as opposed to say LE at 20 or 40 or 60, is dramatically affected by infant mortality)
(b) both numbers are, as Brandon says, heavily influenced by factors that have nothing to do with the health care system.
If you look at other metrics (e.g. cancer survival rates) that are more plausibly directly connected to the health care system, the US does much better; Olaf Gersemann in _Cowboy Capitalism_ has a nice roundup of some such stats.
Brandon,
I neither said nor implied that our problems are attributable to the lack of national health care, which is not necessarily socialized medicine. In socialist systems, the government controls the means of production. That is not the case in GB, for example, where docs are independent contractors.
What this thread is about is assessing the nature of the problem. If we agree that the U.S. does have health care problems -- and I was trying to ascertain whether Glen does believe this -- then we can move on to assess how best to remedy that problem.
Anony--
As to the variety of indices, the IOM, which is hardly a bastion of socialism, has a 2002 report on public health that analyzes a number of these matrices.
Daniel:
Fine. But it doesn't necessarily indicate any problems with our health-care systems. The US health-care systems (or some of them, anyway) may actually do a very good job of caring for a uniquely unhealthy population.
The existence of a health gradient with a strong correlation to income inequality has been demonstrated for hundreds of years?
It's worth pointing out that the "Black Report" in the UK came up with similar findings, despite the UK having a government funded health system.
The point surely is that creating a national health service does not (by itself) solve this problem.
This is a test
I have just found your site as a result of searching for a critique that I read a few months ago of the COmmonwealth Fund's methodology. Can anyone point me to it? There is discussion of the US outcomes being influenced by a more heterogeneous population, higher percentage of immigrants, etc.
cubing the difference will account for a negative sign... if two families would cancel each other out normally, they will also cancel each other out if they cube the differences, they wont add, they'll still subtract.
Anon -- that would be true, except that in their formula they take the absolute value of the cubed difference. So there's no negative sign.
NEWSFLASH - the 2009 WHO ranking for the United States is now 72nd out of 191 nations. You barely edged out Cuba.
This is just one more reminder that the private sector and competitive market forces, not the federal government, are the best means to meeting our country's rapidly expanding health care needs.
I was looking for a way to try and do something positive about it, and just signed a petition with the U.S. Chamber of Commerce here to help emphasize that. We need to get involved!
Anon of 6/23/09 -- There hasn't been a new WHO healthcare ranking since 2000. The number you cite (72nd of 191) is the U.S. rank for one of the five components in the WHO's 2000 ranking.
Um Canda was 30th on the list get your facts straight. I discredited this account from the very first sentence.
Anon -- try reading the whole post! The first paragraph is a paraphrase of the linked CNN.com article, which does indeed say that Canada is in the top 10: "Both the French and Canadian systems rank in the Top 10 of the world's best health-care systems, according to the World Health Organization. The United States comes in at No. 37."
But as I point out later in the post, there is more than one WHO ranking -- and there is no ranking in which France is better than 10 while the U.S. is 37th. In the one with the U.S. at 37, Canada is (as you say) 30th.
Here's the problem. I read a small portion of the report related to Financial Funding, and it didn't take long for me to find a blatant error in your original post, which leads me to question your motives. All forms of funding are included and evaluated, so your assertion that the citizens of publicly funded health-care systems pay $0 is blatantly false, as these citizens pay into the system with tax dollars, and determining their per capita health-care costs is not difficult.
Gary -- you seem awfully anxious to dismiss my analysis! But if you look a little closer, I think you'll see that my main argument stands.
When I wrote this blog post, I hadn't realized that all forms of health funding were included in the FF measure -- so I mistakenly said that everyone would pay $0. However, I corrected that error in my subsequent Cato policy report on the subject. And as my analysis there makes clear, the FF factor makes socialized systems look superior, for essentially the same reason I stated in the original blog post. Here's the relevant passage:
"The FF measure rewards nations that finance health care according to ability to pay, rather than according to actual consumption or willingness to pay. In most countries, a household's tax burden is proportional to income, or progressive (i.e., taxes consume an increasing share of income as income rises). Thus, a nation's FF score rises when the government shoulders more of the health spending burden, because more of the nation's medical expenditures are financed according to ability to pay. In the extreme, if the government pays for all health care, then the distribution of the health-spending burden is exactly the same as the distribution of the tax burden."
I encourage you to read the whole policy report. Even if you ultimately disagree with my FF analysis, there are many other problems with the WHO rankings.
Glen,
You seem awfully anxious to make your point that the ratings are flawed. As an associate professor of economics, I would expect you to do a little more homework before presenting such a strong influencing opinion in an open blog. As I said, the error was blatantly obvious to me after a few minutes of reviewing the FF section of the report. I will gladly review the WHO report further, but I suspect it will not bode well for your conclusions. This information can be colored in many different ways, but in my opinion, the ultimate measure of the effectiveness of a country's health-care system is how healthy its population is in relation to the system's cost per capita. The miraculous medical marvels we can accomplish with our shareholder driven health-care system and the cost of those miracles seems to be inversely proportional to our overall health as a nation. What a beautiful profit-generating industry. The more unhealthy we become, the more miracles we need and the more we're convinced that it's worth the price. Prevention isn't very profitable now, is it, Glen? I think 37 is about right. By the way Glen, the salaries of the physicians is most certainly not the problem. All their salaries combined account for only 2% of the 2.3 trillion dollar Health-care industry. Nor is it tort reform, although you can certainly understand why physicians would think that it is, as malpractice insurance has a significant impact on their bottom line. Hmmmm. Where's all that money going, Glen? To the insurance industry? To Wall Street?
Gary -- clearly you're spoiling for a healthcare fight. I'm sure someone on this big ol' internet will happily oblige you. But I'll stay focused on the topic of this blog post: the WHO rankings.
"As an associate professor of economics, I would expect you to do a little more homework before presenting such a strong influencing opinion in an open blog."
A blog is not the same as a policy paper or academic article. As it turned out, I missed a bit of fine print, but it didn't fundamentally alter the conclusion.
"I will gladly review the WHO report further, but I suspect it will not bode well for your conclusions."
Prejudge much? The conclusion of my report is not that the U.S. system is really the best; it is that the WHO rankings are flawed, and thus cannot tell us about the U.S. system's relative performance.
"This information can be colored in many different ways, but in my opinion, the ultimate measure of the effectiveness of a country's health-care system is how healthy its population is in relation to the system's cost per capita."
Guess what? That is not what the WHO rankings measure. So by your own standard, you should reject the WHO rankings.
But there are problems with your proposed measure as well. Chief among them is the fact that the health of a population is affected by numerous other factors besides healthcare, including nutrition, exercise, genetics, geography, environment, crime, traffic accidents, and so forth. Simply looking at life expectancy and other gross measures of health is wildly misleading. This is a problem with both the WHO rankings and your proposed alternative.
All the rest of your comments are unrelated to the validity of the WHO rankings, so I will let them pass.
I don't know why I continue to be amazed by our pervasive arrogance as a nation, exemplified by your initial skepticism expressed as, and I'm paraphrasing here, "If we're not #1 or at least in the top 10, there must be some inherent flaw in the measurement used to rate our country.
So, what exactly motivated you write a briefing paper discussing all of your criticisms of the WHO healthcare rankings? And how is this unrelated to the healthcare reform debate? Glen, as far as I can surmise, you're a libertarian. It seems very silly to me and a little disingenuous for you to try to pawn off your briefing paper and this blog as objective, independent research having nothing to do with your personal views, which I suspect would include a general suspicion of increased government involvement in our healthcare system.
Here are some excerpts from the WHO report, which you know, of course, is the May, 2000 report:
"Many questions about health system performance have no clear or simple answers –
because outcomes are hard to measure and it is hard to disentangle the health system’s
contribution from other factors."
So far, you and the report seem to be on the same page.
"This report finds that many countries are falling far short of their potential, and most are
making inadequate efforts in terms of responsiveness and fairness of financial contribution."
"These failings result in very large numbers of preventable deaths and disabilities in each
country; in unnecessary suffering; in injustice, inequality and denial of basic rights of individuals.
The impact is most severe on the poor, who are driven deeper into poverty by lack
of financial protection against ill-health."
At this point, I suspect you and the report part ways.
So let's see if I can stay on topic here. If a significant segment of the population has a higher mortality rate (conceding that this is only one measure, bear with me for the sake of argument) due in part to the inequities in the way the system grants access based on the financial funding scheme of same, is it possible for us to agree that this is a legitimate measure of performance for a healthcare system, albeit, only one?
Have you read The Jungle, by Upton Sinclair? In the wealthiest nation on earth, we have a significant portion of our population living in a healthcare jungle. Isn't there a moral imperative here? And if a healthcare system doesn't work for a significant portion of it's population, due in no small measure to the inequities of fairness created by the means with which the system is funded, how is this NOT related to healthcare system performance? In other words, if you have the wealth to afford the best health insurance or you work for an employer who can subsidize your health insurance costs so that you can afford access to an exceptional healthcare system, but your neighbor doesn't have either, and therefore only has access to an inferior healthcare system, don't you think this is one appropriate criterion (if only one) upon which to measure a healthcare system's performance? I'm not seeing the flaw here, unless I can add an underlying agenda to your arguments.
I believe the answer to a single question will resolve this debate for me. Glen, of all the news stations out there in TV land, which one do you believe consistently provides you with the most accurate information?
"I don't know why I continue to be amazed by our pervasive arrogance as a nation, exemplified by your initial skepticism expressed as, and I'm paraphrasing here, "If we're not #1 or at least in the top 10, there must be some inherent flaw in the measurement used to rate our country."
I never said that. I said I was skeptical because "health performance statistics are affected by many things besides healthcare..." I did not assume the statistics were flawed simply because the U.S. wasn't #1.
"So, what exactly motivated you write a briefing paper discussing all of your criticisms of the WHO healthcare rankings? And how is this unrelated to the healthcare reform debate? Glen, as far as I can surmise, you're a libertarian. It seems very silly to me and a little disingenuous for you to try to pawn off your briefing paper and this blog as objective, independent research having nothing to do with your personal views, which I suspect would include a general suspicion of increased government involvement in our healthcare system."
Yes, I am suspicious of government involvement in the healthcare system, and that is part of what motivated my interest in this topic. But upon examination, I think the flaws of the WHO statistics are objectively clear. Among other things, the WHO statistics are flawed because the authors incorporated their own biases into the statistics' construction. So it's rather ironic for you to accuse me of bias for pointing out biased statistics.
"If a significant segment of the population has a higher mortality rate (conceding that this is only one measure, bear with me for the sake of argument) due in part to the inequities in the way the system grants access based on the financial funding scheme of same, is it possible for us to agree that this is a legitimate measure of performance for a healthcare system, albeit, only one?"
I agree that the health of the worst off citizens is important. But I disagree that inequality per se is a relevant measure of healthcare performance, for reasons that I think even progressives should agree with. As I've pointed out before, an inequality measure punishes a country purely for inequality. Thus, a country with good care for half the population and excellent care for the rest would do worse (on the inequality measure) than a country with mediocre care for everyone. Or to put it another way, a country's ranking could be improved by reducing the quality of care for some citizens without improving care for anyone! This makes no sense at all. If you care about the health of the poor, then measure the health of poor directly.
But say you disagree. That means you value equality for equality's sake. That is a value judgment that many people do not share. Yet that value judgment is incorporated into the WHO rankings, which are then presented as objective measures that everyone should take as given in the healthcare debate. It's a case of assuming what you're trying to prove.
"And if a healthcare system doesn't work for a significant portion of it's population, due in no small measure to the inequities of fairness created by the means with which the system is funded, how is this NOT related to healthcare system performance?"
See above. I totally agree that the quality of care received by the worst off is relevant. But that is NOT what the WHO inequality statistics measure! Inequality simply means a relative difference in outcomes between groups, regardless of the absolute quality of care received by anyone.
"I believe the answer to a single question will resolve this debate for me. Glen, of all the news stations out there in TV land, which one do you believe consistently provides you with the most accurate information?"
I don't see how this is relevant. I guess you expect me to say FOX News or something? The answer is that I don't ever watch the news on TV. I get most of my straight news from the L.A. Times, New York Times, CNN, and MSNBC. For opinions, I usually read blogs.
I'm encouraged to hear that you use many different sources for your news, but the question really was simple, and you haven't answered it. It's certainly your prerogative to not answer, but I'll ask one more time.
Glen, of all the news stations out there in TV land, which one do you believe consistently provides you with the most accurate information?
Gary -- that was my way of saying I really don't have an opinion on the question. If I picked out one TV news station, it would essentially be a random choice. I also still don't see how it's relevant. But if you insist, I guess I'll go with... um... CNN maybe?
My hope was that you would have included CSPAN, NPR, BBC WORLD, SNOPES.COM, URBANLEGENDS.COM, ETC.
CNN always seems to be the one people choose as the balance between MSNBC and FOX. However, in my experience, placing them in that position on the spectrum is illusory.
So, back to healthcare. From a systems analysis perspective, how is it that an evaluation of performance that includes a criterion to measure whether or not the system works well for all parts of the system is an invalid, skewed or biased criterion? Imagine if 75% of a public transportation system performed very well, but it was determined that a criterion used to measure the performance of the 25% of the system that didn't work well was biased toward other transportation systems because that criterion happened to evaluate access and affordability. Would you argue that the rating system was biased toward systems that provided 90% of the population with easily accessible, affordable transportation? What if factors such as ability to walk to access points, factors that couldn't be controlled by the transportation industry, affected the ratings. Does that mean transportation systems that accounted for and accommodated for this factor were unfairly favored in the measurement? How is it that a criterion for performance measurement that evaluates an important part of the whole system, I repeat, the whole system, flawed?
Wow, that car works amazingly well....except that the transmission fails regularly.
But the evaluation process that provided this car with an unfavorable rating is flawed because it used performance measurements that include the ability of a car to get you where you need to be, and those measurements are biased toward cars with good transmissions.
What am I missing here, Glen?
Gary -- I thought I had already made this clear, but perhaps not. The key point is that the WHO inequality measures don't measure what you think they do! They measure differences in quality, rather than the level of quality received by the worst off.
Why does this matter? Because when you punish a system for differentials in performance (rather than poor performance at the bottom), a country's ranking can go down because of an improvement in care.
For example, suppose a country gives everyone mediocre care -- say, a 5 on a 1-10 scale. And then a new policy allows half the population to improve their care to a 7. The other half remains at 5. This is an unambiguous improvement: some people get better off, and nobody gets worse off. But an inequality measure classifies this situation as worse.
In fact, an inequality measure would say the situation is worse even if everybody gets better care, simply because the dispersion increases. Say that everyone starts at 5, half move up to 6, and the other half move up to 7. That's worse according to an inequality measure.
The WHO rankings include more than inequality, which means the changes described above would have ambiguous effects on the country's score. But that's a problem, because the changes described should register as unambiguous improvements. And note that the WHO approach places 62.5% of its weight on inequality measures.
Even if you value equality for equality's sake (meaning you think a country should be punished for some improvements that don't make anyone worse off), surely you must recognize this is a value judgment that many people could reasonably disagree with. Yet the WHO authors present their study as an objective measure of healthcare quality that everyone can agree on. That's just wrong.
And finally, let's take your transportation example. I totally agree that a measure of transportation quality should account for the 25% of the population with poor transportation. But if the WHO approach were used, a country's transportation system would be considered worse simply because some people's transportation is better than others. So, for instance, suppose everyone has a 1-hour commute. And then suppose a new Metro system shortens commutes for half the population to 40 minutes, while the rest continue to have 1-hour commutes. That would be considered a worse transportation system, according to an inequality measure like those included in the WHO healthcare study. Is that not obviously nonsense?
It finally dawned on me why we seem to be missing each other regarding a common understanding about the methodology used for this report. You appear to be trying to discredit or grossly underweight the other measures, so that in isolation, improving overall performance would lower a country's overall rating if it wasn't a uniform improvement. But the problem I see in this reasoning is that inequality is not the only factor being measured here, so that an overall improvement in a country's healthcare system would be captured by other criteria, although the inequality issue would indicate that there is still room for improvement, or that there is cause for concern about the direction the system would be moving in (reflected in a dampening of the effect of the improvements on the overall score).
Glen,
Your 62.5% figure is misleading at best, and disingenuous at worst. The figure I come up with is 37.5%, assuming that in your transportation example, you didn't also create or exacerbate the fair financial contribution measure. But that doesn't tell the whole story, Glen, and you know it (or should). The end result would be based on calculations much more rich and complex than your example would lead your readers to believe.
Being 37th for the largest economical superpower just doesn't make any sense to me, does it to you?
Sandy -- No, it doesn't make much sense to me. And it turns out the WHO rankings are flawed. That was my point.
Post a Comment