Because Enough is Too Much

[Update - please see the next post for an updated opinion on this.]

My major waste of time for today (in addition to reading two excellent articles on the North Korea situation - neither of which I agree with, by the way, but both of which are useful in their way - especially the first one (by Robert Kaplan) linked) has been this damned Johns Hopkins report on the number of "excess deaths" in Iraq.

I guess most everyone will know about this. The report, which will be published in The Lancet (yes, the same Lancet that brought you the fake cancer study and the fraudulent autistic enterocolitis article), follows up on an earlier 2004 report using the same methods and reaching similar conclusions.

And what are the conclusions?

Crudely, that the number of "excess deaths" as a result of the Iraq War has been hugely underestimated - it is actually over 10 times as high as average findings by other studies and roughly 20 times as high as the 30,000 estimate President Bush gave late last year. So what gives?

Well, first of all it's worth explaining what is meant by an "excess death." These are deaths that, statistically speaking, would not have occurred without the invasion. In other words, this figure is meant to include not only people killed in fighting, but also those who died as a result of deteriorating public order or medical services, or any number of other "side-effects" of the invasion. Clearly, this is a number that matters: wars do not take their toll exclusively on the battlefield. Any reasonable post hoc assessment of whether the Iraq War was a good idea will need to consider whether Iraq is overall better off than before the invasion - including civilian life. So the methodology of the survey was to take a random sample of roughly 1,000 households from 47 regional "clusters" around the country. These "clusters" were chosen to reflect population demographics. For example, the largest number of such clusters (12) in any region was in Baghdad - and roughly 20% of the population of Iraq does indeed live in Baghdad. Two states were not represented (no explanation was given for this - one of them is underpopulated so no cause for concern there, but the other is one of the Kurdish states. More on that later). Households were chosen randomly within the clusters and asked how many people had died in each of the following years: 2002-3, 2003-4, 2004-5, and 2005-6 (actually, the first "year" covers 14 months - it was meant to establish a prewar baseline). The survey reports that death certificates were shown in 92% of the cases, so for the most part these are also documented deaths. From this, they established that the prewar deathrate had been 5.5/1000 (which is very low compared to the world average, but that seems to be normal for Arabia and is only slightly lower than what the CIA Factbook reported for the same period). Then they found deathrates for each of the other years, averaged these, and extrapolated to the whole population. The overall conclusion was that some 655,000 "excess deaths" have occurred in Iraq since the invasion.

This figure is alarming (to say the least). To put it in perspective - it represents roughly 2.5% of Iraq's overall population. As war casualties go, that's quite high. Leaving aside the problematic case of the Soviet Union (because the communists might have shovelled off some of their own attrocities on the Nazi invasion - official figures show that a staggering 20million Russians died in the war - maybe as much as 1/5th of the population), the country that WWII arguably hit the hardest was Japan. Japan's military was literally wiped out, its largest cities were subjected to intense firebombing, and, of course, we all know about the two nuclear bombs that led to its surrender. All total, Japan lost about 3.6% of its population in WWII. So to believe this report, you have to be in a position to believe that Iraq has, in 3 years, lost a roughly comparable proportion of its population (remember, these are "excess" deaths - so the "baseline" deaths get added on - and at 0.6% we're talking about 3.1-3.2% of Iraq's population gone) to that lost by Japan in 4 years of total war that ended in carpet bombing and nuclear blasts. Needless to say, the survey is a bit hard to swallow.

And yet, surely there's nothing wrong with the methdology? After all, this is the way we generally do opinion polls. I suppose it could be argued that for truly serious matters like death tolls more precise methods should be employed - but even so, even if we increased the sample size or some such, shouldn't the Johns Hopkins "loose" report at least be in the ballpark of the real figure? Indeed, the authors argue that their report is likely to be more accurate than those released so far because the others (and here they're talking about the UN, the Iraqi Ministry of Health, the US Department of Defense, and Iraq Body Count, and various NGOs) all use "passive" data. That is, they all simply comb through government and hospital files and get their numbers that way. Naturally, there's an argument to be made that "passive" data underestimates the real bodycount because not all deaths are reported.

So where's the catch? Even if the official count misses some unreported deaths, how is it that 510,000 corpses just went missing? Surely if the count were off by that much people would notice them lying around here and there? Or something?

Well, I haven't been able to figure it out. But neither can I simply accept that Iraq has already lost over 3% of its population since 2003. That just doesn't seem plausible. So to soothe my cognitive dissonance, I've come up with the following tentative list of possible partial explanations:

• The baseline is off somehow - this won't account for much of the discrepancy (since the CIA found a 6% deathrate for 2002 - not that different from what the survey finds), but it's worth noting that the baseline in the survey only covers one year. That seems fishy to me. After all, it's not as though it's any more difficult to ask people for deaths over the 10 years leading up to the war than to ask the same question covering only one year. Naturally getting results for 10 years invovles more paper work (assuming they bothered to verify these death certificates, I mean), but it doesn't involve any extra number crunching, really. And it would have established a more reliable baseline. For all we know, 2002 might have been unusually quiet (and probably was, actually, considering that under Saddam Iraq was otherwise frequently raiding Kurdish villages or violently putting down various protests, etc.)


• The sampling is off somehow - this seems the most likely explanation. The idea of doing random "clusters" would be a contributing factor. What this presumably means is that specific neighborhoods were chosen, and houses were randomly selected in these neighborhoods. While that may make the interviewer's job easier, it introduces confounding variables to the survey. Deaths as a result of fighting (which are reported to be the main source of "excess deaths" in the report) are likely to "cluster," after all. If one household in a neighborhood is affected, others will be as well. We have to be careful with this charge, however, since it amounts to an accusation of fixing the numbers for political reasons. To get a number like 650,000 "excess deaths," the people involved in the study would have had to have chosen at least some of their "clusters" pretty carefully. I will cautiously say, however, that we have suggestive evidence that they did so. As I mentioned earlier, one of the provinces left out is in the Kurdish regions, which are considerably more stable than the rest of the country. Perhaps this was because they couldn't find any clusters with suitable death rates in this province and didn't want to throw off their average. Further suggestive evidence would be that each of the surveys just so happens to have been released in the month before a US election. Even a cursory read of the survey reveals that the authors are very much against the war ("editorializing" comments and statistics are legion - especially in the random facts listed near the end). This in itself doesn't mean that their information is false, of course. Scientific evidence should not be judged on the basis of its invetigators' political bent. However, I think it is fair to note any potential bias in assessments of reports we have independent reason to doubt.


• The Iraqi Ministry of Health is engaged in a coverup - of course we can't rule this out. The new Iraqi government definitely has legitimacy issues in the eyes of sections its electorate. It would not want to claim as a basis of power a war that resulted in 650,000 "excess deaths," nor would it particularly care to admit that the security situation was this bad. Again, however, I find it implausible that a coverup exists on this scale. It's hard to hide 500,000 extra bodies over the official numbers - especially with the UN and various hostile NGOs running around and poking their noses into your business. So a coverup on this scale doesn't seem plausible. But that's not the same thing as saying it isn't happening.

On the whole, I'm not convinced. First of all, the margin of error in the survey is roughly 200,000 either way, a full third of the total figure they settled on. I'm no big stats buff, but I've never seen a credible survey with a 30% margin of error. Also, if death certificates were really provided in 92% of the cases they recorded, then these deaths must have been registered with someone. So the survey's claim that the huge discrepancy between their findings and the findings of other surveys is due to unreported deaths is just ludicrous. The local hospitals and the Ministry of Health simply can't have lost 80-90% of their records, even if they were engaged in a coverup of some kind!!!

So I'm going to chalk this survey up to political opportunism for the time being and wait to form a full opinion until the dust settles a bit. I find it unlikely that anyone will say anything to make me take it seriously, but I'm not going to rule it out. In principle, as I said, there's nothing wrong with its methodology - and probably at least the main point of the survey is correct: that the body count in Iraq has been somewhat underestimated and a more active survey of some kind should be done. Given the political volatility of the situation, there is reason to believe that the authorities don't want such a survey done, won't encourage any to be done for that reason, and that we'll be in the dark about the true numbers for the better part of the next decade.

A couple of interesting facts in closing. The latest CIA Factbook page on Iraq shows a deathrate of 5.3%, which I also find highly dubious. We'd be asked to believe that the death rate in Iraq is now lower than it was in 2002. Unlikely. Assuming you want to believe the Factbook at all, have a look at the infant mortality rate for something interesting. It was roughly 33/1000 in 2002 and has shot up to 48/1000 for 2006. So I would cautiously say that there is evidence floating around that the heathcare system in Iraq is seriously compromised by the invasion.

I remain a supporter of the war overall, though I'm beginning to form doubts on several fronts. Finding out how accurate the Johns Hopkins survey is means a lot to me because if their numbers are right I may begin to seriously rethink my opinion.