The Science of Nitpicking (2)

(Introducing BS-95 and BS-99)

Last Wednesday, in the first post in this series, I introduced a simple test of whether an op-ed article was a complete piece of bullshit. Basically, if , starting at the beginning of the article, you come across five bullshit statements or arguments in a row you can write the article off as complete bullshit with the statistician's much beloved 95% confidence. Seven in a row takes you to 99% confidence.

Over the past few days I've been looking to try this test out on a promising candidate and, via EP in a comment at Larvatus Prodeo, I got onto Pamela Bone's impassioned piece in Monday's Age. The opening paragraph looked promising:

Feminist debate here - what little of it there is - seems to be mainly about whether Big Brother is empowering to or patronising of young women. Many of the same young women would rather die (metaphorically speaking) than own the title of feminist.

What are the claims made in this paragraph?

• There's not a lot of feminist debate here;


• The debate that there is seems to be mainly about Big Brother


• Many young women don't want to be identified as feminists.

That's three candidates for the status of bullshit claim packed into two sentences. If you want to go to the implied subtext - that feminism in Australia has vanished up its own bum and hence, become irrelevant to the next generation of should-be-feminists, you have five claims which might all possibly be bullshit, plus the questionable argumentative technique of letting the sub-text do all the work.

So, you could be 95% confident that this article is complete bullshit if you could demonstrate that:

• There's a lot more feminist debate around than Pamela Bone has noticed;


• Very little of it is in fact about Big Brother


• At least as many young women are willing to identify as feminists as those who don't;


• Feminism hasn't vanished up its own bum;


• Feminism is still relevant, here and now.

Unfortunately, that demands a little more knowledge of the current state of feminism in Australia than I can claim. So, although I'm inclined to count the three explicit claims of the first paragraph as bullshit, it's necessary to read on a little before I can claim the five in a row I need before I'm entitled to dismiss it out of hand and move on to more informative reading.

In other places women are dying, literally, for the feminist cause. Did Western feminists in the 1970s - so silent these days! - think they had it hard? They had to put up with scorn and ridicule from the media and hostility from conservatives who said they were out to destroy the family. But no one, so far as I can remember, was murdered.

A few examples, of many, of what women in some countries are up against: in northern Afghanistan in May, three women workers at a microcredit organisation (which gives loans to women to start up small businesses) were stoned to death by warlords; in India, a woman social worker in Madhya Pradesh state had her hands chopped off by a man furious because she was counselling villagers against child marriage.

Well, that pretty much buggered things up; after a neat little segue in the second paragraph, we get some reported fact, and the simple "five specious claims in a row" test is out the window.

It's for situations such as this that I've devised the BS-95/BS-99 scoring system. It works like this:

• Count all the questionable assertions of fact, possible errors of argument etc in the article;


• Nitpick away at them and count all the errors;


• Consult the BS-95/BS-99 scoring table (Table 1) to determine how confident you can be that the article is complete bullshit.

Table 1: BS-95 and BS-99

No. of Questionable Assertions/Inferences	BS-95	BS-99
5	5	-
6	5	6
7	6	7
8	7	8
9	7	8
10	8	9
11	8	9
12	9	10
13	9	11
14	10	11
15	11	12
16	11	13
17	12	13
18	12	14
19	13	14
20	14	15
25	17	18
30	19	21
35	22	25
40	25	27
45	28	30
50	31	33

It's probably appropriate to slip in a few words here on the BS-95/BS-99 table. It's the product of a couple of hours of futzing around with a spreadsheet of cumulative binomial probabilities. Assuming that any claim I question has at most a 50% probability of being true (the nominally unbiased position), I calculated the probabilities that, in an article with five substantive claims I could refute at least one of them, at least two and so on. BS-95 represents the number of claims you need to refute to be 95% certain that you're reading (or have just read) a piece of complete bullshit. BS-99 represents the number of refuted claims which entitles you to claim 99% certainty. It's in the nature of the test that you can't claim 100% certainty and that's the way it should be. So there.

As you can see from the table, there's no point trying for a BS-95 if you have fewer than 5 questionable claims to deal with and BS-99 doesn't work below 6. Now let's see if we can apply BS-95 to Pamela Bone's article. A promising place to start is here:

A YouGov poll published in London's Daily Telegraph last month found that 32 per cent of British Muslims believed that "Western society is decadent and immoral and that Muslims should seek to bring it to an end". But men were far more likely than women to say this.

Here's the question that was actually put to the poll respondents:

Which of these views comes closest to your own?

• Western society is decadent and immoral,and Muslims should seek to bring it to an end, if necessary by violence 1%


• Western society is decadent and immoral,and Muslims should seek to bring it to an end, but only by NON-VIOLENT means 31%


• Western society may not be perfect,but Muslims should live with it and not seek to bring it to an end 56%


• Don't know 11%

So, Bone's claim that 32% of British Muslims want to bring decadent, immoral Western society to an end appears to be supported by the poll results: as long as you're prepared to lump the 31% who believe that this should happen by non-violent means in with the much smaller number (1% or roughly five and a quarter respondents out of a total of 526 (unweighted) or 527 (weighted). It's possible that this little detail got lost at the subbies' desk but the end result is patently inaccurate and misleading in a rather alarmist way. Are men "far more likely than women" to hold this view? On that score, the YouGov report (PDF) of the poll results is silent. All we have to go on is the Daily Telegraph report:

Among those who hold this view, almost all go on to say that Muslims should only seek to bring about change by non-violent means but one per cent, about 16,000 individuals, declare themselves willing, possibly even eager, to embrace violence.

Yet again, far more men than women and far more young people than their elders evince this kind of hostility towards the world around them.

You might wonder how 1% of a 527 respondent polling sample can have 16,000 members. The answer is simple; in his report of the poll results the Telegraph's Anthony King is freely projecting the percentages to the Muslim population as a whole. It's a pretty sloppy article, and worthy of a BS-95 check in its own right. Back to Bone, in The Age:

As well, the proportion of Muslim men who said they felt no loyalty to Britain (18 per cent) was more than three times higher than the proportion of women who said the same.

Once again, here are the poll questions:

Turning now to wider issues,how loyal do you think most Muslims living in Britain feel towards Britain?

• Very loyal 28%


• Fairly loyal 48%


• Not very loyal 10%


• Not at all loyal 8%


• Don't know 6%

How loyal would you say you personally feel towards Britain?

• Very loyal 46%


• Fairly loyal 33%


• Not very loyal 6%


• Not at all loyal 10%


• Don't know 4%

[emphasis added]

Here's the Telegraph report:

... YouGov asked respondents how loyal they feel towards Britain. As the figures ... show, the great majority say they feel "very loyal" (46 per cent) or "fairly loyal" (33 per cent) but nearly one British Muslim in five, 18 per cent, feels little loyalty towards this country or none at all.

If these findings are accurate, and they probably are, well over 100,000 British Muslims feel no loyalty whatsoever towards this country.

The proportion of men who say they feel no loyalty to Britain is more than three times the proportion of women saying the same.

Whoa, Hold on there! Where did that 18% total of people who feel little or no loyalty come from? In the second question, where respondents are asked about their own loyalties, as distinct from assessing the loyalty of others there are 6% of respondents who are not very loyal and 10% who are not loyal at all; that's a total of 16% over the two categories, not 18%. The 18% figure appears to come from adding up the not very loyal/not at all loyal percentages for the previous question where respondents weren't asked about their own feelings of loyalty, but how loyal they reckoned everyone else was. It's like conducting a poll on stupidity and inferring that most people think they are stupid, based on their responses to a question something like this:

Which of these views comes closet to your own:

• Most people are stupid;


• A lot of people are stupid;


• Some people are stupid;


• Few people are stupid;


• Don't know.

Note too, that in the Telegraph report, there's no mention of how many Muslim men say that they feel no loyalty to Britain. But somehow, when it gets to Bone's turn in this little game of Chinese whispers, we have a definite figure: it's that wandering 18% again.

Now, the results so far:

Claim: 32% of British Muslims want to bring decadent, immoral Western society

Misrepresentation. Firstly, it's 32 percent of the poll respondents and extrapolation to the general population of British Muslims has to be approached with caution. It ignores the distinction between respondents who want it done peacefully and the much smaller number who are prepared to contemplate violent means. Verdict: bullshit. Count it towards the BS-95 score.

Claim: men were far more likely than women to want this.

Questionable, but not refuted on the available information. So it can't be counted towards the BS-95 score.

Claim: 18% of Muslim men said they felt no loyalty to Britain.
On the available information, this is bullshit, so we count it towards the BS-95 score.

Claim: That 18% of men was more than 3 times as high as the percentage of women who felt no loyalty to Britain.

Questionable, but not refuted on the available information. So it can't be counted towards the BS-95 score.

Blimey, this is getting to be a long piece isn't it? And we haven't even got the count of all questionable assertions and arguments in yet. So much for the idea of coming up with a quick, simple way to rate the bullshit content of an op-ed piece. On the other hand, I may have inadvertently stumbled on a scrupulously thorough and fair way to do it. It's bloody tedious, though. Right now, I think I'll take a break and come back to this after I've had time for some sleep.

Update (Thursday, 11 August): Anthony King, referred to above as "the Telegraph's Anthony King is professor of government at Essex University.

After reviewing the spreadsheet of binomial probabilities from which I derived the BS-95 and BS-99 scores, I've decided that Table 1 needs to be revised. A lot.

I'm getting well and truly sick and tired of putting in so much bloody time on what started out as not much more than a half-arsed throwaway idea.