Fast food shop bans - public policy as virtue signalling

Banning new hot food takeaways is a favourite policy of local councils these days. It's driven by a thing they call "wider determinants of health" (tip to aspiring nannying fussbuckets - this phrase should trip from your tongue nearly as often as "cost to the NHS") and, as I was told by Bradford Council's leader in January, the policy is self-evidently "common sense". I'm guess that this is another example of the words 'common sense' simply meaning 'not based in any way on actual evidence' - all I'd done is ask how the council intended to measure the effect of its policy on levels of child obesity (given this was the validation for its introduction). As there is no evidence and no means of measuring the impact of the policy, shouting 'it's common sense' is the only remaining fall back position.

And the evidence? Seems there ain't none:

The evidence that fast food availability causes obesity among children is even weaker. Of the 39 studies that looked specifically at children, only six (15%) found a positive association while twenty-six (67%) found no effect. Seven (18%) produced mixed results. Of the studies that found no association, five (13%) found an inverse relationship between fast food outlets and childhood obesity. Two-thirds of the studies found no evidence for the hypothesis that living near fast food outlets increases the risk of childhood obesity and there are nearly as many studies suggesting that it reduces childhood obesity as there are suggesting the opposite.

And I'm guessing that, since most of these studies merely assess correlation, anyone looking at these findings would have to conclude that the evidence doesn't support the contention that the availability of fast food doesn't relate in any way to levels of obesity in children (or indeed grown ups). All we get is a protected environment for existing fast food businesses and the active prevention of new businesses in this market. In the end we have a smaller economy and just as many fat kids. Evidence-based public policy? What we have is just public policy as virtue signalling.

.....

The "phone a friend" approach to evidence in policy-making

****

When we read about evidence-based policy-making, we're encouraged to believe that the people doing this stuff are carefully sifting the academic literature, testing their ideas against the body of knowledge and exploring new approaches based on this work.

You'd be wrong - they're just phoning a friend:

Unsurprisingly, good relationships and collaborations between researchers and policymakers help research use. This is a common finding—we already knew that policymakers often prefer to get information and advice from friends and colleagues, rather than papers and journals. One consequence is that policymakers may take advice from academics they already know through university or school. In the UK, that tends to mean Oxbridge, LSE, UCL and Kings.

In our everyday lives, we seek advice from friends and colleagues; we don’t trust people who have been wrong before, or who seem to be biased – or with whom we just don’t get along. Exactly the same applies to the process of policymaking and, in our experience, in the process of research collaboration. Negative stereotypes abound on both sides, and our review found that personal experiences, judgments, and values were important factors in whether evidence was used.

Yes folks that thing you thought was evidence-based could well be just as partial as designing policy via an impromptu focus group over lunch in the Dog and Partridge.

....

Why it's good to admit to being wrong every now and then...

****

I used to work with a chap - forget his name, it was along time ago before I came to Bradford - who, when I'd point out a mistake in his stats or similar, would smile and say: "I may occasionally be in error but I'm never wrong." To be fair this was said with a smile while the mistake was corrected - I always liked him for that.

It was quite a while after this that I discovered Rousseau's idea of the General Will and, importantly, that my colleague's quote was more or less a description of how (paradoxically) the General Will is intrinsically right. Now the problem with this collectivist take on will is, as I'm sure folk have already noticed, that we need to have a way of knowing what that General Will is actually saying.

We can point to democracy as a means of determining what the General Will is saying except that, as us Brits have just discovered, democracy doesn't do this - is 52/48 a statement of General Will to Leave the EU or merely the result of a democratic contest? So, if we can't use voting to determine General Will (and the recent referendum reminds us of this fact) how do we decide? Or maybe the General Will - even badged as 'Common Good' or 'Common Purpose' - really doesn't exist?

For Actualists and latterly Fascists, the answer was simple, the General Will was embodied in the leader and his advisors (in themselves the leaders of the state's 'corporations' - army, business, organised labour, academia and so forth). But we're still stuck with the idea that somebody or some thing is the embodiment of that General Will - meaning, of course, that that person or body is intrinsically right. As the wags might say: "I may have my faults but being wrong isn't one of them!"

It seems to me that, in our sophisticated Western liberal democracies, this General Will has been embodied in a technocratic elite - a sort of Platonic administration by expert. Political decisions are sub-contracted to a process overseen by these experts - at the end of the process the politicians (defined here as the people we elect to represent us) do little other than rubber stamp the conclusions of the experts since these are 'scientific' or 'evidence-based'.

If we take the debate about standardised packaging for cigarettes as an example, we can see that the General Will was embodied in a small number of government bodies, academic departments and lobby groups rather than in the mass of response to the Department of Health's consultation on the proposal:

In total, 665,989 campaign responses were received from 24 separate campaigns. Around two-thirds of campaign responses received were from people who are opposed to the introduction of standardised packaging (total of 427,888 responses) and one-third of campaign responses received were from people who are in support (238,101 responses)...

The problem is that the opposition wasn't from those suitably (in the government's view) qualified to comment and they chose to assess only 'detailed' responses which, surprise surprise, split 53/43 in favour of standardised packaging. It doesn't matter whether or not you agree with the proposal for standardised packaging of cigarettes - the process of confirming the proposal post-consultation ignored the majority of responses because they were insufficiently 'detailed'.

The problem we have here is that there's a reluctance to admit that - regardless of how well 'evidenced' a policy might be, sometimes they are simply wrong. Indeed we know there's evidence of this with the standardised packaging policy:

“From a statistical perspective, none of these changes were different from zero. Over the timeframe of the analysis, the data does not demonstrate that there has been a change in smoking prevalence following the introduction of plain packaging.”

They also insert this important warning: “It is not possible to assign a causal relationship between the changes in the noticeabilty of health warnings or smoking prevalence and the introduction of plain packaging, as there have been a number of other confounding factors that have occurred before and during the period of this analysis.”

All this is merely illustrative of the problem with 'evidence' in making public policy. It's not just that we can't prove the counterfactual (what would have happened if we'd not made the policy decision) but also that appraising whether or not something works in social policy is really difficult - because of those confounding factors implicit in the second paragraph of the quotation above. Again this isn't an argument against the organised and systematic appraisal of public policy but rather a call for something different.

Put bluntly, it would be good for those experts to admit they were wrong every now and then rather than perform tortuous contortions aimed at explaining why, despite all the data (confounded or not) they really aren't wrong.

There is nothing weak about admitting a mistake - of fessing up and saying: "folks, I got that wrong!" Yet it seems that too many of us are constitutionally incapable of making that admission. We make predictions - often sweepingly on the basis of 'I'm an expert and I say' rather than actual research or analysis - and when they turn out wrong, the best we can do is sit quietly in the corner hoping no-one calls us out on our error. Some of the 'experts' are more brazen - denying that was what they predicted, shouting about how you've misunderstood what they said, and insisting that someone else is twisting their words to mean something different.

It's because of this - plus the patronising arrogance us clever folk use too much of the time - that polling tells us that much of the population simply doesn't trust what we're saying. Coupled with shouty and aggressive appeals to authority, we shove aside deductive reasoning and intelligent (if naive) questioning in favour of findings from a focus group of experts or determined by our partisan google searches. Treating the mass of the population as semi-sentient may seem right - what, after all, to those sheep know, they have to be led - but that mass of people doesn't forget and, given the chance, will stick two fingers up at you.

Truth is there isn't any General Will - or Common Purpose for that matter - but rather a moving collection of shared interests that never involve every person. Government - however hard you bash the social policy thing - is a pretty poor way of managing these shared interests. And the futher that government is from the things that actually matter to the folk who (in George Bailey's words) do all the living, working and dying round here, the less effective it becomes.

So my friends, make an effort - admit it when you get something wrong, a prediction doesn't turn out quite as you thought or a policy you backed is a failure. It will be a catharsis for you and will get you a damn sight more respect than trying to pretend you weren't wrong. And, of course, feel free to call me out on this too.

....

Why we don't need an Evidence Information Service (but do need better access to evidence)

****

Whenever I think about evidence-based policy making and the use of data by government, my mind turns to an article by Dr Vince-Wayne Mitchell:

Demographic segmentation variables are cheap and easy to measure, while psychographic variables are more expensive and harder to measure, but can provide more insight into consumers’ psychology. Suggests that a prima facie case exists for the suitability of astrology as a segmentation variable with the potential to combine the measurement advantages of demographics with the psychological insights of psychographics and to create segments which are measurable, substantial, exhaustive, stable over time, and relatively accessible. Tests the premise empirically using results from a Government data set, the British General Household Survey. The analyses show that astrology does have a significant, and sometimes predictable, effect on behavior in the leisure, tobacco, and drinks markets. Discusses managerial implications of the results in terms of market segmentation and promotion.

Dr Mitchell is a very highly regarded researcher in consumer behavious and marketing and I've no idea whether he believes in horoscopes or not. But what these results tell us is that we should treat the findings of research studies with a degree of caution. Just because it's badged as science doesn't mean that it's right or that there isn't some other research telling us something entirely different, even opposite. Indeed we know that sometimes supposedly evidence-based policy is anything but:

In 1980, after long consultation with some of America’s most senior nutrition scientists, the US government issued its first Dietary Guidelines. The guidelines shaped the diets of hundreds of millions of people. Doctors base their advice on them, food companies develop products to comply with them. Their influence extends beyond the US. In 1983, the UK government issued advice that closely followed the American example.

The most prominent recommendation of both governments was to cut back on saturated fats and cholesterol (this was the first time that the public had been advised to eat less of something, rather than enough of everything). Consumers dutifully obeyed. We replaced steak and sausages with pasta and rice, butter with margarine and vegetable oils, eggs with muesli, and milk with low-fat milk or orange juice. But instead of becoming healthier, we grew fatter and sicker.

We see this problem repeated time and time again - governments enact legislation that draws on the scientific evidence, on the environment (diesel cars), in health (vaping), in criminal justice (tagging), and farming (agricultural protection) only for the results to be either sub-optimal or else for evidence to arise showing the policy to be plain daft. In part this is because evidence isn't definitive - in the case of fat, there were a different set of scientists who government ignored saying we should look instead at carbohydrates and sugar. And the same is true on vaping - the experts used by the World Health Organisation and European Union to support severe restrictions on vaping and the sale of vaping products are countered by another set of experts with a very different position who argue that vaping should be encouraged by public health not controlled.

The problem here is that these experts move from being producers of evidence to recommenders of policy. After all, when researchers write up their findings for publication they will be expected by the journal (and probably their funders) to comment on the implications of their findings as well as to describe limitations and the areas where further research should focus. The findings, however, don't necessarily tell us what the policy should be - we may discover, for example, links between high levels of sugar consumption and type-2 diabetes but this doesn't mean we have to introduce a soda tax. Other policy solutions - or none -  are available and just as valid.

This brings me to the proposal set out by Chambers et al in The Guardian, for a new Evidence Information Service:

Our idea is to create a hub for connecting a broad network (hive-mind) of UK scientists and researchers with the political community. At present, the knowledge and expertise of more than 150,000 UK scientists and academics is being underutilised. To ensure the smartest possible democracy we need to create the largest active network of engaged scientists and researchers in the world, and then we need to use it.

Superficially this seems a great idea - make it easy for politicians (and a slightly sinister grouping entitled "policy-makers", who presumably aren't necessarily politicians) to connect with the academic and research body of knowledge. The authors go on to observe that there's an imbalance in the information available - ministers have more policy-making resource than a back bench MP or opposition spokesperson. And, as the leader of an opposition group on a large metropolitan council, I can confirm that this is true. The question is whether creating this Evidence Information Service really improves the way in which policies are decided?

I think not. Indeed, I think there are genuine risks in the proposal were it to be implemented.

Firstly, the connection made for the policy-maker isn't a connection to the evidence but is a connection "with specialist experts in that field". Now it may be that these experts simply hand across their evidence to the policy-maker who goes off to craft his policy on painting bus lanes green or whatever. Or it could be that the policy-maker asks the researcher what his or her policy prescription is rather than just for evidence. It's also likely that confirmation bias kicks in - a left wing policy-maker might seek out or prefer evidence from a sympathetic source while ignoring evidence from a source that seems counter to that policy-maker's ideology (this, of course, applies to conservative policy-makers equally).

The second problem is that the proposed Evidence Information System creates valorised and non-valorised evidence. So evidence provided by the new system is 'good' evidence whereas evidence from outside that system is not to be trusted. And because the Evidence Information System is entirely about institutional researchers (academics, in effect) the value of independent research and evidence from outside that sphere is undermined. By way of example, the proposed new system excludes commercially procured research, good quality journalism (including blogs and websites), many think tanks and opinion polling. There is an assumption that only those "150,000 UK scientists and academics" are a valid source of evidence for policy-making.

Next we have the problem of ideology. By this I don't mean socialism vs neoliberalism but rather than policy-decisions are informed by ideological issues. To use vaping as an illustration, we can see two distinct ideological positions within public health and tobacco control. To simplify a little these are essentially Harm Reduction and Gradual Prohibition. This divide applies throughout, to policy-makers and to researchers. No amount of evidence showing how vaping reduces harm will persuade someone ideologically committed to Gradual Prohibition as the purpose of tobacco control. We can pull across this issue into any area of public policy - we encouraged diesel engines because they had lower carbon emissions but now pay a price as those engines have a negative impact on urban air quality and health. In the proposed Evidence Information System there is no safeguard, no means of knowing whether research is ideologically-framed (or more likely, how that research is ideologically-framed). And we can't assume that researchers - especially in social science fields - don't fall foul of confirmation bias or ideological preference.

Lastly we need to get some idea as to what we mean by evidence (knowing from the start that this is a very contested idea). On the one hand we have data which is just that, a great pile of information that, if we're not careful, throws up evidence along the lines of the Mitchell research I opened with. Without some form of analysis, data is pretty useless but which is the better route - giving policy-makers the tools to interrogate the data or getting that interrogation mediated by (possibly biased, perhaps ideological) academic researchers? We then have - especially since it is social science research that will dominate policy-maker enquiry - the issue of 'soft' evidence. Is a qualitative study gathering the views of 50 teachers on in-class discipline more or less valid than a big analysis of Ofsted reports in 1000 primary schools? And what about discussion, op-ed and speculation, where do these fit in - they're important to academia but do they form part of the evidence base?

I applaud the attempt to get better data, information and evidence in front of those who design, decide and implement public policy but don't think that an Evidence Information System as described is the way to proceed. If the issue is asymmetric access to evidence surely the answer is to get more open data and better (easier to use perhaps) tools for using that data. I also worry that the Evidence Information System would create a different asymmetry in access to and use of evidence by valorising only academic evidence. Finally, governments can and do lean too heavily on evidence in decision-making often, as anyone familiar with England's Local Plan process would attest, to the point of sclerosis or even stasis.

Policy-making will always be a balance between having enough evidence and the need to act. It has always been something of a cop out to say, "we need more evidence", when you actually need to do something. And the interest of voters will always trump evidence in the minds of people who are elected by those voters - this is why there's no comprehensive review of London's 'green belt' and why Australia scrapped its carbon tax. Finally, we need to remember that - remember the diesel engines issue - different policy area conflict and the policy decision is not simply a matter of looking at one set of evidence but rather at a series of sets that can point to radically different decisions.

There's a good case for better connections between researchers and the real world - not just politicians but business people, writers, charities and schools - but this proposal doesn't achieve this outcome and would create a service with limited access. Far better would be to negotiate a public library license with publishers making all that evidence - and the search tools needed to use it - available in every community and for every person.

....

Judges back Department of Health - 550 people dumped on the dole

****

In a terrible decision the Appeal Court upheld the Government's proposed ban on cigarette vending machines:

Cigarette vending machines will be removed from pubs, clubs and restaurants after the top judges yesterday upheld a Government ban.

The measure, drawn up by the Department of Health to combat under-age smoking, will come into force in October.

 As a result of this decision an entire industry will close:

Tobacco industry chiefs said it would ‘abruptly and entirely eliminate’ the vending industry, costing at least 550 jobs.

And – even worse – the judges acknowledged that there’s no evidence to justify the decision:

...statistics used by the DoH to justify the ban were ‘little more than guesses’, the judge said

So 550 people go home tonight knowing they’ll have no job in October as a result of a decision based on evidence that is just a guess.

As an aside the self-important Ben Goldacre had this to say about people in public health:

People who work in public health bend over backwards to disseminate evidence-based information to the public

This single decision shows just how wrong Ben is (again).

....

Alcohol Concern Cymru - the most ignorant of nannying fussbuckets

****

Although BBC Wales does its best to make Alcohol Concern Cymru not seem like a bunch of idiots even they can't quite manage it with this story:

An alcohol charity claims there a "silent epidemic" of heavy drinking among elderly people in Wales

But Alcohol Concern Cymru's (AAC) report quotes statistics that have been challenged by drinks industry body The Portman Group. The group argued overall trends showed "a positive and continuing decline in the rates of excessive drinking".

Andrew Misell of AAC said there was anecdotal evidence from elderly support workers to back their claim.

All your typical row until you get to look at why AAC issued their scary press release:

The paper compares figures from Welsh Government-commissioned Health Surveys of 2003/4 and 2009 as "evidence that the proportion of older people drinking more than the recommended amount is rising".


AAC said the number of over 65s who said they had drunk more than the recommended maximum in the previous week rose from 22% (men) and 7% (women) in 2003/4 to 34% (men) and 17% (women) in 2009.

See scary - a huge increase in wrinkly boozing is evidenced from the surveys. Except that - as the BBC eventually explain:
 

However, BBC Wales understands that as a result of changes in methodology adopted by the compilers of the Welsh Health Survey in 2006 the two sets of statistics are not comparable.

Ah, there you are you see! No increase in drinking. However AAC keep wriggling:

When challenged on the paper's use of statistics Mr Misell said: "Those statistics were taken as an illustration. It's certainly the case that more work needs to be done in terms of finding out what exactly is the pattern of drinking among older people.


"The point of the paper is that it's a hidden problem. If you talk to people working with older people they will say there's quite a lot of anecdotal evidence to support the fact that alcohol is a problem."

The problem is so hidden that the statistics can't pick it up and are showing old folk drinking less rather than more. Something must be done!

....

Multipliers, markets and why the New Economics Foundation make me cross

The frighteningly bright Jon Beech has challenged my frothing rant about the New Economics Foundation and indeed my contention that much of their economics work is “evidence-free”. In doing this Jon provides what I find a very helpful positioning of NEF:

NEF's primary agenda is to counter the consumerist agenda that's piped out through radio TV etc, and to help people to reflect on what their relative wealth actually buys them: neuroses and over-leveraged anxiety. They question the economies of scale which result in uninterested interest.

Now were NEF simply engaged on a moral crusade then that would be a fair position but they are clearly positioned as researchers engaged in a scientific (in so far as we can call economics science) process. Which – as I said in my rant - requires evidence. And science, being the troublemaker that it is, evidence means empirical study. It means the raising and subsequent testing of hypotheses. And this NEF does not appear to do. It isn’t sufficient (and this applies as much to those who prefer a libertarian analysis to a statist analysis) simply to select the statistics you like, find a few quotes and cite your own earlier work. Unless of course you're a simple blogger like me!

Let us take just one of NEF’s “products” – LM3:

LM3 was developed by NEF (the New Economics Foundation) as a simple and understandable way of measuring local economic impact. It is designed to help people to think about local money flows and how their organisation can practically improve its local economic impact, as well as influence the public sector to consider the impact of its procurement decisions. It was designed to be quick and relatively easy, and to highlight where an organisation can improve its impact.

This is presented as an econometric tool (indeed I used it in my MSc dissertation) for assessing local economic impact but it is very difficult to track back to the theory underlying the model or indeed to find whether NEF actually undertook robust, peer-reviewed tests of its validity before launching it (with the accompanying consultancy offer) onto a credulous public. And where it has been tested this is what we find:

Notwithstanding this conclusion, difficulties in data collection combined with inaccuracies inherent to the LM3 process created a large margin of error in the findings.

And this is before we have considered the theoretical critique of the Keynesian multiplier. So apologies for getting all academic for a minute:

The multiplier is a central concept in economics and especially regional studies where it is widely used to assess the long term impact on employment and output from different forms of investment. As such it represents a significant part of the Keynesian aggregate demand model of the economy and can be described as the impact of the marginal propensity to consume (mpc) on a given investment or expenditure where the higher the level of mpc the bigger the multiplier (Heertje & Robinson, 1979).

The problem is that measures of the multiplier do not take account of the input source (i.e. where the money is earned) and that it is very difficult to define what we mean by “local” or “local economy”. This is significant since by applying the LM3 model to public expenditure we do not take account of the opportunity costs associated with that public spending (i.e. what the taxpayer would have spent it on had it not gone in tax). With the result that:

It is quite misleading to leave public policymakers with the notion that their spending is not at the expense of the private sector because it may be autonomous or have multiplier effects

None of this intended to say that Keynes was wrong about the multiplier (although there is plenty of doubt about his arguments in this respect) but to point out that NEF appear not to have been rigorous in their appraisal of LM3. Or rather that I cannot find any peer-reviewed, critical appraisal of their model – there is nothing evident from their website and my (limited) searches find very little published work applying the LM3 model in an academic research setting. Even in NEF’s own published work in my area of specialist interest – street markets – there is only an assessment of inputs (how much consumers spend) and no evidence of how that money then flows (e.g. derived from sales figures, margins and staffing costs of market traders). I happen to agree with NEF’s conclusions about the negative job impact of supermarkets but then so does almost every piece of independent economic research which doesn’t excuse NEF from skimping on the evidence.

This is just one of NEF’s products – I cannot comment on other products or reports – but to me it suggests that Jon is right in suggesting that this think tank sees its role as challenging the moral basis for consumerism and perpetual growth rather than providing a substantial and robust evidence base for us to understand how an alternative might work.

...

Policy-based evidence making (or How Government is killing science with help from academics)

If I hear the phrase “evidence-based policy making” drop from the lips of a Labour minister or one of their bureaucratic lickspittles I shall scream. Labour came to power puffing this idea – and then ignored any evidence that got in the way of its mission.

But it’s worse – Labour’s academic apologists use bad research to support the Government’s case on education, on crime and on terrorism. Instead of a proper scientific enquiry these “researchers” use qualitative methods – mostly gathering the opinions of a few experts and presenting that as research – to arrive at “evidence”.

I completed a Master of Science degree that – had I so chosen – could have been gained without any quantitative analysis. (I hasten to add that I chose to do proper research).

This is not evidence it is opinion

The social sciences are dominated by innumerate people who even dream up cod intellectual justifications of their make-believe approach to enquiry. Nothing wrong with opinion guys - but it is not evidence.

So I am grateful for the Heresiarch coining the term:

Policy-based evidence making

It sums up everything wrong with the way we conduct government, train our future governors and sideline scientific enquiry.