***
Many experts on polling question the effectiveness of the “what if” question:
“If politician X was leader of Party Y why would you be more or less likely to vote for them”
There are too many chances to be wrong for this to make much sense as a question – Politician X isn’t leader, you many already have a 100% likelihood of voting for Party Y, the election is in the future. In essence this type of question expects people to make a prediction of their own behaviour following a given change – a we are not especially good predictors of our own future behaviour under such circumstances. Indeed, these questions are really just a variation on “who would you prefer to be leader” or “which policy would you prefer”.
Which brings us to the way in which these questions are used to justify policy decisions. By way of benign example, let’s talk about libraries. Most of the population are not members of a public library and have at best a very occasional engagement with the library service. This isn’t to say we don’t want libraries – we’ll fight hard to protect these vital local services even though they are not services we (or anyone we know) makes use of.
Faced with this problem – declining library membership – the local council undertakes a review part of which involves surveying the public. But first the council thinks of lots of exciting things to do with or put it its libraries – computers, coffee machines, self-service issuing of books, book clubs, kids parties - you name it. And in the survey the council asks whether these things will make non-users more or less likely to use the library.
Now let’s assume that library non-users (or a majority of them) say that some or all of the changes will make them more likely to use the library. Seizing on this, the council rushes through the changes – despite that fact that current library users have said they don’t want these changes. Ah, says the Council, the policy will attract loads of new users to the library and we’ll be in clover!
So what happens? Well the current users of the library don’t like the changes (noisy kids, impersonal service, coffee stains all over the table and so on) and some decide to get their reading material from Oxfam or Waterstones. And those non-users who said they would be “more likely” to use the library? Not a sign of them. Result of the policy changes? Probably further library closures, reductions in the book fund and another review.
This is precisely what happened with banning smoking in pubs – non-users said they didn’t like pubs because of the smoke. And that they would be “more likely” to use the pub is smoking was banned. Existing regular pub users – overwhelmingly – opposed the ban preferring good extraction, separate smoking rooms and other measures. But no – banning smoking would make pubs more popular. Those non-users said so, didn’t they?
In truth those non-users said nothing of the sort – they didn’t go to pubs because they didn’t like pubs. And the smoking was just one factor – mostly they didn’t see the pleasure in sitting a drinking away from their comfortable homes. Nothing to do with smoking, nothing at all. And, following the ban, these non-pubgoers have not started going to the pub while at the same time loads of previous pubgoers now stop at home where they can smoke. With fewer regulars, the remaining hardy folk began to drift away and the local was left with three customers at 10pm on a Friday night.
The result? The pub closes. The community loses a local facility. Ordinary, harmless folk have nowhere to go for a pint and a fag – other than at home of course. Some of that much vaunted social capital is lost. The football team folds – it was a PUB team after all. The gardening club slowly declines. Other groups become ever more cliquey. Why? Because a ban brought in to satisfy folk who never used pubs resulted in those pubs closing.
....
Showing posts with label qualitative research. Show all posts
Showing posts with label qualitative research. Show all posts
Wednesday, 7 April 2010
Wednesday, 6 January 2010
Soft outcomes need hard measures to be acceptable in policy-making
***
Julian Dobson writes Living with Rats – you should all read it. Not only is it well-written, insightful and thoughtful but he talks about things that matter to real people outside the bureaucratic circle – why some places are crap, why regeneration is about people rather than buildings and why too often local initiative is stifled by the demands of national government, big business or great institutions. He’s also a Hammer.
However, sometimes Julian lets his writerliness get ahead of things and this post, “Why is it so hard to be soft?”, is one example. What follows is a gentle fisk – responding to a series of questions posed by Julian. At the heart of this is the common fallacy that economics is about money (it isn’t) and that opinion or feeling is impossible to quantify (let me introduce the Opinion Poll). Julian’s words are in italics:
It's hard to measure the quality of a person's life. But you'd have to be a bit soft to think you could do it entirely by calculating their income.
Well yes of course which is why we don’t measure quality of life using income measures – we use other objective measures: what we consume, our health, our conditions of living, our ability to act freely. All of these are strong quantitative measures to which can be added the tremendous contribution of market research towards the quantitative measurement of opinion.
It's hard to assess the success of a policy against a range of desirables, such as economic output, sustainability and equality. But you'd have to be a bit soft to think the one that's easiest to measure numerically is by definition a good proxy for all the others.
No it’s not – so long as we set clear quantifiable outcomes (and such exist for economic output, sustainability and equality) then measurement is merely a matter of collecting and assessing the necessary data. If a policy does not have, say, equality objectives then to measure it against such “objectives” is mistaken and bad research – a classic flaw in policy evaluation is that the scope of the evaluation is set in hindsight without consideration of the original intentions behind the policy-making.
It's hard to measure how worthwhile and rewarding a person's job is, and harder still to measure it across a workforce. But you'd have to be a bit soft to think that the net number of jobs was more important than the sense of self-worth, motivation and aspiration people feel in doing them.
It seems obvious that, for the quarter of the working age population without a job, the “net number” (not sure what it’s netted from) of jobs is of crucial importance. And the knowledge that, if they had a job, it would be worthwhile and rewarding ain’t much comfort!
It's hard to calculate the value of a cultural asset like a theatre or art gallery or public artwork or live music venue. But you'd have to be a bit soft to think you could do it just by bums on seats or spending on merchandise and tickets.
There are real measures of cultural value beyond ticket sales, market price or merchandise. Crucially, approaches seek to capture either the opportunity cost (life without the cultural asset) or to price culture. Simply saying “oo isn’t it lovely” doesn’t wash!
It's hard to calculate how people feel about where they live and how involved they are in their communities. But you'd have to be a bit soft to imagine the price of houses or the proportion of people voting will tell you the whole story
First, house prices are a nice proxy for what people generally think of a place – that thinking may not reflect everything but it does reveal where people want to live! We can measure involvement (and be “shocked” at how low it is), we can use self-referenced definitions of community and we can use good old opinion research to ascertain what they think about where they live.
...
The point of all this isn’t to dismiss qualitative “research” – if you want to frame a question, get an instant response to an idea, provide some context to research or guide our tactical decision-making. But qualitative research must not lull us into rejecting scientific method – the testing of hypotheses with repeatable experiments with results that are quantitatively comparable.
...
Julian Dobson writes Living with Rats – you should all read it. Not only is it well-written, insightful and thoughtful but he talks about things that matter to real people outside the bureaucratic circle – why some places are crap, why regeneration is about people rather than buildings and why too often local initiative is stifled by the demands of national government, big business or great institutions. He’s also a Hammer.
However, sometimes Julian lets his writerliness get ahead of things and this post, “Why is it so hard to be soft?”, is one example. What follows is a gentle fisk – responding to a series of questions posed by Julian. At the heart of this is the common fallacy that economics is about money (it isn’t) and that opinion or feeling is impossible to quantify (let me introduce the Opinion Poll). Julian’s words are in italics:
It's hard to measure the quality of a person's life. But you'd have to be a bit soft to think you could do it entirely by calculating their income.
Well yes of course which is why we don’t measure quality of life using income measures – we use other objective measures: what we consume, our health, our conditions of living, our ability to act freely. All of these are strong quantitative measures to which can be added the tremendous contribution of market research towards the quantitative measurement of opinion.
It's hard to assess the success of a policy against a range of desirables, such as economic output, sustainability and equality. But you'd have to be a bit soft to think the one that's easiest to measure numerically is by definition a good proxy for all the others.
No it’s not – so long as we set clear quantifiable outcomes (and such exist for economic output, sustainability and equality) then measurement is merely a matter of collecting and assessing the necessary data. If a policy does not have, say, equality objectives then to measure it against such “objectives” is mistaken and bad research – a classic flaw in policy evaluation is that the scope of the evaluation is set in hindsight without consideration of the original intentions behind the policy-making.
It's hard to measure how worthwhile and rewarding a person's job is, and harder still to measure it across a workforce. But you'd have to be a bit soft to think that the net number of jobs was more important than the sense of self-worth, motivation and aspiration people feel in doing them.
It seems obvious that, for the quarter of the working age population without a job, the “net number” (not sure what it’s netted from) of jobs is of crucial importance. And the knowledge that, if they had a job, it would be worthwhile and rewarding ain’t much comfort!
It's hard to calculate the value of a cultural asset like a theatre or art gallery or public artwork or live music venue. But you'd have to be a bit soft to think you could do it just by bums on seats or spending on merchandise and tickets.
There are real measures of cultural value beyond ticket sales, market price or merchandise. Crucially, approaches seek to capture either the opportunity cost (life without the cultural asset) or to price culture. Simply saying “oo isn’t it lovely” doesn’t wash!
It's hard to calculate how people feel about where they live and how involved they are in their communities. But you'd have to be a bit soft to imagine the price of houses or the proportion of people voting will tell you the whole story
First, house prices are a nice proxy for what people generally think of a place – that thinking may not reflect everything but it does reveal where people want to live! We can measure involvement (and be “shocked” at how low it is), we can use self-referenced definitions of community and we can use good old opinion research to ascertain what they think about where they live.
...
The point of all this isn’t to dismiss qualitative “research” – if you want to frame a question, get an instant response to an idea, provide some context to research or guide our tactical decision-making. But qualitative research must not lull us into rejecting scientific method – the testing of hypotheses with repeatable experiments with results that are quantitatively comparable.
...
Sunday, 1 November 2009
Policy-based evidence making (or How Government is killing science with help from academics)
If I hear the phrase “evidence-based policy making” drop from the lips of a Labour minister or one of their bureaucratic lickspittles I shall scream. Labour came to power puffing this idea – and then ignored any evidence that got in the way of its mission.
But it’s worse – Labour’s academic apologists use bad research to support the Government’s case on education, on crime and on terrorism. Instead of a proper scientific enquiry these “researchers” use qualitative methods – mostly gathering the opinions of a few experts and presenting that as research – to arrive at “evidence”.
I completed a Master of Science degree that – had I so chosen – could have been gained without any quantitative analysis. (I hasten to add that I chose to do proper research).
This is not evidence it is opinion
The social sciences are dominated by innumerate people who even dream up cod intellectual justifications of their make-believe approach to enquiry. Nothing wrong with opinion guys - but it is not evidence.
So I am grateful for the Heresiarch coining the term:
Policy-based evidence making
It sums up everything wrong with the way we conduct government, train our future governors and sideline scientific enquiry.
But it’s worse – Labour’s academic apologists use bad research to support the Government’s case on education, on crime and on terrorism. Instead of a proper scientific enquiry these “researchers” use qualitative methods – mostly gathering the opinions of a few experts and presenting that as research – to arrive at “evidence”.
I completed a Master of Science degree that – had I so chosen – could have been gained without any quantitative analysis. (I hasten to add that I chose to do proper research).
This is not evidence it is opinion
The social sciences are dominated by innumerate people who even dream up cod intellectual justifications of their make-believe approach to enquiry. Nothing wrong with opinion guys - but it is not evidence.
So I am grateful for the Heresiarch coining the term:
Policy-based evidence making
It sums up everything wrong with the way we conduct government, train our future governors and sideline scientific enquiry.
Subscribe to:
Posts (Atom)