The desire to anticipate events is a constant in human history. The world is a big, scary, weird place and it is reassuring to know what is around the corner. Fortunes have been made by apparently superior predictiveness, and fortunes have been lost on models that turned out not to anticipate risks as well as hoped.
The same is true of politics. We all want to know what will happen at the next election. Indeed, talking to politicians and advisers, I’m always astonished by how much mental energy goes into trying to ‘figure’ what will happen, rather than trying to shape what will happen. I’m sure there is a sociological reason for this (We want to position ourselves to take advantage of the most likely scenarios? We want to raise our status with others by appearing to have foresight? We are trying to repress our depressing powerlessness over future events by focusing on their inevitability?).
This leads to very irritable debates among people whose expectations and predictions vary. There’s a lot of status at stake in being ‘right’.
Not that I’m any different. I spend a ludicrous amount of time trying to work out what ‘will’ happen, and I always come up with the same answer.
The range of possible results at the next election is very wide. Even the totally unexpected shock isn’t that improbable. In the last eight elections I can think of three unexpected events that shifted the polls dramatically in the last year of a parliament – The winter of discontent, victory in the Falklands war and the fuel protests. That’s three ‘black swans’ in eight elections. Two of those undoubtedly affected the result of the Election. In Politics, Black Swans are as common as ducks.
Faced by the huge amount of noise in the data, we can do two things to satisfy our urge to predict. The first is to average out previous polling to election outcomes, and use this to construct some sort of model for the next election. This is the smart, data based, thing to do. This is the basis for the increasingly sophisticated modelling produced by psephologists like Stephen Fisher, Rob Ford and Will Jennings.
Now the trend on which these prediction are built is that, in general,the past usually suggests that oppositions lose some support in the run up to an election, and Government’s usually recover some support. (There’s a lot more to the different models than that, but bear with me, psephologists). So it’s reasonable to assume that something similar will happen again. Reasonable, but far from certain. There are examples of this trend not happening at all: 1979 being the most obvious, but you can also make a case for 2001.
So the margin by which a perfectly sound prediction could be wrong is very large. Steve Fisher’s latest suggests that the Labour share of the vote next year will be within 26 and 38 with a 95% confidence. Tell that to an MP, and they will not be greatly impressed by your knowledge of the future and the human heart. “I predict that Labour will almost certainly get one of the election results we’ve had in the last 30 years….” does not tend to impress. This also means that even if the trend suggests one thing, there is no inevitability about that trend. There’s no de facto reason why the next election can’t be like 1979. Nor 1983.
This is where, our second option arrives. We can look at the past, and apply our judgement, and with this, our prejudices.
It would be perfectly reasonable to make the argument that the past election 2015 most resembles is 1987. A first term opposition leader whose party suffered a significant defeat at the last election, facing a fairly established but divisive Prime Minister with the economy finally recovering from a deep, painful recession. A year before the election, the opposition leads by five points or so, but goes on to lose by a large margin.
Yet it would also be entirely reasonable to argue that 1970 is a better comparison. An opposition leader widely dismissed as inadequate, and with a poll lead that is large but not decisive, facing a Prime Minister who has strong personal ratings but who leads a divided government that has not delivered significant personal incomes growth. Result: Decent opposition majority, confounding pundits and expectations.
Does either of these narratives feel more ‘right’ to you? Do you see clear flaws in one example, but not the other?
If so, I expect that is a reflection of your past experiences, or your personal feelings about the current political situation, and the current parties and the strengths of their leaders. We’re imposing our own judgements on both the data and the few examples of the past that we have to hand.1
Personally, I find myself always returning to the averages, but I can’t deny that I find myself drawn to the more pessimistic of the available past models for the Labour party. This is likely because my first experience of politics was the 1992 General Election, which has predisposed me to a certain political caution. It’s also partly because I think it’s better to assume no election is won until it is.
However, that is my prejudice, not a data driven analysis. We can choose to live with the great uncertainty the data really gives us, while also highlighting the past examples that most fits our own analysis. What we can’t do is be certain of the future.
If we want to be constructive rather than mystic, however, perhaps the best thing we can do is understand why the more pessimistic examples turned out the way they did, and invest our efforts in avoiding those mistakes.
- An example: I was once asked to discuss AV with a group of senior progressively minded Labour people. There was much talk about the divide of the left in the Eighties handing Thatcher power. After a while, I could bear it no more and piped up to point out that in fact the data showed that AV would have given the Tories an even bigger majority in the 83 & 87 elections, as SDP/Alliance voters would have given Thatcher more of their second preferences. There was a brief pause, then the conversation continued as if I hadn’t spoken. It was really annoying. I always think of how much it irritated me when women tell me that they regularly get treated like this in rooms full of men. [↩]