Tuesday, 20 October 2015

Academic unanimity in favour of EU membership is to be expected, but derogatory views about public attitudes are less acceptable


Nigel Farage has called a campaign for the impartiality of British academia in the forthcoming EU Membership referendum.  His argument is that academics are so widely and heavily funded by the EU they cannot be properly free-thinking.  On this he is surely right.  In July, Universities for Europe and Universities UK (which represent 133 higher education institutions) came out with unequivocal backing for the campaign for Britain to remain in the EU.  How can this be reconciled with the idea that universities should be bastions of open-mindedness and free speech?  How can any academic speak out without fear when the 'company policy' is against him? 

I've spent four years studying public support for the EU (or lack of it) at a British University.  I know about the pressures to conform to the orthodox wisdoms of academics. (My two excellent PhD supervisors are most certainly blameless here, as is Matt Goodwin who I worked for on his latest Ukip book - published next month).  The tacit rule is don't question the EU funding arrangements for research.  And never criticise the holy grail: the €14.7 billion Erasmus+ programme.  You will have an irate Erasmus exchange student sitting right next to you.  Sometimes I have felt like a dangerous heretic.  Not once have I heard a qualified academic speak out in favour of leaving the European Union at a public event, such as an academic conference.  

A survey before the last election of 1,019 academics for the Times Higher Education Supplement found only a lonely 4 who supported the Eurosceptic Ukip. Reason enough you might say, to return to the days post 1603 of the 'fancy franchises' when Oxford and Cambridge Univerisites had two MPs each of their own. My arriviste institution, the Univeristy of London, was granted one through the Second Reform Act of 1867.   Nowadays however, it's generally accepted that the more cerebral do not necessarily make better voters.  They like the rest of us have their own interests.  And these, as much as their unique insights explain how academia has become so embarrassingly detached from the collective wisdom of the average voter.

Armed only with one vote like the rest of the public, academics can still exert considerable influence on our politics through their research.  As I show below, it's one central idea from academia that frames how the media report the public debate on the EU, and unsurprisingly, it's a favourable idea for the 'remain in the EU' lobby.

The idea actually stems less from decent research than a prejudice academics hold about the wisdom of public opinion on EU matters.  My research work has challenged the central 'EU friendly' rule that dominates European Studies departments and colours voter behaviour research on the EU issue.  It's this: the public cannot be trusted to make sensible decisions on EU integration.  They don't understand its complexities.  They are uninterested in its remoteness. And, most seriously, they are led by their emotions (when not by the political class).  The intellectual justification is in part poor empirical research and part suspect psychological theory.  More specifically in Britain, the public, so the argument goes, see immigration and economics as separate concerns within the overall EU debate, but too often their rational economic judgements get infected with more visceral (non-rational) feelings of national identity that drive immigration attitudes. To use the academic vernacular, EU support is structured two-dimensionally, and these dimensions are 'cross-cutting' or 'orthogonal'.  Simply put: the two types of support which make up the overall 'remain or leave' judgement in the public mind are separate but conflicting.  Rational thoughts can't be squared with visceral feelings.  How could voters make head or tail of it? 

The derogatory stance of academia towards the voting public on the EU is an all important subtext to the forthcoming referendum.  Whilst the 'remain' lobby would not be so unwise as to say explicitly the public are confused and can't be trusted, the Europhile camp believe it.  They will use the 'two dimensions' idea in a more subtle way to frame the debate.  They want arguments about immigration and economics detached so they can play on economic fears, whilst branding the immigration side of the debate an emotional side-show.  So far they are succeeding.  In the public debate, the idea of independent economic and immigration aspects of the question are independent is all pervasive.  When you hear the ubiquitous words 'soft' (economic driven) and 'hard' (identity driven) Euroscepticism you are hearing a narrative of divided, incompatible attitudes.  Even opinion pollsters measure interest in immigration as separate from EU membership (and then conclude there's not much interest in the EU question).  But the public don't see it this way.  They have no difficulty reconciling their economic and immigration views.  For my PhD, I have researched different types of attitude towards EU membership, particularly on immigration and economic questions.  Having analysed a range of survey data, I have found no empirical evidence suggesting economic and identity attitudes on the EU question are anything but one-dimensional.  Economic and identity concerns are tightly correlated - i.e. they face the same way, either against or for membership.  They only form observable separate dimensions when survey indicators are 'cherry-picked' to conform to the beloved two-dimensional theory (E.g. Boomgaarden et al., 2011, Mapping EU attitudes: conceptual and empirical dimensions of EU support.  European Union Politics, 12(2) 241-266).

Some of my research is displayed below.  The table shows a range of 'items' or survey question responses from British voters (vertically) to the 'Intune' survey of 2009 (random-probability sampling).  Some of these questions ('Diffuse Identity items') tap voter feelings of attachment and belonging to Europe and represent deep-seated emotional ties - 'hard support' if you like.  The second set of questions are more specific and are meant to tap rational (economic) evaluations of membership and EU performance, such as whether it benefits me personally or the country. This group of items represents so called 'soft support'.  According to the received wisdom of academia, these two sets of attitudes should not load onto one dimension, but load onto two dimensions.  In other words, the set of hard identity items should correlate with each other, and so should the set of soft performance evaluation items.  But the two groups should not correlate together as one because they are different dimensions.  This is not the case.  The one dimensional model is a better fit to the data than the multidimensional model (denoted by a higher Bentler-Bonnet test score and a lower Standardised RMR test score).  This means it more accurately reflects public attitudes to Europe.  The public see the questions of identity and economics as one.  The idea of two dimensions is a fiction, according to my research. 



Imaging British public attitudes as one-dimensional helps explain why support changes in response to new information - and it does. If British EU opinion is a 'great concrete block' of deep-seated unconscious prejudice - as Tony Blair's pollster Philip Gould* said in 2002 - why should it move in a meaningful and measurable way?  But it does.  You will have to wait for my PhD to be published for more on the structure of EU support, and why levels of British public support go up and down.  Suffice to say now, the main implication of my work is that we need to take the wisdom of the voting crowd far more seriously than the existing academic literature would suggest. 

For campaigners, this is yet more reason to adopt a bottom-up approach where the public are engaged by the respective sides and take part in the debate.  Don't think they will be influenced by elite arguments like they were in 1975.  The public know a lot more about the EU now, and elites are less trusted.  In my PhD I also provide multivariate modelling of time-series data, showing that the power of politicians to influence public attitudes has reduced sharply, particularly during the Eurozone crisis between 2008-2012.  The public are increasingly drawing their own conclusions, armed with new information on EU performance.  They now have sufficiently well structured views to be repelled by patronising politicians who they don't trust much.  They are far more likely to listen to the rest of the public, which could make crowd sourced 'bottom-up' campaigning a decisive tactic, if anyone knows how to do it.  And don't imagine that their judgements won't change sharply in response to new information on the immigration issue, on EU or domestic economic performance, or moving evaluations of the party leaders.

I don't believe that either side of the debate (particularly remain) are ready to really trust the public with a crowd-sourced campaign.  The expensive PR agencies of central London will be in heavy demand once again, producing winning arguments and clever slogans led by often suspect opinion polling.  Inside their lavish offices the detached members of the political elite will devise plans to manipulate the conscious thoughts and unconscious desires of the public.  The public will never be granted ownership of the campaign with a buzzing and gamified online strategy.  By this I mean voter / activist / spokesmen profiles and fund-raising; grass-root campaigning leaderboards and prizes; cartoon, blog, and vlog competitions; online ratings of politicians or arguments and prediction tournaments on what might happen.  If you haven't heard of this stuff, its because it has never happened.  Instead the public will be cajoled as a plaything, not treated as players - never granted ownership of the campaign, for example on which spokesman should get to appear in the media. Excellent spokesman could emerge from the general public providing a greater voltage to the debate, perhaps having been voted as a winner within an online competition.  Who is to say there is not a brilliant communicator out there amongst the general public who has yet to be unearthed?  Such a campaign would not be devoid of PR spin. Any campaign that could show authenticity in listening to the public would have the right image.  The voters would feel part of their own movement.

The blog below on the pernicious impact of Sigmund Freud and family discusses why our intellectual culture has been anti the 'wisdom of crowds' idea - that the many are smarter than the few.  It's the underlying reason why academics justify crowd stupidity and is most prevalent on the more established 'remain' side of the EU debate.  The remain lobby has never believed in the wisdom of crowds since the days of Monnet.  But their elite-centric view of the world of grand top-down plans and big centrally coordinated groups, is fast being pummelled into a flat expanse of rubble by the Internet.  If there is a long term underlying cultural change that will see 'remain' lose, this is it.  The crowd will understand it faster than the planners and Europhiles. The future is theirs.

     

Sigmund Freud and family have a lot to answer for - a rant against Freudianism in theory and practice
Apart from advocating cocaine as an anti-depressant, Sigmund Freud first suggested that our behaviour owes more to emotions and desires lurking in the unconscious than the rational assessments we make consciously.  Western civilisation represses these base instincts Freud maintained. Subsequently, such repression would either be considered bad (as the Nazis thought, along with the lunatic Wihelm Reich, Herbert Marcuse and a school of modern day Marxists); or for good (as the generation of post-war paternalists believed).  What united both the fascists, Marxists, lunatics and the post-war interventionists, was a believe that the unconscious desires of the masses were there to be either let loose or managed by elites who always knew more.  Effectively, by giving a pseudo-scientific account of the dangerous capacity of the masses, Freud had legitimised any amount of subtle controlling by our leaders, benign or not.  ‘Nudge theory’ is merely the latest incarnation of the supposedly benign variety.  Blair was perhaps the most pernicious recent exponent of mass manipulation of the public into support for his just causes, such as his liberal interventionist foreign policy and support for EU membership. Without compunction, he and Alistair Campbell happily 'sexed up' the dossier justifying war in Iraq in 2003.
Before the Internet, and the emergence of a new faith in ‘crowd wisdom’, twentieth century business followed the political manual of Freud, thanks in part to Freud's nephew, Edward Bernays. He pioneered techniques to deceive the public into rampant consumerism (helping to contribute to the Wall Street Crash of 1929).  He was the first to talk of 'public relations', a disingenuous term for as practice in 1920s America of tapping public emotions over reason. One of his first jobs was to break the taboo on women smoking in America by shoving cigarettes in the mouths of suffragettes (who were then photographed) to connect the activity with female emancipation.  The Freudians understood cigarettes at the time to be represented in the subconscious mind as penises, so suddenly women had willies.  After WW2 he worked for the American government stoking up the emotions of Americans against a perfectly legitimate new government in Guatemala, the original 'Banana Republic', that threatened US banana interests and was made to pose a communist threat to US security through his falsehoods.  If you haven't watched Adam Curtis' documentary, 'The Century of the Self' (BBC, 2002), please do. ('TheMayfair Set' by him from 1999 is brilliant too – spot a similar theme).  Bernays’ favourite word was stupidity.  In fact he called most people around him stupid.
Naturally enough, the alternative position to believing in 'crowd wisdom' is considering the public stupid and advocating paternalism.  They are really two ends of the scale.  Perhaps the greatest exercise in paternalism since 1945 has been European integration. Think of the dominant narrative of European Studies departments about British voters and the EU: that European integration is objectively beneficial but remote and complicated, beyond the cognition of generally uninterested voters.  Instead those voters are falsely conscious of their interests.  They are obsessed with the threat posed to British identity by EU governance and EU immigration.  (See my critique of Hugo Young's 'This Blessed Plot' on this website - May 2013).  In his final interview as Prime Minister, Blair offered a frank summary of the false  consciousness theory in respect to EU integration: 'The British people are sensible enough to know that, even if they have a certain prejudice about Europe, they don't expect their government necessarily to share it or act upon it' he said. 
More broadly, this narrative on public support for the EU is incompatible with a whole branch of classical political science rooted in the assumption that man is capable of pursuing his own interests, or as Anthony Downs puts it (1957:2) “that there is a pattern – or ordering of political behaviour that is rational, i.e. ‘reasonably directed towards the achievement of conscious goals’. This is fundamental to the study of voter attitudes, because ‘only if human actions form some pattern can they ever be forecast or the relations between them be subject to some analysis’. The alternative approach, that has the currency in European Studies departments, elevates psychological, social and emotional factors over economic ones in voter decision-making and is sometimes called 'behavioural economics'.
My PhD thesis (summarised in my blog from April 2015) argues that the whole study of public support for EU membership has become deeply confused and unhelpful because it is now dominated by this pessimistic approach to voter motivations, incorrectly specifying two distinct and conflicting dimensions to public EU support: soft (cognitive / economic / rational) and hard (emotional / identity - from the unconscious). No consistent theory is supplied about how these dimensions might be related and when each component becomes more important, and hence no rational theory is possible for what causes over time change in attitudes. It is really a failure to find a third way approach between believing in the wisdom of crowds and believing in their stupidity. 
This link is an easy to understand exposition of the behavioural theory from Prof. Richard Thaler, given in a lecture a few months ago at the LSE. (University of Chicago, co-author of ‘Nudge’ - that attempt to intellectualise paternalist government).  I dislike the sneering tone of Thaler, seemingly a common trait with most behavioural economists. The lecture is full of tiresome jokes about how daft (other) people are. The most ghastly behavioural economists (like Thaler) love anecdotes deriding their opponents and are deeply patronising of the public.  Paul Dolan, a professor at the LSE also talks briefly after Thaler (at 39:40). He says one preposterous thing. Before GE2015, he was a huge fan of Betfair as 'a really brilliant prediction market' in politics - suggesting, somewhat inconsistently, belief in the predictive power of 'crowd wisdom'. However since the election in May his view has undergone a 180 degree volte face, because the market got it wrong.  This is just one example of how people who try and occupy the middle ground between believing in crowd wisdom and psychological theory are just confused.  Of course, the informative value of a betting market on a future outcome cannot be judged on one case, however important that case may be.  Favourites do not always win, they only win a percentage of the time. Therefore the predictive quality of markets must be judged probabilistically, on multiple cases over time.
One of Thaler's buddies is Cass Sunstein, who wrote a largely admirable book on crowd wisdom called 'Infotopia' (2007).  Unfortunately he is also a 'nudger' like Thaler, believing that failures in crowd wisdom are caused by groups becoming 'cocooned' by 'information cascades' (essentially herding), and need to be corrected by the great and the good - the nudgers like him.  But when do we know that such action is required?  And who can we ever trust to nudge?   What do you think?  Listen to the podcast and don't waste £11.99 on Sunstein and Thaler's book.


* The late Philip Gould was well meaning.  He thought through his focus groups New Labour could actually listen and measure what the public wanted.  This was an improvement on Labour in the 1980s, but the 1997 Labour election victory was not 'an end to the elitist politics that has governed Britain for the last 100 years' - see here at 3.46:00.  In government Blair maintained the old paternalism.

Monday, 14 September 2015

An African City

A BBC World Service programme worth listening to early on Saturday and Sundays is "Weekend".  It's is a marathon 150 minute discussion of current affairs and global cultures, hosted by the excellent presenter, Julian Worricker.  It happens between 0500 and 0730 GMT (0600 and 0830 British Summer Time). 

Two interesting guests appear every week to comment on the show's content, without anything more than an a layman's interest in each topic.  They are usually well selected natural combatants.  A few weeks ago, the iconoclastic Peter Oborne made a guest appearance and it was early morning fireworks.  More recently Giles Fraser made his debut, the pious Guardian columnist and South London clergyman.  Unfortunately his performance is no longer available online.

The Reverend had to comment on a new African hit drama, which tracks the lives of some very posh and beautiful young Ghanaian women (who turn out to be sometimes loveable, othertimes ghastly).  They have returned to their homeland from America and Britain, encountering a range of Third World problems jeopardising their new First World life-styles.  Episode 4 can be watched by clicking here where one of the ladies attempts to recover her shipment of luggage, most keenly her personal sexual aid, entombed in Ghana's inefficient customs system.  The ten episodes comprising the first series are only twelve minutes long and only available on YouTube.  A second series is coming, aided by crowd-funding and now bought by a range of African broadcasters, aware of the proven popularity of the show.  Getting funding for home-grown television in Africa is clearly harder than in Britain.




For a mass of Africans, particularly its women, An African City is popular because of its aspirational flight of fantasy, rather like Sex in The City.  But people like Giles Fraser would prefer it if African television was concentrating on the continent's existing miseries: poverty, violence, disease and general squalor.  In short, he wants Africa itself to adopt his missions through its popular culture, never losing sight of its burdens that must be vanquished.

Upon his sincere expression of shock that a mention of Accra's slums never featured in any of the twelve programmes, he was duly reminded by the marvellously robust producer of the series, Nicole Amarteifio, that she retained the right to produce successful entertainment, the value of which, lay precisely in its unreality. 


Her pain and anger at Giles Fraser's criticism was authentic and clear to hear.  And it takes an objection from an African for it to resonate.  Africans want to get on with their own lives, on their own terms, without their changing culture being forever trammelled to the confines of a patronising Western image.  This is the West's love of pigeon-holing the great continent as a 'basket case', an outlet for its own guilty conscience that prefers to govern than watch, listen and enjoy.

 

Friday, 24 July 2015

Labour members supporting Corbyn aren't crazy. They are addressing fundamental issues of representation within the party with the next General Election five years away.

It's not weird that Corbyn is winning the Labour Leadership contest.  And the electorate of Labour Party members aren't deluded either.  Instead they are poorly understood right now.  Being told by the object of their resentment (the existing party elite) they are morons in need of a heart transplant if they support Corbyn - and should get a grip, taking their responsibilities more seriously - is red rag to a bull when we understand their real motivations.  As are the current campaign tactics of Burnham, Cooper and Kendall:  a barrage of emails and wearisome direct mails, texts and cold calls, force-feeding endless policy messaging down member's throats.  What was required was a community-based campaign from the bottom-up with the electorate contributing to the policy debate and the campaign with crowd-sourced collateral, and the Blairite / Brownite candidates being humble and listening.  It's too late to do that now and I don't believe these candidates know how to do it anyway. (See my 'gamification' proposals in the article entitled 'Singh on Song' on the right-hand column of this page).  Even at 2/1 Corbyn remains a decent bet, assuming the anti-Corbyn side do not manufacture the result by coalescing around a single 'stop-Corbyn' candidate. This blog looks at the recent YouGov polling which provides important insights on why the Labour membership is supporting Corbyn. 

Usually the best way to understand voter behaviour, and one that I would normally default towards, is the classic economic model grounded in assumptions of voter choice and self-interest.  Voters make 'political choices' in response to either the specific policy offering of political leaders which they think benefit them (Downs, 1957) or make broad assessments of party and leader performances on key issues that everyone agrees upon, such as better public services and a stronger economy and for most people less immigration (Clarke et al. 2014).  Every now and then this normal business of the voter is interrupted and trumped by more deep seated concerns about voters' fundamental position or identity within the political system.  This is no less self-interested or rational, particularly given the context and the timing of the leadership election.  Labour members know that the next election is a full five years away, so now is the time to get fundamental about the type of leadership they want, at least for a year or two, without the distractions of policy and performance minutiae.  Deep down, they probably know that Corbyn probably won't last five years, so 'getting real about the election' can be postponed until the next leadership contest which might involve more inspiring alternative candidates.  Indeed, this time the membership have been presented with what are frankly dud alternatives, again, having endured a couple of unlovable recent leaders culminating with an embarrassing one also in Ed Miliband.  Right now we are witnessing the Labour membership, given for the first time real power to select their leader by the new rules of the contest, address the deep-rooted failings of the party that lie behind the poor policy and performance.  They feel alienated and detached from a metropolitan elite that has hijacked the Labour Party over the last 20 years with its professionalism and centralisation, leading to an overly controlling and insincere form of leadership, all contributing to an increasingly detached elite from its grass-roots.  They want to feel part of the Labour Party again with some say from the 'bottom-up', rather than being excluded and patronised from the 'top-down', as has been the case under New Labour and subsequently under Brown and Miliband.  In the lexicon of political science, Labour Party members have become motivated by questions of 'social representation' not 'policy representation' when choosing their next leader. 
 
Now we have the best evidence so far that Jeremy Corbyn is leading the Labour Leadership contest and by a significant margin.  A YouGov / Times survey (22/07) of 1,054 Labour Party members puts Corbyn on 43% of first preference votes, ahead of Burnham on 26%, Cooper on 20% and Liz Kendall trailing on 11%.  Taking account of second preferences, Corbyn would beat Burnham in final round voting by 53% to 47%.  Even if this polling is a whopping 5% out, the basic story remains the same.  Corbyn is ahead. N.B. Those who criticise online polling methods should reflect that YouGov's vast online panel in the UK of nearing a million people, made the polling of sufficient Labour party members possible.  And the polling information it has delivered tells us enough about the dynamics of the campaign to inform on where the losers are going wrong.

Most of all, the poll has set the campaign alight.  The prospect of Corbyn winning this election is a fascinating prospect. He is one of the most left-wing and outspoken Westminster MPs who has rebelled against the recent Labour governments over 500 times.  Now he is quite likely to become leader of that party.  Having been 200/1 with Bet365 and 100/1 with Ladbrokes when he entered the race, he has been backed all the way down to 2/1 second favourite.  He has therefore a 33.3% chance on winning according the betting markets.  When I was studying A-level politics in the late 80s, if you had to pick out a classic Militant leftie driven by Marxism (and we all subscribed to Marxism Today at my school), it would have been Dave Nellist, Derek Hatton, Dennis Skinner, Eric Heffer or Corbyn, still now a member of the Socialist Campaign Group and a Morning Star columnist, and still very angry.  Check him out here berating Krishnan Guru-Murthy on Channel Four news: he has the capacity to be fearsome.

The Militant Jeremy Corbyn in the 1980s - this photo resembles a police mugshot
 



The YouGov poll sheds light on why Corbyn is proving a popular choice, particularly among newer, younger members, trade union affiliated voters and £3 'registered supporters'.  All we hear from the Labour party establishment is that these voters are crazy morons, having an 'emotional spasm', together with the MPs who allowed Corbyn onto the ballot paper in the first place.  According to this script, Labour Party members are engaging in an escapist fantasy when they should be acting responsibly in choosing the best leader for the country.  Not only is this tone likely to further disconnect elites from publics in British politics, it shows a lack of understanding of what is motivating Corbyn supporters.  Voter pessimists would do well to examine the findings of the YouGov poll in detail.

In particular, YouGov asked the Leadership contest electorate what they felt were the 'qualities most needed in the next Labour leader'.  If the voters were motivated in picking a candidate to increase the possibility of a Labour government, we might expect them to place emphasis on the effectiveness of the candidates in winning elections, a long term view five years from the next GE.  However, this is not the case.  Just 27 per cent chose this strength as important.  By contrast, voters in the election were more interested in the candidates being 'in touch' (62%), fighting the Conservatives hard (53%) and being a 'strong leader'.  These are more intrinsic qualities of the here and now.   The voters in this election are not so interested in the instrumental purpose of finding an election winner as feeling they have a leader who they can connect to, and be proud of, in the period running up the next vote.  In short, my reading of these results is the voters are sick of the type of slick, professional leaders who they don't feel good about being in charge.  They want someone - like Corbyn - who will make them proud to be Labour supporters by socking it to Cameron with some passion and belief.  This motivation characterises Corbyn supporters in particular.  Just 10% of them are interested in the candidate 'understanding what it takes to win an election' whereas 30% of Burnham supporters do, 39% of Cooper's and a staggering 63% of Kendall's.   This is the real cleavage of the campaign.  Those who want Labour to be an election winning machine first and foremost are against Corbyn, those who place emphasis on someone who takes on the Tories strongly, sticking up for the (left-wing) concerns of ordinary people, are for Corbyn.  For these voters it's not so much about electoral victory in 2020, it's more about restoring some dignity to the Labour movement in the meantime by not selecting another dull, careerist professional for the job. 

Thursday, 7 May 2015

Predicting GE 2015: Which group of forecasters should we rely upon: 'the expert crowd', 'the betting crowd' or the 'mass public crowd'?

Author note: Albert Tapper (@mincer) was political market-maker at the Sporting Index spread betting firm for the 1997 General Election and is now researching Ukip and this year's General Election for a forthcoming book by Dr Matthew Goodwin and Caitlin Milazzo.  
 
Postscript, Sunday 10th May 2015: 

The article below was written on Thursday 7th May, before the polls had closed on this extraordinary UK Election. Then the unexpected result came through, met by universal shock.  Against all the expert forecasts and final odds of 14/1 on Betfair, the Conservatives won the 325 seats required for an 'overall majority' in the House of Commons, delivering one of the greatest upsets in British electoral history.  David Axelrod, President Obama's election strategist described it as the most stark failure of polling that he has ever seen.  The British Polling Council (BPC) called an immediate inquiry.  The final tally of 331 Conservative seats was 41 seats higher than the last betting market estimation from Sporting Index of 290 seats.  It was 50 seats higher than the daily average of 281 seats from the three main academic forecasters: Electionsetc.com, Electionforecast.co.uk and the Polling Observatory  (16 April 2015 - 6 May 2015 inclusive).  The final prediction from Electionforecast, released on the 7th May was also for 281 Conservative seats.  Given these forecasts and the betting markets had barely moved ten or fifteen seats in the preceding six months, the scale of the sudden movement was stunning.  If, that is, the change was 'a sudden movement' or 'late swing'. The Conservatives may have been leading all along, just never measured as doing so.  The 'stewards inquiry' by the BPC will discover this and its causes (Lib Dem / Labour 'switchers' or Ukip 'penumbra'?)  Either way, the betting markets and the academic forecasts had got the result badly wrong, particularly the 'expert' academics.  Both groups will struggle to pass blame onto the miserable collective effort of the pollsters, upon which their forecasts were very largely constructed.  The emphasis given to polling data in any forecast model or individual voter or betting decision is intrinsic to the quality of that decision, each forecaster has the power to reject such information.

(Footnote: The betting market figures are taken solely from the Sporting Index spread market, the most liquid of all political betting markets where seats are traded like a share price.  By contrast, the seat forecasts from fixed odds bookmakers such as Ladbrokes and Betfair are derived of probabilities from their relatively illiquid constituency betting markets and are therefore a less reliable guide to crowd wisdom.  Crucially these fixed odds company predictions are not tradable and are produced for public relations purposes only.  Unsurprisingly, during the course of the campaign these poorer predictions have been in regular arbitrage, notionally at least, with the tradable spread betting markets.)    

In this regard, this article provides new insights into how the betting markets and the expert forecasts moved during the campaign, both relative to each other and over time.  On Twitter, the more vocal elements on both sides have struggled to conceal their antipathy towards the other at times.  Bettors have charged experts with arrogance and unworldliness, whilst experts readily dismiss bettors as simplistic or partisan.  No surprises then that large differences between the two sets of forecasts have opened up, despite their self-professed mutual reliance on the same (now suspect) polling data.  In the light of these differences and the systemic issues revealed in the polling, it is worth speculating whether the betting markets were, late in the day, becoming increasingly sceptical about the reliability of the polling, and if so why?  In short, the market could have grown wise with age by beginning to discount the polling info - whilst the academic forecasts made no change to the powerful influence of vote intention polling on their models.  One rationale for discounting, given prior to the event, comes from Matt Singh of Number Cruncher Politics, who argued that the relatively low vote share polling 'snapshots' for the Conservatives were incompatible, historically, with the relatively high polling numbers for Conservative 'economic competence' and 'prime ministerial competence' ratings.  No previous party had failed to win overall power with such good fundamentals.  If the market was making this judgement against the reliability of the polls, it may explain the last few day divergence between the betting line predictions and the expert forecasts (see fig 3 below) - although a differential was in fact opening up long before the last few days of the campaign.  This would be a rational or economic explanation for the divergence, as opposed to the common charge used against betting market accuracy of right-leaning bias.  Punters tend to be right-leaning, particularly the city based clientele of Sporting Index, and they like backing their favoured team, the Conservatives.  

Original article, Thursday 7th May (describing the performance of experts and betting markets in predicting the election):
 
Prediction matters


Getting political prediction right, counts.  From influencing capital markets to cutting through the bluster of political bandwagons, decent forecasts are also central to political strategy, like when to adopt riskier vote-winning plans or shore-up core support with basic messages.  Perhaps most importantly, voters must also judge correctly how well parties are faring if they are to vote tactically.  As Stephen Fisher of Oxford University has pointed out, in the ultra-competitive contest of GE2015, new types of tactical behaviour are emerging called ‘coalition-directed voting’.  An example of this are Labour voters supporting the Conservatives in the short term, against a feared SNP bid for a coalition with Labour, so long as Labour aren’t doing sufficiently well to get an overall majority themselves (in which case the SNP would be less of a threat).  Where though, should these voters turn for a reliable prediction of the result to inform such tactics?  Are the betting lines or the expert academic models best?  Is there an alternative source of crowd wisdom beyond the potentially biased judgements of academics and betting crowds? 

From late 2014, I began noting down the daily forecasts made by three separate groups.  Firstly a collection of academic 'expert' judgements, secondly the forecasts of groups of ‘punters’ reflected in betting markets, and finally, a more random sample of the mass public predicting their own local constituency results.  Analysing this basic data, I ask now which group has been leading or following the other?  The results show that academics have done well in leading insights on the high likelihood of an SNP landslide in Scotland and also a hung parliament outcome, should these outcomes materialise.  However the experts and the public still diverge on the relative performance of Labour versus the Conservatives.  Because of this, there are going to be very public winners and losers when the actual results become known from tomorrow.  In summary, if the Conservatives beat expectations and get in excess of 290 seats (and Ukip get more than two or three seats), then it will be a vindication for the betting market and mass-public crowd.  If Labour does well, say more than 275 seats, the academic crowd will be closer to the actual result.  Either way, we must remember this is only one skirmish between each crowd – just one roll of the dice.  One side will have won a battle and not the war, although it may feel worse for the losers tomorrow morning.  The debate will rage on about which crowd is the wisest.

Experts v The Public

Before the final definitive verdict tomorrow, your instinctive preference for either the expert or the mass public view will probably depend on a hunch rooted in one of two alternative views of human knowledge.  For millennia man has debated whether the best guide to the world around us is the knowledge or reason of the few (a Platonic tradition), or the practical experiences of many (an Aristotelian tradition).  Suffice to say here, some forecasters believe future voting behaviour is a highly technical matter, which is best understood by a few experts, capable of weighing up the validity of key theories and managing masses of data to refute or corroborate them.  The mass public wanders ignorantly, befuddled, helpless, deaf and dumb, without discernment.  If you believe this elitism, then you probably follow the academic crowd for now, until they are proved right or wrong by the results.  Alternatively, you may feel sceptical that a collection of academics, or punters, can be sufficiently diverse and independent in forming their opinions, and are more prone to either behave in a partisan or ‘herd-like’ manner.  In this case you are a pluralist or a 'liberal' in its truest meaning.  You may also feel that voting behaviour is too complicated and delicate a matter, driven by a mass of local-level (constituency) information that is beyond the comprehension of a bunch of academics and their whirring models, or even a crowd of incentivised bettors.  If you take the argument this far, you are probably an economic liberal like Margaret Thatcher's favourite economist / philosopher, F.A. Hayek.  There is certainly a strong element of Hayekian epistemology (theory of knowledge) in why betting markets are superior to the cleverest individuals at prediction: the environment of the voting decision is like a vast information storage system, which can't be fathomed by any single individual, like the mass of supply and demand cues of an economic market cannot be understood by one 'planner'.

In this case, somewhat bravely, you could turn to the crowd in its purest form, trusting mass publics across every constituency to tell you, collectively, who they think will be their next MP.  This is perhaps the ultimate compilation of localised crowd-sourced intelligence in a British election.  For the first time in electoral studies, online polling technology is revealing new insights into these predictive views of the mass public, allowing us to ask more of them, more regularly and more cheaply.  Much has been made about the increased volume of polling on how people intend to vote because of this online polling revolution.  In the last six months of 2009 there were just 103 vote intention polls.  In the last six months of 2014 there were 283.  But the vote intention question is mainly fodder for the academic models and betting calculators, as is the welcome addition of Lord Ashcroft’s constituency polling by telephone, recognising the need to find local patterns in voting swings in a post-UNS (uniform national swing) multi-party election world.  By contrast, the true ‘crowd wisdom’ question is not how people will vote, but who do they think will win?  To do this on a constituency-by-constituency basis, to mine a new mass of localised knowledge, requires vast national surveys calling on enormous online panels. Even YouGov’s 600,000 strong UK panel is not really big enough for the job, yet.  This type of crowd forecasting will improve with time, as we are able to ask sufficient volumes of people in each individual constituency for their predictions.  The key assumption is Condorecet’s Jury Theorem which states that if group members have a greater than fifty per cent chance of making the correct decision (they have at least sone wisdom), then the probability of a correct majority vote will increase rapidly towards unity as the group size increases to infinity (Condorcet, 1785, Murr, 2015).  So if the crowd are 50.001 per cent right then that is good enough.  The prediction will only get better as more people are asked the question.  Unfortunately, the same logic works in reverse.  If the crowd are 49.999 per cent right, the crowd will get more reliably wrong as members are added. 

Despite a rather small sample for the job, this idea got an early run out at this election thanks to the fascinating work of Rookie political scientist Andreas Murr of Oxford, who drew on an internet survey of 17,000 voters conducted by YouGov in February of this year (that is an average of just 25 members of the public in each constituency).  If it does well, then the future of predicting elections could be about to change.  We may become attached to asking vast online panels what they think will happen, harnessing new seems of local information in vital local areas previously passed over previously by group of experts.  Online panel growth is the disruptive technology that may be changing not only how future political events are forecast, but sporting ones too.  Swap 'parliamentary seats' for 'Division 2 season points' and the same benefits from exercising crowd wisdom apply.  The wisdom of crowds idea, made applicable because of the internet and growing online panels, dangles valuable new insights for bookmakers and the betting public alike.  And the online panel companies understand this.  They consider their panels not just a source of predictive information, but also the basis of new social communities of predictive activity, a sort of gambler's Facebook. Rather than asking panellists boring survey questions to obtain obscure football league information and aggregated wisdom, why not offer them the chance to reveal their knowledge by playing games at the same time, such as Fantasy Football Manager?  The growth of online poker showed how gaming communities spring up quickly between ego driven poker players.  Predictive sporting tourneys may be next.  YouGov are the most advanced British online panel company to attempt the 'gamification' of their panel experience, in a bid to morph from market research firm to social media giant.  It faces stiff competition from the bigger American firms such as Research Now, who now have over six million global panel members.  It remains to be seen whether British YouGov have the capacity to out innovate their American competition to grab the bigger prize.  I shall return to the constituency level crowd wisdom model at the end of this article.

Expert Academics

Back to March of this year, just over two months before the election poll.  The scene is a large purpose-built lecture room at the London School of Economics.  Like a sugar addict in a sweet-shop, I was enjoying my favourite day in academia since leaving the bookmaking industry in 2009.  The subject of the scholarly inquiry was prediction, but the personnel involved were far removed from shift-race-goers or grubby betting shop types. 

Sixty of the world’s finest political scientists specialising in voting behaviour and opinion polling were having their traditional pre-election day get together.  Here were the brightest collection of individuals I’d seen assembled under one roof.  And they were staking their hard won reputations, not money.  Many had flown in from around the world to review a dozen different seat forecast models predicting the UK General Election, the blue-ribbon political horse-race of the global elections calendar.  The winning model would be the one that contained least ‘total seat error’ between predicted seat totals for each of the parties and the actual result on May 7th. 

This year the contest had the makings of a classic. Heightened interest among the betting public was making it the biggest non-sporting betting event of all time, with turnover up three-fold on 2010, according to Ladbrokes.  The British contest, already tight, had become even more of a challenge, spiced by the presence of two live dark horses in the field with little previous form: the SNP and Ukip.  The extent to which these unknown variables could weigh heavily on the voting result was causing much debate and uncertainty.  Many academics who had based their careers on the reliability of the grand old model of UNS (Uniform National Swing) were feeling sombre.  There was a deep concern that the British first-past-the-post system might not be able to cope with the new multi-party dynamics.  These concerns it was felt, were made more serious by the central implication of the new Fixed Term Parliament Act, designed to facilitate the last coalition government, which makes an emergency ‘ad hoc’ election now more difficult to call.  Therefore the academics worried.  The British polity could be condemned to five years of unstable and ineffective government based around vote-by-vote deals, uneasy coalitions or delicate ‘confidence and supply’ arrangements.

The academics faced a quandary.  As top forecasters, they recognised the deep uncertainty of the General Election result, but also knew that in the current academic era, there was public demand for a confident forecast.  Deep down, they felt there were no simple answers, no one idea that explained it all.  In the terminology of expert political judgement (Tetlock, 2005), those in the room were ‘foxes’ not ‘hedgehogs’, scrappy creatures who believe in a complicated synthesis of many little ideas and multiple approaches towards solving a problem.  The best electoral forecasters in the world had not gained their reputations through one knock-out punch, but hard research and gradual learning.  But now journalistic simplicity was required of them, whilst maintaining their honesty.   Their industry was booming and gold was on offer.   The political science of predicting elections was in rude health, popularised by Nate Silver’s best-selling book ‘The Signal and the Noise’ (2012).  And in an era where academics are demanded to make an impact on wider society to secure research funding, a premium is placed on being ‘outward-facing’.  Torn therefore, by intellectual modesty and humility on the one hand, and public demand for simple answers on the other, some stars of the field are emerging within British political science who can do both.  Among those presenting models was Rob Ford, well known for his book ‘Revolt on the Right’ on the rise of Ukip, co-authored with Matthew Goodwin which won the Political Book of the Year in 2015.  (Matthew is writing an eagerly awaited update book on Ukip, for which I am a researcher).  Also demonstrating his model was Chris Hanretty from the UEA and his team (Benjamin Lauderdale, Nick Vivyan and Jack Blumenau), whose electionforecast.co.uk is now an integral part of the BBC’s Newsnight coverage, and who also provides the UK election model for Nate Silver’s booming FiveThirtyEight US media business.  Presiding over the event were the eminent John Curtice and Simon Hix, with guest of honour, Sir David Butler, inventor of the Swingometer and a founding father of psephology, now 90.  In short, here was the ‘expert crowd’ of GE2015 in one room.  If good judgement on the General Election is related to high IQ and years of learning, this was the place to find it. 

Fig. 1 ‘The Expert Crowd’.  The young rising stars of British political science (in background, Chris Hanretty, Rob Ford and Stephen Fisher) are grilled about their General Election predictive models by older established names, John Curtice, Sir David Butler (90), Simon Hix (standing) and Vernon Bogdanor – Foreground




The twelve predictions are listed in table 1 below, along with the remarkably similar results of an ‘expert survey’ of 465 Political Studies Association academics, 45 journalists and 27 pollsters.  Of the forecasting conference predictions, all predict a ‘hung parliament’, half with the Conservatives and half with Labour holding the most seats.  The average seat prediction shows Labour winning just four seats more than the Conservatives (283 versus 279).  Ukip are predicted to win three seats, the SNP forty-one and the Liberal Democrats twenty-one.  This would be a disappointing result for the Conservatives, with the party falling short of being able to cobble together a minority government.  Assessed against the final polling it looks a fair collective judgement, although it may prove to be slightly short of the SNP and Conservative seats and long of Labour.

Table 1. ‘Expert Crowd’ predictions 2015 Forecasting Conference, LSE, 27th March 2015



 
For all the wisdom of these academics, what possible reasons may exist to doubt their aggregated judgement as a ‘crowd’?

I think there are two possible doubts about the claim made by the researchers of the Political Studies Association expert survey (Chris Hanretty and Will Jennings) that averaging academic forecasts engenders ‘wisdom of crowds’ benefits.  The first problem is the diversity and independence of the group.  Both these factors are fundamental assumptions of James Surowieki’s theory, discussed in an excellent early podcast (2004) from the man here, yet the congregation at the LSE in March and the Expert survey of PSA members looks little more than unrepresentative ‘expert panels’.  In short, the models and the judgements of both may be drawing on insufficiently narrow sources of information, concentrating heavily on vote intention opinion polling, compared to the information resources utilised by the mass of the general public up and down every constituency in the land.  The fear exists that the expert models sink or swim with the polling, however well aggregated and weighted, along with some adjustments for historical behaviour such as a late Conservative rally or ‘reversion to mean’ (voters tend to fall back on what they did before) which turns ‘snapshots’ of polling into predictions of future results.  Worse, a certain ‘group-think’ or ‘herd mentality’ may be in-play, making the academic crowd not independent of each other.  In particular, for the members of the Political Studies Association, their high level of political knowledge may be of each other’s well publicised work, continually circulated to each other, particularly the models of the Polling Observatory, Elections Etc, and Electionforecast.co.uk.  Yes they have produced different forecasts, but to what extent are these differences the result of widely sourced data, or merely subjective adjustments to their working parts, or their assumptions?   Voting behaviour scholars from America, particularly Michael Lewis-Beck, made this objection to the British models at the LSE conference.  He felt there were just too many ‘moving parts’, too many formulas and assumptions in the models for the results to reflect the data rather than inevitably the forecasters subjective viewpoints.  And here we come to the second problem.  Potential bias inherent within the academic community.

Academics have a habit (often irritating to the public) of insulating themselves from the charge of bias by claiming superior knowledge, in particular command of their own data.  More than often it is justified, but it is a problem when it this becomes almost cultural, to the extent that new information from other sources can be dismissed without proper consideration.  The charge here is of a certain intellectual arrogance or hauteur, and it's not hard to find among political scientists, however ‘fox-like’ they are when approaching their own data.  Take for example a letter that was sent among an elite sub-group of the PSA members that specialise specifically in voting behaviour and elections (EPOP), about the relative results of their own grouped opinions in the recent PSA survey.  Somewhat contemptuously, it reminded the rest of the group that:

Colleagues will have noticed the PSA’s survey of election experts last week… (We) have separated out the predictions from those who actually know things about elections (ie, us) from those who don’t (everybody else).


Whilst this may have been ‘tongue-in-cheek’, there is little hiding the considerable doubt held by academics about the quality of public judgement more generally, a theme running through voting behaviour work dating back to the seminal 'American Voter' studies of the 1960s.  These painted a picture of the public as intellectually disorganised, affective in their motivations when not largely uninterested and uninformed about representative government (Campbell et al, 1960; Converse, 1964).  Why should this group be any better at predicting politics than they are practicing it, as measured by the public forecasts implied in betting market prices.  Here is not the place for a debate about whether betting prices benefit as ‘opinion backed by money’ or mislead with a republican / conservative confirmation bias, simply to show for now the arbitrage or mismatch of estimates between some of the academic models and the public markets.  Most startling was the initial forecast of the Polling Observatory, which opened on the 16th March forecasting 265 Tory seats whilst the leading city spread betting firm, Sporting Index, was forecasting 282.  It is hard to say whether there is partisan bias in the markets or the academic forecasts from this alone, but the arbitrage between the two throws some caution on both, a dispute which will only be resolved by comparisons of a long run of forecasts with results.  For now, whilst the academics can charge the markets with a pro-tory bias, the bookmakers can equally claim that the academics do not represent national political opinion either, as this self-selecting survey, conducted by the Times Higher Education Supplement this April showed.  Just four out of 1,019 respondents said they supported Ukip, eleven per cent Conservative and forty-six per cent Labour.  

Without knowing the final result as yet, how can we resolve who has performed better over the course of the election campaign, and therefore shed light on who is less prone to bias or wiser to information: the betting market crowd or the academics?   One method is to look at which side has converged in its predictions on the other (indicating that it is following), or whether one side has diverged in its opinions (indicating it is opposing the other), or whether they have mutually converged on each other in their estimations as the campaign has progressed (indicating increasing agreement). 

Con v Lab

Firstly, let’s look at the question of bias within betting markets, favouring the right because punters tend to be conservative leaning, and also within academic crowds, because public sector university workers tend to be much more left-wing than average.  In fact, as figure 2 shows, there is actually very little disagreement between the average of the three academic models predicting Conservative seats, and the betting market from city spread firm, Sporting Index.

Figure 2



The betting market average line for the last 50 days of the campaign is just 4.4 seats higher than the average for the academic lines.  The model from Stephen Fisher has indeed been more consistently bullish about Conservative chances (averaging 290.8 seats) than the Sporting Index market, averaging 285.5 seats.  However the academic average is lowered by the bearish forecast from the Polling Observatory, averaging just 269.5 seats for the Tories over the last part of the campaign.  Despite drawing on pretty much the same polling evidence, a range of over twenty seats between the bottom academic forecast of Tory seats and the top, must leave the public wondering who to believe?  The charge that a set of subjective assumptions is ultimately driving these models, which are over-laden with ‘moving parts’, has added weight.

At this stage, I am dropping the top and bottom forecasts to concentrate on the middle forecast of Hanretty, Lauderdale and Vivyan (electionforecast.co.uk) in comparison with the betting lines from Sporting Index, as it seems closest to the average of the academic prediction, and therefore fairly representative.  Figure 3 shows the daily over-time trends in both lines since 1st December 2014.  How have the two predictions related to each other over the course of the long campaign?  Firstly, both lines are clearly close to each other in forecasting Conservative seats, and it is not immediately noticeable that there is much difference in the predictions prior to mid-March.  In the first part of the date range between 1st December and 15th March, the daily average for each was between 282.4 (academic) and 282.2 (betting).  It is only in the run-up to polling day from the 16th March that the lines start to diverge, as the added volume of money for the Conservatives pushes the betting line above the academic forecast line, ending nine seats higher on polling day at 290 seats versus 281 seats from electionforecast.co.uk.  Here there is genuine disagreement between the betting crowd and the expert crowd.  Was the market coming to distrust the static polling lines?  Bettors think that the polling, by itself, underestimates the Conservative seat total by around 14 seats (using YouGov’s ‘Nowcast’ for 6th May of 276 Conservative seats which translates their polling numbers into seats on that day).   The academic line, driven largely by polling, also overestimates the final Tory seat tally compared to the YouGov 'Nowcast' estimate, because it also expects a last minute rally for the party.  But compared to the YouGov number, this is only five more Conservative seats.  In a tight contest, who is right in this dispute could be crucial, and we will only know tomorrow morning.

Figure 3



SNP

The battle between the betting crowd and the expert crowd is multi-faceted at this election, it does not just depend on who gets the balance between Conservative and Labour seats right.  One of the major stories of the campaign has been the rise of the SNP.   Here the picture paints expert wisdom in a favourable light to betting market wisdom, because the forecast of electionforecast.co.uk was spectacularly efficient in estimating early – way before the betting markets – a deluge of extra seats for the SNP.  Figure 4 shows how the betting market line of Sporting Index SNP seats took a while to converge on the expert academic line.  This moment was one of the major opportunities to make money in this campaign and shamelessly, I must admit to flagging it up here, at the beginning of December, although I didn’t have a bet myself!  The betting public have been reluctant to join the SNP bandwagon, and once again, this may be because punters (south of the border at least) have been reluctant to support the Scottish Nationalists for partisan reasons.  It is noticeable that once again, in the dying days of this campaign, the Sporting Index betting line has dipped below the academic line.  This shows there are plenty of SNP sellers about still, and the firm Sporting Index, may be in the enviable position of cheering on SNP wins, whilst shedding tears at tight Conservative victories tomorrow morning.

Figure 4



 The question of whether there will be a hung parliament, has also been central to this election campaign.  One of the psychological traits of the punter is an aversion to cheering on draws.  Bettors like excitement and dramatic results, sometimes letting their hearts rule their head.  When I worked at either Sporting Index, Ladbrokes and Betfair, a 0-0 score in football was usually the best result for the layers (the bookies).  For this reason, ‘No goal-scorer’ (includes own goals) is usually your best bet in any football match, without looking at any form, because it is the most unpopular one.  For the same reason, punters in this election have been keen on betting on an overall majority, in particular a Tory one, on fixed odds lines and spread lines.  This would be a terrible result for the fixed odds firms in particular.  Because of this, the line denoting the betting crowd’s prediction of a hung parliament has been consistently lower than the electionforecast prediction, which now makes it a 100 per cent certainty.  Punters still hold out for a 6 per cent chance of a Tory majority and it is only in the last few days that it has dropped below the 10 per cent mark, seemingly for good.  Figure 5 shows that betting wisdom, or lack of it, has gradually converged on the consistent view of the experts (the black line), that no party will win 326 seats required for a majority in the House of Commons.  As with the case of the SNP, following expert wisdom on the question of a hung parliament would have been lucrative, and there is still room to oppose a Tory majority for some cash.


Figure 5



Ukip

 Finally we turn to the question of Ukip, which has been the subject of my own research during this election for Drs Matthew Goodwin and Caitlin Milazzo’s forthcoming book.  Has expert wisdom or public wisdom proved a more reliable guide here? (See figure 6).  This is a more complicated story.  The graph shows that both the expert predictions of Ukip’s vote share (electionforecast in mauve and electionsetc in khaki) and the recently established vote share market from Sporting Index in red, hold out for an impressive Ukip performance of around 10-14 per cent of the vote, up from 3.1 per cent in 2010.  This prediction may have declined slightly since the 1st December 2014 and in the last few days, but the estimated Ukip vote has hardly been 'squeezed' as some commentators have suggested.  The betting market prediction for Ukip seats has however collapsed, from over ten at the start of December, to less than 3.5 now.  Both the academic forecasts shown in the graph for Ukip seats (in light blue and black) had never been higher than five and the electionforecast prediction never higher than two.  So we might think that the market has been unwise, and overly exuberant about the ability of Ukip to convert seats into votes, the opposite of its attitude to SNP seats.  One interesting psychological aside here is the predisposition of Ukip voters to take risks.  This relationship is shown clearly in all British Election Study surveys that have asked the question.  In short, Ukip voters, like their leader are punters through and through, and have enjoyed backing their candidates around the country, seemingly against their realistic chances. 

However, this criticism of the Ukip betting crowd, draws heavily on hindsight and ignores how the campaign has played out.  Farage has failed to spark a serious media bandwagon, perhaps because the SNP took it (analysis of media citations of party leader names over the course of the campaign, suggests this).  When Sporting Index opened their market, I discussed the matter with their market-maker (Aidan Nutbrown), and there was considerable uncertainty about what Ukip could achieve, as well as recognition there would always be buyers of Ukip out there.  Ladbrokes even had a market on the party gaining 100 seats or more.  The situation was too uncertain for the betting market to apply a strict mathematical formula of votes to seats, like with the academic models, because expectations existed at the time that they could break through.  Further, there are strong reasons that the electionforecast model was not recognising local campaign effects which could play into the hands of Ukip.  Whilst a small and inexperienced party at campaigning, it could at least target ten or more seats relatively hard.  Difficult to assess micro level factors were never really considered by the largely macro models of electionforecast, and arguably still aren’t.  It is still quite possible that Ukip could win 3-4 seats, 3-4 times what electionforecast have consistently predicted.  Like the Conservative / Labour seats battle, the Ukip question will only be resolved tomorrow.
 Figure 6.



Conclusions

The research above has looked at whether the ‘expert crowd’ or the ‘betting crowd’ has performed better as a guide to the likely election outcome.  On certain questions, requiring a high level of quite technical political knowledge, notably SNP seats and whether there will be a hung parliament, the experts have led the way.  The betting public have gradually come into line with what the experts have been saying, pretty much all along.   However, looking at the Labour-Conservative contest, and whether the betting crowd has displayed a right-wing bias or the experts a left-wing one, it is perhaps still too early to say.  We will have to wait for the result to assess the bullish betting market forecast about Conservative seats, relative to the expert academic forecast.  One notable feature of the academic models at least, as opposed to the survey of 500 'trade association' members, is the remarkable variance in their forecasts. Taken individually, they leave the public and potential tactical voters none the wiser as to what will happen.  The average however is close to the view of the betting markets (just 4.5 seats lower), so the bias theory in either direction may be overstated.  Forecasters are as keen as the betting public to get this election right.

Lastly, let’s give final consideration to the mass public as a whole - the voters who will decide this election.  Although regularly reviled for their lack of wisdom and herd-like behaviour ever since the damning 19th Century crowd studies of Gustave Le Bon, the mass public are the largest group and inherently diverse and more independent when compared to academics or bettors.  However, their view, with the exception of endless anecdotal TV and radio interviews particularly by the BBC, has often been drowned out in this election by elite opinion echoing through social media channels.  Bookmakers too have been guilty of not giving voice to their punter’s views, instead talking about their own opinions, as if the views of their traders and PR men matter more than their customers.  Good bookmakers, very rarely ignore the betting behaviour of their customers.  Ladbrokes in particular, via its otherwise excellent spokesman Matthew Shaddick, has been all to keen to rubbish the crowd wisdom behind political betting, also a cardinal sin in bookmaking PR.  For example, on 27th March he said that betting markets are 'not a particularly good predictor of the results'.  If a Ladbrokes spokesman is not capable of standing behind the inherent forecasting value of his own markets, who is?

The addition of a massive free-play public prediction tournament (100,000 players plus), rather like the hugely successful SkyBet Saturday Super Six game on football, conducted online, and yielding crowd sourced seat forecasts, would have added greatly to our understanding of mass public crowd wisdom on the election.  Demonstrating a trust in public judgement that has been sadly lacking during the election,  It could have been run by the media, bookmakers or both, pitching the wisdom of the widest possible crowd against expert opinion, extending the range of participants beyond the narrower confines of those who place real money bets on politics and their associated biases. 

The mass public have also been tapped for their insights via focus grouping, as featured in Lord Ashcroft's regular polling reports. Sometimes however it feels this element of Ashcroft research is included to carry a joke or two about politicians or to add colour next to the dry survey reports, rather than uncover some serious insight on mass crowd wisdom.  There have also been two monthly quantitative surveys by other polling companies, which include items looking at general election public prediction, one by TNS and the other by ICM for the Daily Telegraph / Guardian.  The most recent was by ICM for the Guardian yesterday (6 May).  Whilst the vote intention question showed support evenly divided between Labour and Conservatives on 35 per cent each, the prediction question suggested a three point victory margin for the Conservatives on 35 per cent versus Labour on 32 per cent.     

The wisdom polling asks the respondent to predict the next government or Prime Minster.  The problem with national 'wisdom' polling, as with the mechanism of Uniform National Swing that translates vote shares into seat shares, is it may obscure critical patterns in local support at the constituency level that have developed as our two party system has fragmented.  To garner information from the local publics on this key local information, more promising are 'local MP' questions, with the hope of tapping information held by the public about their own constituency campaigns, such as the amount of activity being expended by the candidates and other rumours and political gossip, currently locked away in local networks.   

At this level, polling companies  are becoming promising conduits for the expression of mass-crowd wisdom, via their burgeoning online panels.  These are growing at such a rate, it is not difficult to envisage that in 2020 we will be able to properly test sufficient sample sizes of voters in every constituency to measure their own understanding of their own local results.  This is a hugely exciting development.  For now, hardly noticed for his presentation beyond the walls of the LSE conference, we have the research of Andreas Murr, and his pioneering 2015 constituency crowd wisdom model.  Whilst calling on only 25 voters per constituency this time around (in February), his analysis generated an estimate of 292 Conservative seats.  This prediction is remarkable for its closeness to the current betting market.  What is more, with ever growing online panels facilitating larger surveys at future elections, the accuracy of this type of crowd sourcing can only increase.  If the total of Tory seats tomorrow morning hits the 292 total (or more), then the purest and most localised form of crowd wisdom will have scored an early and notable success, not least for being higher on Conservative seats than eleven of the twelve academic models.  This may be at the start of a revolution in how we forecast not just elections but referendums and non-political events too.  In the age of the Internet and massive online opinion poll panels, we can start trusting the public more than the experts.