Improving the craft of prediction

"Seat projections that miss the mark, as many did with Bihar, do not necessarily condemn the survey data on which they are premised.” Picture shows voters in Madhepura during the fifth phase of the Bihar elections.  

Now that the heated rhetoric, frantic campaigning, and tamasha of televised exit poll panels is behind us, it is time for introspection regarding the >Bihar election results. I do not mean introspection on the part of the winning and losing parties (though that is also warranted), but rather by the pollsters and >political analysts who spent so much time puzzling over survey data to forecast and understand the election results. What can India’s pollsters and poll consumers learn from Bihar?

Sam Solomon
I am an American scholar working closely with different Indian public opinion researchers to understand the methods. The following suggestions have been derived from what I have learned over the course of my research.

More humility and scepticism
First, the volatility of Indians’ political preferences should make any consumer of pre-polls sceptical of projections based on these polls. This is not to say that pre-polls are not important. Pre-polls capture a snapshot of electoral conditions in a State or across the country, and can, in fact, be quite useful in tracking changes in public opinion. Differences in the CSDS (Centre for the Study of Developing Societies) pre-poll results for Bihar, fielded at the end of September, and post-poll results, fielded after election days in October and November, revealed that the campaign in October hurt the National Democratic Alliance (NDA) and helped the Mahagathbandhan consolidate its vote bases. That is an insight that could not have been found without the pre-polls that serve as a benchmark for public opinion.

However, snapshots only capture a particular moment in time. They are not forecasts. Many Indians do not make up their mind until very close to election day. Thus, pre-polls are unable to measure their final voting intentions. CSDS’s post-poll for Bihar found that 14 per cent of Bihar’s voters did not make up their mind until the day of the election; another 12 per cent did not make up their mind until a day or two before voting. About a quarter (26 per cent) of voters made up their mind before the campaign started in September. As Sanjay Kumar and Pranav Gupta have written in these pages, this is not atypical of Indian elections. Building hawa (wind) is part of political parties’ election strategies. Pre-polls fielded far in advance in election, while useful as snapshots, will be less reliable as forecasts since they will miss this hawa, which clearly broke for the Mahagathbandhan in this election cycle.

Second, shifting alliance arrangements — as we saw in Bihar with the new Janata Dal (United)-Rashtriya Janata Dal-Indian National Congress Mahagathbandhan combine — make the business of predicting election results even more challenging. Seat projection models, in fact, rely on many assumptions that exist outside the framework of the survey. Professor Rajeeva Karandikar of the Chennai Mathematical Institute, who used to conduct seat projection models for CSDS, has written extensively about how he models seat projections in a piece for The Hindu Centre for Politics and Public Policy last year. When new alliances like the Mahagathbandhan surface, the modeller must make a subjective judgment about how the vote for previous alliances — the JD(U)-BJP alliance, in this case — will split in the new electoral landscape. These judgments can be well-informed by experience, but in elections like Bihar’s, where unprecedented alliances like the Mahagathbandhan emerge, they are speculative. Consumers of polls should be especially sceptical of seat projections for elections in which new alliances are contesting.

Changes to polling culture

Third, the accuracy of polls would likely be improved with more data about India’s caste demographics. This is especially in the case of Bihar, where caste was such a determinative factor in voting patterns. Pre-eminent pseophologist and politician Yogendra Yadav said as much in a Facebook post the night before the Bihar election results were counted: “Over the last two decades, almost all exit polls have over-estimated the BJP or the alliance favoured by the upper caste. This error could be anything between 2 and 4 per cent points. This is not due to any upper caste conspiracy (no media house wants to get its exit poll wrong) but due to a sampling bias built in the methodology of exit polls. When you stand outside the polling booth, the voters who agree to be interviewed tend to be more from powerful social groups.”

The issue here is that the last comprehensive caste census was conducted by the British in 1931; India’s 2011 census only provides data on the share of Scheduled Castes, Scheduled Tribes, Muslims, and Others. Within the ‘Others’ group are the upper castes, Other Backward Classes (OBC), and Extremely Backward Classes (EBC). Indian pollsters, therefore, do not have a proper estimate of the true percentage of upper castes, OBCs, and EBCs within each State, and are not able to definitively know when their surveys are over-representing the share of upper castes in their sample. If upper castes are, in fact, over-represented in survey samples, polls are more likely to miss the mark when upper castes are highly consolidated behind a party or alliance, as they were behind the Bharatiya Janata Party in the Bihar elections. This underlines how surveys of the Indian public are likely to improve in accuracy once the results from the Socio-Economic and Caste Census of 2011 are released.

Finally, the obsession of the media with seat projections obscures the real value of polls and puts unnecessary strain on the credibility of India’s polling industry. Whether greater transparency in seat projections, as Yogendra Yadav has called for many times, or an outright ban on seat projections, as Praveen Rai has called for in the pages of the Economic and Political Weekly, is a better remedy is open for debate. What should not be questioned is that credible polling agencies would benefit from increased disclosure of their methods, as well as greater public awareness that seat projections are founded on assumptions which are independent of their surveys. Perhaps the forthcoming Indian Polling Council, announced by Yashwant Deshmukh, can take up this matter. It must be underlined that seat projections that miss the mark, as many did with Bihar, do not necessarily condemn the survey data on which they are premised.

Pollsters will sometimes miss the mark. Many did in the recent Bihar elections. However, these instances should not undermine the credibility of all pollsters in the public mind. Surveys that are conducted by rigorous random sampling techniques are still the most methodical means of measuring public opinion in India. With another election behind us, pollsters now must take the opportunity to learn from their results and improve their craft. The public should also take the opportunity to learn about how to interpret polls more intelligently.

(Sam Solomon is a Fulbright-Nehru fellow associated with the Centre for the Study of Developing Socities-Lokniti)

Related Topics
This article is closed for comments.
Please Email the Editor

Printable version | May 11, 2021 11:40:09 PM |

Next Story