Numbers aren’t neutral

Analysing data without providing sufficient context is dangerous

Updated - January 29, 2018 07:39 am IST

Published - January 29, 2018 02:15 am IST

An inherent challenge in journalism is to meet deadlines without compromising on quality, while sticking to the word limit. However, brevity takes a toll when it comes to reporting on surveys, indexes, and big data. Let me examine three sets of stories which were based on surveys and carried prominently by this newspaper, to understand the limits of presenting data without providing comprehensive context.

Three reports

The Annual Status of Education Report (ASER), Oxfam’s report titled ‘Reward Work, Not Wealth’, and the World Bank’s ease of doing business (EoDB) rankings have been widely reported, commented on, and editorialised. In most cases, the numbers and rankings were presented as neutral evaluations; they were not seen as data originating from institutions that have political underpinnings. Data become meaningful only when the methodology of data collection is spelt out in clear terms.

Every time I read surveys, indexes, and big data, I look for at least three basic parameters to understand the numbers: the sample size, the sample questionnaire, and the methodology. The sample size used indicates the robustness of the study, the questionnaire reveals whether there are leading questions, and the methodology reveals the rigour in the study. As a reporter, there were instances where I failed to mention these details in my resolve to stick to the word limit. Those were my mistakes.

The ASER study covering specific districts in States is about children’s schooling status. It attempts to measure children’s abilities with regard to basic reading and writing. It is a significant study as it gives us an insight into some of the problems with our educational system. However, we must be aware of the fact that these figures are restricted only to the districts in which the survey was conducted. It cannot be extrapolated as a State-wide sample, nor is it fair to rank States based on how specific districts fare in the study. A news item, “Report highlights India’s digital divide” (Jan. 19, 2018), conflated these figures.

For instance, the district surveyed in Kerala was Ernakulam, which is an urban district; in West Bengal it was South 24 Parganas, a complex district that stretches from metropolitan Kolkata to remote villages at the mouth of the Bay of Bengal. How can we compare these two districts with Odisha’s Khordha, Jharkhand’s Purbi Singhbhum and Bihar’s Muzaffarpur? It could be irresistible for a reporter, who accessed the data, to paint a larger picture based on these specific numbers. But we may not learn anything when we compare oranges and apples.

Questionable methodology

Oxfam, in the ‘Reward Work, Not Wealth’ report, used a methodology that has been questioned by many economists. Inequality is calculated on the basis of “net assets”. The economists point out that in this method, the poorest are not those living with very little resources, but young professionals who own no assets and with a high educational loan. Inequality is the elephant in the room which we cannot ignore. But Oxfam’s figures seem to mimic the huge notional loss figures put out by the Comptroller and Auditor General of India. Readers should know that Oxfam’s study has drawn its figures from disparate sources such as the Global Wealth Report by Credit Suisse, the Forbes’ billionaires list, adjusting last year’s figure using the average annual U.S. Consumer Price Index inflation rate from the U.S. Bureau of Labour Statistics, the World Bank’s household survey data, and an online survey in 10 countries.

When the World Bank announced the EoDB index last year, there was euphoria in India. However, this newspaper’s editorial “ Moving up ” (Nov. 2, 2017), which looked at India’s surge in the latest World Bank ranking from the 130th position to the 100th in a year, cautioned and asked the government, which has great orators in its ranks, to be a better listener. In hindsight, this position was vindicated when the World Bank’s chief economist, Paul Romer, said that he could no longer defend the integrity of changes made to the methodology and that the Bank would recalculate the national rankings of business competitiveness going back to at least four years. Readers would have appreciated the FAQ section (“ Recalculating ease of doing business ”, Jan. 25) that explained this controversy in some detail, had it looked at India’s ranking using the old methodology.

0 / 0
Sign in to unlock member-only benefits!
  • Access 10 free stories every month
  • Save stories to read later
  • Access to comment on every story
  • Sign-up/manage your newsletter subscriptions with a single click
  • Get notified by email for early access to discounts & offers on our products
Sign in


Comments have to be in English, and in full sentences. They cannot be abusive or personal. Please abide by our community guidelines for posting your comments.

We have migrated to a new commenting platform. If you are already a registered user of The Hindu and logged in, you may continue to engage with our articles. If you do not have an account please register and login to post comments. Users can access their older comments by logging into their accounts on Vuukle.