Every year since 2006, India's education system gets a shock from the findings of the Annual Status of Education Report (ASER), the all-India (rural) survey of the learning levels of primary school students carried out by the education advocacy group, Pratham. Yet while it might now be clear that kids are not learning enough in school, it is not entirely clear just how much they are learning, as serious questions exist about the quality of data on learning collected by both Pratham and the government.
Widely respected both nationally and internationally, the ASER survey is regarded as the best assessment of true learning levels in India. Since 2006, the survey has shown the proportion of children age 5-16 who are able to perform basic reading and mathematic tests. In 2013, less than half the children in class 8 could do simple division and less than half the children in class 5 could read two paragraphs of a class 2 text.
However when the ASER data is seen at the state-level for specific grades over time, what seem like inconsistencies begin to creep in. At the Grade 5 level, reading ability at the highest level appears to have dropped from 88% to 32% over five years in Madhya Pradesh, for instance. MP seems to have experienced a similar precipitous crash in its math achievements for class V. In 2007, Bihar had a far higher proportion of kids able to do division in Class V than Kerala. West Bengal shaved 22 percentage points between 2007 and 2008 in the number of kids in class V who could read a story. Chhattisgarh experienced similarly huge fluctuations from year to year.
Yet the experts behind ASER do not agree that this indicates that their data might have problems, or should not be used at the individual state or grade level. “The sample sizes at the state level are large. The variations that you point out have been documented in previous reports as well,” Rukmini Banerji, director of ASER Centre, explained to The Hindu. “The issue of 'learning loss' is well documented in the US - especially in the context of gaps in schooling such as summer vacations. What is also known from research done in literacy acquisition is that it takes several years of exposure to interventions to make literacy sustain over time. Clearly much more research is needed in India to understand children's learning trajectories over time,” Ms. Banerji said.
One of the most startling of these inconsistencies is the state of Tamil Nadu. Despite high levels of overall literacy and development and high enrollment in higher education, Tamil Nadu has consistently appeared at the bottom of the all-India table of ASER findings. In the latest survey, TN was the country's worst-performing state at the class V level in reading and second only to Assam at the bottom of the table in mathematics, easily surpassed by Bihar, UP, MP, Jharkhand and Chhattisgarh.
Unsurprisingly, the state all but dismisses the ASER report. “We do not agree with the findings of the ASER report,” a senior School Education Department Official said, asking not to be named. The official said that they believed surveys should be acrried out in a school environment and not at hme, as ASER does and expressed doubts about the sample. “In February we will be doing an internal study to assess learning levels in Tamil, English and Mathematics in all primary and upper primary government schools. That will give fairer picture of the learning levels,” the official said.
At the national level, the National Council of Educational Research and Training (NCERT) has been similarly sceptical of ASER's findings. “We believe that such surveys should be conducted in the school and not at home,” NCERT's spokesperson Hemant Kumar told The Hindu , adding that he would not comment on ASER specifically. The NCERT's National Achievement Survey (NAS), which measures students achievements on 20 selected questions each of a range of questions on mathematics and language, is far more comprehensive, methodologically sound, and a better measure of comprehension, Mr. Kumar added.
“Both the NAS and the ASER data have limitations, and more importantly they are not designed to be comparable, and so I think it is not useful to try to figure out which one is 'more correct',” Karthik Muralidharan, assistant professor in the department of economics at the University of California, San Digo, told The Hindu. By testing in schools, the NAS numbers could be skewed upwards because the weakest students would likely to be those away from school, Prof. Muralidharan said. “[I]t is possible that both are correct and that scores are improving at the top of the distribution (as shown by NAS), while averages are going down (as shown by ASER) because we are not paying enough attention to basic literacy and numeracy,” Prof. Muralidharan said.
(with inputs from Asha Sridhar in Chennai)
