How can we know if a social programme works? How can resources be spent effectively for maximum impact? What does it take to translate evidence into actionable policies?

Policymakers and development practitioners often debate questions such as these. CLEAR/J-PAL South Asia’s Evaluating Social Programmes (ESP) course has been designed to unpack such debates and provide participants the frameworks and tools to evaluate the impact of social programmes for better decision-making. 

In a country as large and diverse as India, governments, academics, and practitioners often face a variety of challenges and pressing questions as they work towards developing effective – and cost effective – policies. 

With increasing demands from development stakeholders for greater transparency and accountability within policy or project implementation, there is an urgent need for strong monitoring and evaluation systems that measure progress and, if required, guide the changes that need to be made in order for a project’s successful completion. 

ESP enables participants to become better decision-makers by exploring how evidence-based, policy-relevant insights can be produced, understood, and used. This week-long workshop, held annually, aims to introduce key concepts on monitoring and evaluation by providing participants an in-depth understanding of the importance of measuring impact and how these evaluations can be designed in real-world settings. 

Participants are trained on theoretical underpinnings of conducting an evaluation, such as the components of a high-quality evaluation, determining sample size, mitigation strategies to tackle threats to the validity of experimental evaluations, and techniques for data analysis. They are also equipped with tools to measure and understand impact, such as needs assessments, logical frameworks, and theories of change. Participants also have the opportunity to consider the ethics and practical challenges of implementing randomised evaluations and interpreting evidence through discussions with subject matter experts and members of their cohort.

First delivered in 2005, CLEAR/J-PAL South Asia’s Evaluating Social Programmes has reached hundreds of practitioners across the region. The programme has also evolved over the years, adopting the latest approaches within the field and using more engaging pedagogy such as context-specific case study sessions, group work, and office hour discussions with CLEAR/J-PAL South Asia staff.

“I felt a course like this would give me more appreciation of some of the theoretical aspects around what kinds of decisions are needed for deciding on evaluation techniques, to be able to work with teams on oversight and quality of these evaluations, and also have more understanding on how to interpret results and apply them to our work in a cyclical fashion,” said Tannistha Datta, Child Protection Specialist at UNICEF India and participant in our ESP 2023 cohort.

The course is designed specifically for mid- to senior-level practitioners and researchers. One such senior practitioner who benefitted from ESP is Pooja Trivedi (participant in ESP 2023), the Director of Growth and Strategy at Sanmat - an organisation which focuses on bridging governance gaps in the implementation of government schemes. 

Trivedi shared that the course refreshed and further strengthened her technical skills, and gave her insights from a programmatic and implementation perspective. “When I lead programmes, I will be more confident in discussing the studies and give strong inputs. ESP helped me identify and understand the mistakes we made in the past.”

CLEAR/J-PAL South Asia also offers a scholarship for students or PhD scholars at Indian Universities, part of its larger effort to strengthen the capacity of academics and researchers in South Asia to prepare the next generation of evaluators in the region.

Ankit Raj, a Masters Student at Sawai Man Singh Medical College and the recipient of ESP 2023’s student scholarship, stated that the “systematic organisation and smooth flow of the course was another factor which I found very different from many other courses conducted in this field… The close interaction with speakers and [other] participants, especially during case studies, further strengthened my learnings.”

Another key feature of ESP is the guided practice in interpreting and using evidence through case study, group work, and office hour sessions, which allows practitioners to immediately apply their lessons to their work. 

Pranav Kothari, an ESP 2015 participant, was the Vice President at Educational Initiatives, an ed-tech firm based in India. Pranav was the project director of Mindspark, a personalised adaptive learning platform and was overseeing the development of Mindspark Centres, which sought to bring a Hindi version of the tool to urban slums in New Delhi. 

He applied to CLEAR/J-PAL South Asia’s ESP with a specific goal – to be sure that every child in a Mindspark Centre is learning – and was keen to undertake a systematic assessment of the program and its implementation to understand whether or not it worked as intended. 

Over the course of a week, Kothari had put together a basic evaluation plan for his program.

Seeing the potential in the program, J-PAL and Educational Initiatives set up a randomised evaluation in Delhi which demonstrated that the Mindspark platform led to increased learning levels across all groups of students. This has provided valuable evidence on the use of computer-adapted learning in low-resource settings as a promising supplementary tool in a post-Covid classroom.

"While we had been working on the ground for over two years, we wanted a third-party evaluation to determine whether our intervention was useful or not. This course helped us plan and conduct a randomised evaluation, allowing us to identify how well our program worked and where we could refine it. These results have led to newer versions of the program, and have helped us scale Mindspark to reach over 300,000 students, or about 500 times the number of students that were initially studying in the centers. I have also learned many skills which helped Ei in doing impact evaluations of other educational interventions.”

Pranav Kothari, Chief Executive Officer, Educational Initiatives

Impact evaluations — especially randomised evaluations — are increasingly being recognised in India and around the world as powerful tools to design programmes that are backed by robust evidence. Courses such as ESP have an important role to play for the growing movement of evidence-based policymaking to become a systemic and sustainable practice.

For more information, email us at clearsateam@clearsouthasia.org.