This article was first published in the Abdul Latif Jameel Poverty Action Lab blog in June 2021. 

Is it possible to evaluate social norms? How can empowerment be measured? Can we quantify agency? These are some critical questions of interest to stakeholders working in gender and development in South Asia and elsewhere. 

Gender programs frequently seek to influence these sometimes-nebulous constructs, but they can often seem difficult to measure. However, this is not always the case. At the launch of J-PAL South Asia’s Gender sector in 2019, Seema Jayachandran, co-chair of the sector, presented her and her colleagues’ work on developing a five-question women’s agency index. 

The index incorporates mixed survey methods and is designed to be short and comprehensive, making it more feasible to incorporate into a study than a longer (and thus more costly) survey. This presentation spurred keen interest from diverse stakeholders across the gender space who sought more detail on strategies to measure gender outcomes.

In response to this increased interest, the Centers for Learning on Evaluation and Results (CLEAR), hosted at J-PAL South Asia, conducted a five-day online workshop on Applying a Gender Lens to Program Evaluation in February 2021. This workshop covered content on the measurement of gender outcomes, such as gender norms, agency, attitudes, empowerment, and behaviors, in the context of impact evaluations. It also introduced participants to practical tools and best practices used by J-PAL affiliates in their research.

Workshop curriculum: Learning by doing

CLEAR/J-PAL South Asia developed this course for practitioners at non-governmental organizations, international development organizations, and foundations working within the Gender sector in South Asia, to expose them to monitoring and evaluation (M&E) concepts relevant to their work. 

This interactive workshop was structured progressively, starting with the building blocks of impact evaluations and moving into more advanced explorations of the constructs of interest. Providing techniques tailored to the specific needs of participants’ programs, this workshop brought together 25 participants from organizations such as the Initiative for What Works to Advance Women and Girls in the Economy, the Bill and Melinda Gates Foundation, EMPower Foundation, and SEWA Bharat.

The workshop started with a presentation on the importance and necessity of measuring gender outcomes. Participants were then introduced to M&E in development research and were trained in developing a robust theory of change. The lectures drew extensively on research and lessons from J-PAL’s Gender sector to examine methods used to objectively measure outcomes that are often socially-embedded and seemingly abstract: agency, empowerment, and norms. Finally, participants learned how to develop accurate and precise measurement tools for these hard-to-measure concepts, and practiced applying these concepts using an interactive case study on survey design. 

The participants also worked on an intensive group project throughout the week, applying concepts from the lectures to develop a theory of change for a gender program focused on shifting empowerment, agency, and/or norms. They  identified indicators for their outcomes and goals, and developed survey questions to measure these indicators. To develop survey questions, they applied tools and techniques introduced in lectures, such as vignettes or “ballot boxes”;  determined sources of bias that may affect their measurement; and proposed strategies to counter these biases. On the final day of the workshop, participants presented their group work and were given feedback and comments by CLEAR/J-PAL staff. 

Applying lessons to real-world work

In a follow-up survey after the workshop, participants shared their appreciation for the examples and case studies that dived deep into measurement strategies used in recent J-PAL studies on gender, such as a study evaluating the impact of a school-based gender attitude change program in Haryana, and a study on an adolescent empowerment program in Bangladesh. These concrete examples helped them to understand how theoretical concepts can be operationalized in the field. 

They also reported enjoying the group assignments, which brought together diverse perspectives and provided an opportunity to apply concepts from the lectures. On the whole, workshop participants found it very useful to explore concepts like empowerment and agency, and left with different tools to measure these constructs that they can use in their day-to-day work.

A key goal of the CLEAR Initiative, funded by the Independent Evaluation Group at the World Bank, is to develop monitoring and evaluation capacity among practitioners to advance the Sustainable Development Goals. Our hope is that building capacity on gender and measurement through such workshops will help practitioners advance SDG 5, Gender Equality. 

Similarly, this workshop helped advance the goals of J-PAL’s emerging Gender sector, which generates knowledge and insights about the measurement of gender outcomes in impact evaluations. J-PAL regional offices in Africa, Latin America, South Asia, and Southeast Asia have also piloted training and evidence synthesis content that touches on the topic of gender and measurement. More recently, a literature review on women’s agency was developed by the J-PAL Global Gender sector. 

There is a great appetite in the community of stakeholders that work on gender issues for capacity-building content on this topic, which was reflected in the positive and constructive feedback provided by workshop participants. To learn more about this topic, please see the Practical Guide on Measuring Women’s Empowerment in Impact Evaluations.