The project-level Women’s Empowerment in Agriculture Index (pro-WEAI) is a survey-based tool that measures women’s empowerment and inclusion in agricultural development projects. In a recent discussion paper led by Kathryn Yount, Professor and Asa Griggs Candler Chair of Global Health at Emory University, a team of colleagues from Emory University and the International Food Policy Research Institute (IFPRI) assess the measurement properties of the survey questions behind pro-WEAI. Are the survey items really measuring the concepts they are intended to measure? In this blog, Yount and Jessica Heckert, a co-author and Research Fellow at IFPRI, discuss the importance of validating survey items and how this process helps to strengthen measures of women’s empowerment.
Photo credit: BRAC
A4NH: Why is it important to validate survey questions?
Heckert: It is important to validate survey questions because we need to know that we are measuring the right thing. If things aren’t measured well or there is a lot of error in the measurement, the analyses we do with those indicators won’t work out well. We might not see significant findings when they are actually there, or vice versa.
Yount: Another major benefit of validating survey questions is to confirm comparability across settings. Without a widely validated scale for measuring women’s empowerment, we have a much more difficult time making sense of the findings across studies. This challenge has been substantial in the field of women’s empowerment because people have used a wide variety of measures with different degrees of validation.
A4NH: How does the process of validating survey items help us answer research questions about nutrition, gender, and women’s empowerment?
Heckert: We have a lot of underlying theories that suggest that improving household gender dynamics or empowering women may improve the nutritional outcomes of women themselves and/or their children. There has been considerable effort to fill in this broad framework with empirical evidence. To generate that empirical evidence reliably, we need valid measures of women’s empowerment and/or other aspects of household gender dynamics.
Yount: One of the obvious benefits of Women’s Empowerment in Agriculture Index (WEAI) is the measurement of empowerment for men and women. The comparability of measurement across genders helps us understand how disparities in empowerment can be related to disparities in health and nutrition.
A4NH: What are some methods for validating survey items?
Yount: People have used a variety of techniques to validate survey items. My team uses psychometric methods, which have a home in psychology and have proliferated to other social sciences but have not fully been integrated in the work on measuring women’s empowerment. Basic descriptive methods include pairwise correlations to assess the association between conceptually similar survey questions and test-retest reliabilities to assess the stability of the measurement over short periods of time. Survey questions that are intended to reflect underlying constructs, such as women’s agency, self-efficacy, or voice, can be validated using factor analysis, which can be exploratory for new scales or confirmatory for previously validated scales. As we continue to use these methods across more contexts with more common instruments, we will hopefully get to a point where we are using more confirmatory approaches so that we can identify a set of scales that are valid in a wider range of contexts.
Heckert: Another thing that we consider is whether respondents answer questions in ways that reflect their own experiences. To examine that, we have done cognitive interviewing, a process of identifying errors in how individuals interpret and formulate responses to questions.
Yount: There also is a strong rationale for collecting qualitative data in the contexts of these surveys. Having the ability to refer to narrative interviews of women to get a sense of how they talk about their experiences in context helps us to triangulate, interpret, and understand the results of other validation approaches.
A4NH: What methodological approach did you use in your recent paper on the measurement properties of pro-WEAI?
Yount: Normally, our research group has a stepwise process for validating survey items that includes formative qualitative research, item generation, cognitive interviewing, and exploratory and confirmatory factor analysis. In this paper, however, we used item response theory (IRT) methods. We originally thought we would use factor analysis, but the pro-WEAI questionnaire contains nuanced question matrices where only women who report participating in groups, agricultural activities, or other livelihood activities are asked agency-related questions about those activities. IRT methods are well suited for these kinds of question matrices and so-called ‘partially ordered’ response options. Non-participation is treated as a nominal category that may be meaningful for understanding women’s agency or lack of agency. Thus, we used information on women’s participation in activities, groups, and organizations, as well as their reporting on agency-related items among those who participated to assess the validity of the questions.
In future work, IRT methods also can be used to create ‘short forms’ from validated ‘long forms,’ and to compare the usefulness of a unidimensional scale (indicator) for agency rather than multiple separate scales (or indicators). One goal might be to move towards a ‘women’s empowerment question bank’ that researchers and practitioners can use for impact evaluations and program monitoring. For example, with sufficient validation of instruments across contexts, we might identify a valid 50-question agency long-form and a valid and precise short-form that includes a subset of the 50 questions. We are still working toward that frontier in the measurement of women’s empowerment, but others in psychology are doing this kind of work with other constructs.
A4NH: What were the results of this analysis and how will it help to strengthen pro-WEAI?
Yount: We found that one dimension of intrinsic agency performed well – attitudes about intimate partner violence. These five questions have been used widely in the DHS and MICS. As a part of pro-WEAI, these questions cohere well and are valid across diverse contexts and projects. Questions capturing other dimensions of intrinsic and instrumental agency performed differently. Our guidance for those dimensions focuses on how to revise pro-WEAI modules and questions to better capture the experiences of women. Overall, the IRT analysis was useful for making recommendations for instrument improvement. We hope that these methods will become more routine in applied research on measuring empowerment.
Heckert: These results have given us lots of food for thought about how to strengthen the WEAI instruments moving forward. One takeaway is that we may need to revisit how response options are worded to make sure they are meaningful. Another is that we need to think more carefully about non-response and skip patterns. Who is not responding and why? This is particularly important with regard to the questions we ask about income-generating activities. Third, we have built a stronger theoretical framework of how pro-WEAI maps to different theoretical ideas of empowerment, specifically intrinsic, instrumental, and collective agency. We need to develop more measures of collective agency, for example, and be more selective about measures of instrumental agency, where we already have multiple indicators.
A4NH: How will this work influence future work on pro-WEAI or other versions of WEAI?
Heckert: Kathryn described her team’s step-by-step process for validating survey items, and we are considering what aspects of this process we can use to strengthen the family of WEAI instruments. For future work to adapt WEAI for nationally-representative surveys and/or value chains, we are definitely taking these ideas into consideration and attempting to apply them as we move forward with this portfolio of work.
Yount: I am very excited about the possibility of a collaborative study to develop a shorter, more streamlined WEAI for national monitoring. The set of items would have strong resonance with the full pro-WEAI but would be useful for monitoring national trends and programs over time. This interdisciplinary work is advancing the field in exciting ways. The combination of the expertise at IFPRI and Emory is exceptional, and I am delighted by our past work and look forward to future work together.
Heckert: I agree that good interdisciplinary collaborations require careful cultivation are really the best way to push the field forward.
Discussion papers about pro-WEAI:
Other GNIE blogs about measuring women’s empowerment:
This post is part of a blog, the Gender-Nutrition Idea Exchange, maintained by the CGIAR Research Program on Agriculture for Nutrition and Health. To add your comments below, please register with Disqus or log in using your Facebook, Twitter, or Google accounts. You must be signed in or registered in order to leave a comment.
