HOW CAN YOU TELL IF YOUR SURVEY QUESTIONS WORK? USING COGNITIVE TESTING TO IMPROVE MEASUREMENT OF WOMEN’S EMPOWERMENT IN NUTRITION AND HEALTH

In the second phase of the Gender, Agriculture, and Assets Project (GAAP2), researchers are developing the project-level Women’s Empowerment in Agriculture Index (pro-WEAI) to measure women’s empowerment in agricultural development projects. The pro-WEAI includes an optional module to measure women’s empowerment in health and nutrition decisions. In this blog, Anika Hannan, a soon-to-be MPH graduate from Emory University, discusses cognitive testing conducted in Bangladesh to strengthen the pro-WEAI nutrition and health module.

Sitakunda Upazila, Chittagong, Bangladesh (Photo credit: Anika Hannan)

The nutrition and health module of the pro-WEAI will be used to measure women’s empowerment in nutrition and health in various geographic areas and cultural contexts. So how do researchers know if women from different areas will understand the survey questions? Cognitive testing enables researchers to assess and improve a survey module before it is widely implemented, which can be an essential step in validating newly-developed tools.

What is cognitive testing?

Cognitive testing, or cognitive interviewing, is an evidence-based qualitative method for assessing whether respondents understand survey questions as the researchers intended. The method addresses four stages of cognition required for accurate survey responses: comprehension, retrieval, judgment, and response. Researchers can use cognitive testing results to identify questions that were difficult for respondents to understand or answer, and adjust them to reduce error in the survey results and maintain validity across cultural contexts.

Methodology

Cognitive testing of the pro-WEAI nutrition and health module was done in two upazilas, or sub-units of districts, in rural Bangladesh: the Sitakunda upazila in the Chittagong division and the Aditmari upazila in the Rangpur division. These upazilas are part of the area where the Agriculture, Nutrition, and Gender Linkages (ANGeL) project is being evaluated. ANGeL is one of the projects in the GAAP2 portfolio that have piloted the pro-WEAI nutrition and health module.

Respondents included 48 mothers with children under age two. Mothers were recruited and categorized into three household types: (1) husband and wife with no co-residing parents; (2) husband and wife living with the husband’s mother; (3) wife whose husband was absent due to labor migration.

What did they find?

• Comprehension: Most respondents (72 percent) understood key survey questions. Those who misinterpreted questions focused on the domain rather than the decision-making portion of the question. Most respondents also understood key concepts. Commonly misunderstood concepts included “contraceptive method” and “special foods for children” (i.e., foods such as fortified cereals and baby food bought for and given specifically to children). For example, many respondents interpreted “special foods for children” as “fruits and vegetables,” which is not what the question intended.
• Retrieval: Most respondents (79 percent) could remember decisions made during their most recent pregnancy or breastfeeding period. Respondents noted this time period was “not too long ago for them to remember.”
• Judgment: Most respondents (94 percent) stated that other women in their community would not find the questions difficult to understand. The remaining respondents thought others may have difficulty because they would not understand the question, would need more time to think, would be too shy to answer, or would not be educated enough to understand the question.
• Response: Most respondents (92 percent) did not cite difficulty or unease in answering the questions. Those who did find questions difficult said they were “unfamiliar with the question type” or “not used to thinking in terms of the question.” Additionally, some respondents (4 percent) stated that answering questions on sensitive topics like decisions about having another child or using contraception made them uncomfortable.

These findings point to areas in the pro-WEAI health and nutrition module that can be improved. For example, questions can be reworded to provide examples of concepts that respondents did not interpret as intended, such as types of contraception or examples of special foods, to help respondents better understand these questions. These results will ultimately help the pro-WEAI research team address potential errors and strengthen the survey module’s ability to evaluate nutrition-sensitive agricultural programs.

References:

• Johnson KB and Diego-Rosell P. 2015. Assessing the cognitive validity of the Women’s Empowerment in Agriculture Index instrument in the Haiti Multi-Sectoral Baseline Survey. Survey Practice 8(3).
• Malapit HJ, Sproule K, Kovarik C, Meinzen-Dick RS, et al. 2014. Measuring progress toward empowerment: Women's empowerment in agriculture index: Baseline report. Washington, DC: International Food Policy Research Institute.
• Willis GB and Miller K. 2011. Cross-cultural cognitive interviewing: Seeking comparability and enhancing understanding. Field Methods 23(4): 331-341.

Previous GNIE blogs about pro-WEAI:

• Martinez E. 2017. Progress in designing and implementing the project-level WEAI.
• Heckert J and Kim S. 2016. Developing a nutrition-sensitive pro-WEAI.
• Balagamwala M. 2016. Development of a project-level WEAI begins at the GAAP2 inception workshop.

This post is part of a blog, the Gender-Nutrition Idea Exchange, maintained by the CGIAR Research Program on Agriculture for Nutrition and Health (A4NH). To add your comments below, please register with Disqus or log in using your Facebook, Twitter, or Google accounts. You must be signed in or registered to leave a comment.