Research Evaluation is a systematic approach to gauge the effectiveness and impact of interventions or programs. However, a notable gap exists in many evaluations – the consideration of culture¹. Cultures evolve over time and across places, moulding our perceptions of self and the world. These cultural differences also shape the way children communicate and connect in and out of school. For e.g., facial expressions and vocabulary of feelings differ across countries and states². This influence is often overlooked. 

To illustrate the significance of culture in evaluation consider this analogy: Imagine stepping into a shoe store in India and finding only UK and US shoe sizes. Strange, isn’t it? Just as a shoe store caters to different shoe sizes based on the customer, research evaluations need to cater to a participant’s context and include cultural factors. This way, evaluations fit well with the cultural contexts of the participants, making them fair and accurate.

Culturally Responsive Evaluation (CRE) involves understanding participants’ contexts, embracing cultural nuances, perspectives, and experiences, and integrating diverse factors like race, gender, ethnicity, socioeconomic status, and language into evaluation designs³. The differentiating factor between standard evaluation and culturally responsive evaluation lies in the principle of contextualisation. This strategic decision ensures evaluations align seamlessly with participants’ cultural contexts, enhancing assessment accuracy and relevance.

The Importance of Culturally Responsive Evaluation⁴

1. Promoting Equity: CRE ensures that all participants thrive in the assessment process, irrespective of their cultural background.
2. Enhanced Validity and Utilisation: By tailoring evaluation methods to cultural contexts, findings become more relevant and applicable, increasing their credibility and practical impact.
3. Co-Creation of Programs: Involving diverse stakeholders ensures programs are shaped to meet the specific cultural needs and preferences of participants, resulting in more effective and meaningful interventions.
4. Trust and Collaboration: Embracing cultural responsiveness builds trust between all parties involved, encouraging open collaboration and shared insights. This synergy enhances the overall quality and effectiveness of the evaluation process.

Creating a Culturally Responsive Evaluation for Social-Emotional Learning:

In our evolving educational landscape, a key question arises: How can large-scale standardised testing become culturally responsive? Here are ways to conduct a culturally responsive evaluation⁵:

1. Before Conducting the Evaluation:
   1. Determine the Purpose: Decide whether it needs analysis, process evaluation, or impact evaluation. Consider the region’s history, the evaluation’s rationale, and its benefits to the stakeholders.
   2. Consider the Research Team: Are there team members who understand the culture of the region? If not, have they been briefed on the context of the region?
   3. Involve Stakeholders: Engage educators, parents, and community members in crafting assessment questions aligned with cultural values.
   4. Design Inclusive Research Questions: Collaborate with evaluators, project teams, and stakeholders to design culturally relevant questions.
   5. Design with Inclusion: Account for language proficiency, communication preferences, and non-verbal cues in assessments.
   6. Champion Multilingualism: Offer assessments in native languages with interpreters to bridge language gaps.
   7. Include Contextual Elements: Tailor assessments to cultural influences using regional literature and real-life situations such as folk tales, festivals, and food. 
   8. Identify Cultural Discrepancies: Spot and rectify differences to align with cultural norms. For instance, modify questions that may assume uniform experiences across cultures.
   9. Acknowledge Emotions and Skills: Develop assessments highlighting emotions and skills for diverse learning styles. Consider, cognitive and non-cognitive social-emotional learning skills.

2. During the Evaluation:
   1. Inform Stakeholders: Communicate the purpose of the evaluation and the intended benefits for them. Allow sufficient time for introductions and rapport building. 
   2. Interpreter: In cases of language differences, involve an interpreter from the community to facilitate communication.
3. After Evaluation
   1. Analyse Data: While analysing the data, consider the research setting. Were the participants able to fully express their thoughts? Were there any external factors or individuals present during data collection that could potentially influence the answers? In the case of outliers and unexpected results, contemplate the cultural factors that might have contributed to these outcomes. 
   2. Disseminate Data: Explore various methods to disseminate the data effectively to the intended audience. For example, consider utilizing blogs, podcasts, webinars, and other means. 
   3. Reflect: Take a moment to reflect on any possible cultural influences that might have been overlooked during the evaluation process.
4. Challenges and the Gap in Evaluation: 
   1. Contextual Hurdles: Embracing cultural responsiveness requires a comprehensive approach that spans the entire assessment process, acknowledging diverse cultures.
   2. Cultural Bridging: Adapting tests to fit cultures entails meticulous examination of questions and rectifying any biases. Cultural adaptation, crafting, and critical analysis of assessment components are pivotal for an inclusive environment.
   3. Understanding the complexity of the concept: Cultural differences play a pivotal role in how a construct is conceptualized. Moreover, addressing disparities in children’s learning levels adds yet another layer of complexity. For instance,  navigating the diverse facets of SEL constructs – from cognitive to non-cognitive dimensions – may pose a unique challenge. 

Conclusion

Designed an assessment? Here’s a checklist to ensure its cultural responsiveness:

1. Is it designed in the participants’ language?
2. Does it encompass diverse cultural experiences in its examples?
3. Did you involve locals, parents, and teachers while creating the assessment?
4. Was it tested on individuals with varying learning levels?
5. Does the assessment reflect contextual elements that resonate with the cultural backgrounds of the participants?

 As we reflect on the power of cultural context in shaping evaluations let’s ask ourselves: How can we, as educators, evaluators, and learners, collaborate to create evaluations that authentically reflect the diversity of human experiences?

About the authors

1. Kushnier, L., Nadin, S., Hill, M.E., Taylor, M., Jun, S., Mushquash, C.J., Puinean, G. and Gokiert, R. (2023). Culturally responsive evaluation: A scoping review of the evaluation literature. Evaluation and Program Planning, [online] 100, p.102322. doi: https://doi.org/10.1016/j.evalprogplan.2023.102322 ↩︎
2. Frierson, H. T., Hood, S., and Hughes, G. B. “Strategies That Address Culturally Responsive Evaluation.” In J. Frechtling (ed.), The 2002 User-Friendly Handbook for Project Evaluation (pp. 63–73). Arlington, VA: National Science Foundation, 2002. ↩︎
3. Handbook of Practical Program Evaluation, Fourth Edition By Kathryn E. Newcomer, Harry P. Hatry and Joseph S. Wholey Copyright © 2015 by Kathryn E. Newcomer, Harry P. Hatry and Joseph S. Wholey ↩︎
4. https://www.nsf.gov/pubs/2002/nsf02057/nsf02057_5.pdf ↩︎
5. Frierson, H. T., Hood, S., Hughes, G. B., and Thomas, V. G. “A Guide to Conducting Culturally Responsive Evaluations.” In J. Frechtling (ed.), The 2010 User-Friendly Handbook for Project Evaluation (pp. 75–96). Arlington, VA: National Science Foundation, 2010 ↩︎