A visualization on statistics choices

Another lovely resource - a decision tree to help choose the appropriate statistic to apply. 

And if you want to have step-by-step instructions on how to create it in SPSS, see this SAGE resource 

A visualisation of data visualization choices!

I always love it if people can simplify the complicated decisionmaking processes that we have in our heads, to a simple decision tree. A visualisation of data visualizationn choices!

 

 

Applying systems theoretical concepts to understand sustainability of education intervention outcomes

 

This Master’s dissertation addresses the research question: To what extent can the systems concept ‘extended dynamic sustainability’ be used to explain why some results of a donor-funded education development intervention were sustained ten years after its conclusion?

To address that question, the researcher identified a specific case to explore with systems thinking: an ex-post evaluation conducted in 2016, and commissioned by an international donor, the United States Agency for International Development (USAID). That ex-post evaluation confirmed that an education development intervention, the Kimberly Thusanang Programme (KTP) implemented between 1998 and 2006, resulted in sustained outcomes, which were directly linked to the KTP’s goal of improving school governance in the Francis Baard education district the Northern Cape.

The Master’s research builds on the ex-post evaluation’s analysis. Using qualitative data analysis, the researcher identified the types of sustainability found in the ex-post evaluation data set. Then, by applying Stockmann’s (1993a) ‘extended dynamic sustainability’ concept, the Master’s research found that the KTP intervention and some of its benefits were dynamically sustained through the general causal sustainability mechanisms of problem-solving, modelling and multiplication.

These findings are likely useful to research intervention sustainability, to design sustainable development interventions, and to evaluate intervention success. Further exploration of these general sustainability mechanisms needs to be conducted to determine if these mechanisms are generalisable to other development interventions and their sustained outcomes.

Available on the University of Johannesburg Content Repository

EvalEdge Podcast about Storytelling in Evaluation

 

In this episode of EvalEdge, Asgar Bhikoo and I talk about Storytelling in Evaluation Practice. This episode focused on exploring current lessons related to the use of story-telling as innovation in evaluation practice in Africa. For more information, check out Digital Stories for Impact and Social Impact Storytelling: Using Impact Data to Drive Change.

Here is the impact story tool referenced in the podcast, also here on the Civicus repository. 

There are alternatives to Experimental and Quasi-Experimental Impact Evaluation Methods.

Some of my clients are really interested in measuring their impact. RCTs and other quasi-experiments are first on their list of suggested designs. But our repertoire of IE designs and methods have grown.

This DfID working paper says:

Most development interventions are ‘contributory causes’. They ‘work’ as part of a causal package in combination with other ‘helping factors’ such as stakeholder behaviour, related programmes and policies, institutional capacities, cultural factors or socio-economic trends. Designs and methods for IE need to be able to unpick these causal packages. 
Demonstrating that interventions cause development effects depends on theories and rules of causal inference that can support causal claims. Some of the most potentially useful approaches to causal inference are not generally known or applied in the evaluation of international development and aid. Multiple causality and configurations; and theory-based evaluation that can analyse causal mechanisms are particularly weak. There is greater understanding of counterfactual logics, the approach to causal inference that underpins experimental approaches to IE. 

Methods that I am currently interested in include 

Qualitative Impact Assessment Protocol 

The QuIP gathers evidence of a project’s impact through narrative causal statements collected directly from intended project beneficiaries. Respondents are asked to talk about the main changes in their lives over a pre-defined recall period and prompted to share what they perceive to be the main drivers of these changes, and to whom or what they attribute any change - which may well be from multiple sources.
Typically, a QuIP study involves 24 semi-structured interviews and four focus groups, conducted in the native language by highly-skilled, local researchers. However, this number is not fixed and will depend on the sampling approach used. The research team conducting interviews are independent and blindfolded where appropriate; they are not aware who has commissioned the research or which project is being assessed. This helps to mitigate and reduce pro-project and confirmation bias, as well as enable a broader and more open discussion with respondents about all outcomes and drivers of change.

Qualitative Comparative Analysis 

Qualitative Comparative Analysis (QCA) is a means of analysing the causal contribution of different conditions (e.g. aspects of an intervention and the wider context) to an outcome of interest. QCA starts with the documentation of the different configurations of conditions associated with each case of an observed outcome. These are then subject to a minimisation procedure that identifies the simplest set of conditions that can account all the observed outcomes, as well as their absence. The results are typically expressed in statements expressed in ordinary language or as Boolean algebra. QCA is able to use relatively small and simple data sets. There is no requirement to have enough cases to achieve statistical significance, although ideally there should be enough cases to potentially exhibit all the possible configurations.