M&E Blog

Research vs Evaluation

2011-10-06T13:18:00.000+02:00

I found this on AEA 365 and really like the way it explains the difference between research and Evaluation

Weekly Funny - Why we will have a census in 2011

2011-08-26T17:26:00.000+02:00

From: Hayibo. Breaking news. Into lots of little pieces

Asked to stop singing and explain why the census was really taking place, a spokesperson for Zuma, Bignumbas Twala, confessed that it was actually the result of a bet between Zuma and neighboring President Robert Mugabe.

“It’s typical Presidential stuff really,” said Twala. “It starts with a silly comment, ‘like my countries bigger than yours.’

“It escalates to, ‘Well I’ve got more stashed away for my retirement in a numbered Swiss bank account’ and it just keeps going, all the way to ‘But I rule over the most people – so there!’

“And once you get there, all that’s left is to actually sit and count everybody.

“It’s not ideal, but a bet’s a bet,” he said.

Asked what was staked on the bet Twala said the winner would get to spend Father’s Day 2012 with Julius Malema – and know for sure that he really did rule over more people than his rival.

Weekly Funny - Evaluation Feedback

2011-08-12T15:29:00.000+02:00

Evaluation feedback can be a sensitive issue...

Telephone Equipment for Evaluators

2011-08-04T10:41:00.000+02:00

From time-to-time my consultancy conducts telephonic surveys and teleconferences as part of our normal evaluation work. I have been extremely impressed with the two South African companies that we bought our equipment from, and I want to share their contact details with you.

To give you an indication of why I was impressed - Within five minutes of contacting Phonatics about headsets, I had a quote, and it was delivered on the same day I paid for the equipment.

After losing our conference phone's manual, we emailed the general info@ email address on the Konftel website - 10 minutes later we received an emailed manual and someone called us to ensure that we got what we were looking for.

Knowledge Management Toolkit

2011-08-02T13:20:00.000+02:00

Knowledge Management for Health put this KM toolkit together that might be useful for Health Practitioners and those in the M&E field who are also concerned with ensuring that the "learning" from our evaluations do not get lost.

It will help those who are:

Looking for a primer on KM
Developing a KM strategy
Interested in knowledge sharing strategies
Interested in how to find knowledge and the best ways to organize it
Interested in tools to create new insights and knowledge
Interested in tools for adapting knowledge to inform and improve policy and program decision-making
Evaluating KM activities or programmes

Weekly Funny: Correlation and Causation Confusion

2011-07-28T22:13:00.000+02:00

This week's funny is again brought to you with the courtesy of xkcd. It demonstrates, quite adequately, what can go wrong in the process of interpretation of data.

Information IS (could be) beautiful!

2011-07-28T08:56:00.000+02:00

Ooh, ooh! This is so beautiful! Information is beautiful is David McCandless' blog dedicated to beautifully executed infographics.

Here is an example they picked up from the OECD better life Initiative done by Moritz Stefaner and co.

The length of the "flower petals" indicates the rating of the countries on indicators such as Housing, Income, Jobs, Community, Education, Environment, Governance, Health, Insurance, Life Satisfaction, Safety and Work Life Balance. For information about how they measure these, check out the oecd betterlife website

SAMEA Conference: 5 - 9 September 2011

2011-07-27T11:05:00.000+02:00

More details here...

Weekly Funny: Survey Results Can't be Trusted

2011-07-22T14:43:00.000+02:00

SPSS, PASW and PSPP

2011-07-21T13:10:00.000+02:00

When IBM acquired SPSS (Statistical Package for the Social Sciences) in 2009, they changed the program's name to PASW (Predictive Analytics SoftWare), but with the next version it became SPSS again. Today I read about PSPP and thought "Oh goodness, did they change the name again?" Turns out that PSPP is an open source verion of SPSS and it allows you to work in a very similar way to SPSS. This is what their website says:

PSPP is a program for statistical analysis of sampled data. It is particularly suited to the analysis and manipulation of very large data sets. In addition to statistical hypothesis tests such as t-tests, analysis of variance and non-parametric tests, PSPP can also perform linear regression and is a very powerful tool for recoding and sorting of data and for calculating metrics such as skewness and kurtosis.PSPP is designed as a Free replacement for SPSS. That is to say, it behaves as experienced SPSS users would expect, and their system files and syntax files can be used in PSPP with little or no modification, and will produce similar results.

PSPP supports numeric variables and string variables up to 32767 bytes long. Variable names may be up to 255 bytes in length. There are no artificial limits on the number of variables or cases. In a few instances, the default behaviour of PSPP differs where the developers believe enhancements are desirable or it makes sense to do so, but this can be overridden by the user if desired.

I will give it a test drive an let you know what I think!

PS. to all the "pointy-heads": In the right margin of my blog you will find a link to a repository of SPSS sample syntax!

Using Graphs in M&E

2011-07-20T09:46:00.000+02:00

(The pic above is from Edward Tufte's website - Ive always been a fan of his work on data visualization too!)

One of my colleagues found a really simple yet detailed explanation about uses of graphs. It is written my Joseph Kelly and it is focused on financial data, but still applicable to evaluators who work with quants.

Using Graphs and Visuals
to Present Financial Information
Joseph T. Kelley

This is from the intro:

We will focus on seven widely-available graphs that are easily produced by most any electronic spreadsheet. They are column graphs, bar graphs, line graphs, area graphs, pie graphs, scatter graphs, and combination graphs. Unfortunately there is no consistency in definitions for basic graphs. One writer’s bar graph is another’s column graph, etc. For clarity we will define each as we introduce them. Traditionally we report data in written form, usually by numbers arranged in tables. A properly prepared graph can report data in a visual form. Seeing a picture of data can help managers deal with the problem of too much data and too little information. Whether the need is to inform or to persuade, graphs are an efficient way to communicate because they can
• illustrate trends not obvious in a table
• make conclusions more striking
• insure maximum impact.

Graphs can be a great help not only in the presentation of information but in the analysis of data as well. This article will focus on their use in presentations to the various audiences with which the finance analyst or manager must communicate.

Enjoy!

Recall Bias in Survey Design

2011-07-18T16:03:00.000+02:00

I'm working on a survey which intends to measure whether a person's participation in a fellowship increased their research productivity (i.e. number of publications, new technologies developed and patented). At baseline the person is asked to report about their publications in the two years preceding the measurement. After two years of participation in the programme, the person is asked to reflect on their publications record since the start of the programme.

Besides the fact that publications usually have a long lead time, a recall bias may also be at play. The European Health Risk Monitoring Project explains response shift bias as follow:

Recent happenings are easier to remember but when a person is asked to recall events from the past, accuracy of the recall gets worse while time span expands. Long recall periods may have a telescoping effect on the responses. This means that events further in past are telescoped into the time frame of the question.

In my example, if the question asks if a person published a journal article in the past 2 years, the respondent might place the journal article which was published 2.5 years ago into the time frame of 2 years. Those people who do not publish regularly might be better able to provide accurate information. Those who publish frequently could potentially check their facts, but they are unlikely to do so if the survey is not seen as sufficiently important.

The EHRM recommends the following strategies for trying to address this type of bias:

The process of recall of events from the past can be helped by questionnaire design and process of interview. The order of questions in the questionnaire can help respondents to recall events from the past. Also giving some landmarks (holidays, known festivals etc.) can help to remember when some events happened. Also, use of a calendar may help a respondent to set events into the correct time frame.

Weekly Funny - Classroom Evaluations

2011-07-15T10:35:00.000+02:00

Compliments of Spiked Math

Resource: Reproductive Health Indicators Database

2011-07-13T14:55:00.000+02:00

This announcement about a very useful resource came through on SAMEA talk earlier this week.

MEASURE Evaluation Population and Reproductive Health (PRH) project launches new Family Planning/Reproductive Health Indicators Database

The Family Planning/Reproductive Health Database is an updated version of the popular two-volume Compendium of Indicators for Evaluating Reproductive Health Programs (MEASURE Evaluation, 2002).

New features include:

* a menu of the most widely used indicators for evaluating family planning/reproductive health (FP/RH) programs in developing countries

* 35 technical areas with over 420 key FP/RH indicators, including definitions, data requirements, data sources, purposes and issues

* links to more than 120 Web sites and documents containing additional FP/RH indicators

This comprehensive database aims to increase the monitoring and evaluation capacity, skills and knowledge of those who plan, implement, monitor and evaluate FP/RH programs worldwide. The database is dynamic in nature, allowing indicators and narratives to be revised as research advances and programmatic priorities adapt to changing environments.

Go to Family Planning/Reproductive Health Indicators Database

South African Consumer Databases

2011-07-11T14:45:00.000+02:00

Eighty20 is a neat consultancy that works with various databases available in South Africa to provide businesses, marketers, policy makers and developmental organisations with data-informed insights. I am subscribed to their "fact a day" service, which provides all sorts of interesting statistical trivia, but also exposes the various databases available in South Africa.

Today, their email carried an announcement about a new service called XtracT beta which apparently allows you to "crosstab anything against anything"

They say:

XtracT is the easiest way to access consumer information databases in South Africa. Just choose what interests you (demographics, psychographics, products, media, etc), and a filter if you wish, and a flexible cross-tabulation will appear.

Details about how it works can be found on the XtracT website, and they even have a short tutorial video to explain it.

In case you wondered about their logo... This t-shirt might give you a hint!

Weekly Funny: The Dunning Kruger Effect - Again

2011-07-08T11:43:00.000+02:00

It really has evaluation implications!

Evaluation Basics 101 - Involve the users in the design of your instruments

2011-07-08T00:13:00.000+02:00

Early this week, I got back from my work-related travel to Kenya, but then I ran straight into two full days of training. We planned to train the staff of a client on a new observation protocol that we developed for them to use. The new tool was based on a previous tool they had used. Before finalising the tool, we took time to discuss the tool with a small group of the staff and checked that they thought it could work. We thought the training would go well.

Drum roll...It didn't. On a scale of 0 to going well, we scored a minus 10. It felt like I had a little riot on hand when I started with "This is the new tool that we would like you to use".

Thinking about it - I should have crashed and burned in the most spectacular way. Instead, I took a moment with myself, planted a slap on my forehead, uttered a very guttural "Duh!" and mentally paged through "Evaluation Basics 101 - kindergarten version". Then I smiled, sighed, and cancelled the afternoon's training agenda. I replaced it with an activity that I introduced as: "This is the tool that we would like to workshop with you so that we can make sure that you are happy with it before you start to use it".

Some tips if ever you plan to implement a new tool (even if it is just slightly adjusted) in an organization:
1) Get everybody who will use the tool, to participate in the designing of the tool
2) Do not think that an adjustment to an already existing tool exempts you from facilitating the participatory process
3) Do not discuss the tool with only a small group from the eventual user-base. Not only will the other users who weren't consulted riot, even the ones that had their say in the small group are likely to voice their unhappiness.

When we were done, the tool looked about 80% the same as it did at the start, and they did not complain about its length, its choice of rating scale or the underlying philosophy again.

Lesson learnt. (For the second time!)

Weekly Funny: "A dog's brain is probably as effective...

2011-06-29T22:14:00.000+02:00

as the most sophisticated statistical software on the market..." Says dog house diaries.

(Click for larger pic)

Close observations of "Spikkels" and "Trompie", my resident English Springer Spaniels, provide anecdotal evidence to support this theory. The Spaniels will have to try their tricks on the other "boss person" in our household for a few days. I'm off to East-Africa for a bit of work.

ANA Results - 2011

2011-06-29T11:54:00.000+02:00

I have previously blogged about the implications of the Department of Basic Education's Annual National Assessments (ANAs) for educational evaluations. Yesterday, the grade 3, 6, and 9 results were released. The detailed report can be found on the FEDSAS website.

Add caption

Some highlights from the Statement on the Release of the Annual National Assessments Results for 2011 by Mrs Angie Motshekga, Minister of Basic Education, Union Buildings: 28 June 2011

“Towards a delivery-driven and quality education system”
Thank you for coming to this media briefing on the results of the Annual National Assessments (ANA) for 2011. These tests were written in February 2011 in the context of our concerted efforts to deliver an improved quality of basic education.

It was our intention to release the results on 29 April 2011, at the start of the new financial year, so that we could give ourselves, provinces, districts and schools ample time to analyse them carefully and take remedial steps as and where necessary. Preparing for this was a mammoth task and there were inevitable delays.

Background

We have taken an unprecedented step in the history of South Africa to test, for the very first time, nearly 6 million children on their literacy and numeracy skills in tests that have been set nationally.

This is a huge undertaking but one that is absolutely necessary to ensure we can assess what needs to be done in order to ascertain that all our learners fulfil their academic and human potential.

ANA results for 2011 inform us of many things, but in particular, that the education sector at all levels needs to focus even more on its core business – quality learning and teaching.

We’re conscious of the formidable challenges facing us. The TIMMS and PIRLS international assessments over the past decade have pointed to difficulties with the quality of literacy and numeracy in our schools.

Our own systemic assessments in 2001 and 2004 have revealed low levels of literacy and numeracy in primary schools.

The Southern and Eastern African Consortium for Monitoring Education Quality (SACMEQ) results of 2007 have shown some improvements in reading since 2003, but not in maths.

This is worrying precisely because the critical skills of literacy and numeracy are fundamental to further education and achievement in the worlds of both education and work. Many of our learners lack proper foundations in literacy and numeracy and so they struggle to progress in the system and into post-school education and training.

This is unacceptable for a nation whose democratic promise included that of education and skills development, particularly in a global world that celebrates the knowledge society and places a premium on the ability to work skilfully with words, images and numbers.

Historically, as a country and an education system, we have relied on measuring the performance of learners at the end of schooling, after twelve years. This does not allow us to comprehend deeply enough what goes on lower down in the system on a year by year basis.

Purpose of ANA

Our purpose in conducting and reporting publicly on Annual National Assessments is to continuously measure, at the primary school level, the performance of individual learners and that of classes, schools, districts, provinces and of course, of the country as a whole.

We insist on making ANA results public so that parents, schools and communities can act positively on the information, well aware of areas deserving of attention in the education of their children. The ANA results of 2011 will be our benchmark.

We will analyse and use these results to identify areas of weakness that call for improvement with regard to what learners can do and what they cannot.

For example, where assessments indicate that learners battle with fractions, we must empower our teachers to teach fractions. When our assessments show that children do not read at the level they ought to do, then we need to revisit our reading strategies.

While the ANA results inform us about individual learner performance, they also inform us about how the sector as a whole is functioning.

Going forward, ANA results will enable us to measure the impact of specific programmes and interventions to improve literacy and numeracy.

Administration of ANA

The administration of the ANA was a massive intervention. We can appreciate the scale of it when we compare the matric process involving approximately 600 000 learners with that of the ANA, which has involved nearly 6 million.

There were administrative hiccups but we will correct the stumbling blocks and continue to improve its administration.

The administration of the ANA uncovered problems within specific districts not only in terms of gaps in human and material resources, but also in terms of the support offered to schools by district officials.

ANA results for 2011

Before conducting the ANA, we said we needed to have a clear picture of the health of our public education system – positive or negative – so that we can address the weaknesses that they uncover. This we can now provide.

The results for 2011 are as follows:
In Grade 3, the national average performance in Literacy, stands at 35%. In Numeracy our learners are performing at an average of 28%. Provincial performance in these two areas is between 19% and 43%, the highest being the Western Cape, and the lowest being Mpumalanga.

In Grade 6, the national average performance in Languages is 28%. For Mathematics, the average performance is 30%. Provincial performance in these two areas ranges between 20% and 41%, the highest being the Western Cape, and the lowest being Mpumalanga.

In terms of the different levels of performance, in Grade 3, 47% of learners achieved above 35% in Literacy, and 34% of learners achieved above 35% in numeracy.

In the case of Grade 6, 30% of learners achieved above 35% in Languages, and 31% of learners achieved above 35% in Mathematics.

This performance is something that we expected given the poor performance of South African learners in recent international and local assessments. But now we have our own benchmarks against which we can set targets and move forward.

Conclusion

Together we must ensure that schools work and that quality teaching and learning takes place.

We must ensure that our children attend school every day, learn how to read and write, count and calculate, reason and debate.

Working together we can do more to create a delivery-driven quality basic education system. Only this way can we bring within reach the overarching goal of an improved quality of basic education.

Improving the quality of basic education, broadening access, achieving equity in the best interest of all children are preconditions for realising South Africa’s human resources development goals and a better life for all.

I thank you.

Read the full Statement and some reactions to this statement:

Statement by the Western Cape Education Department
News report by the Mail and Guardian Online
Statement by the largest teacher union SADTU
Statement by the official opposition

Weekly Funny: Pie Charts

2011-06-24T11:07:00.000+02:00

Pie charts are sometimes the most effective way to represent data, but sometimes, they are really useless. Dog house diaries provide an example of the latter

Launch: June 2011 Report on the Progress In Implementing The APRM In South Africa

2011-06-23T13:13:00.000+02:00

Progress in implementing the APRM in South Africa Details:

Where:	Pan African Parliament - Midrand
Date:	Tuesday 28 Jun 2011 -Tuesday 28 Jun 2011
Time:	10:00 -13:00

Event description:
The South African Institute of International Affairs (SAIIA), the Centre for Policy Studies (CPS) and the African Governance Monitoring and Advocacy Project (AfriMAP) will launch the South African APRM Monitoring Project (AMP) Report on Tuesday 28 June 2011 at the Pan African Parliament, 19 Richards Drive, Gallagher Estate, Midrand, commencing at 10:00am. The report, entitled Progress in implementing the APRM in South Africa, is the first attempt to gauge the views and opinions of civil society about the APRM and its progress in this country, while measuring the commitment levels of the government of South Africa in implementing its National Programme of Action in critical areas such as justice sector reforms; crime; corruption; political participation; public service delivery; press freedom; managing diversity; deepening democracy and overall governance, amongst other issues. The report is a culmination of a year-long collective effort among CSOs to jointly assess and analyse governance in South Africa. It finds that progress has been admirable in a few areas, but slow in several others. Details about the event here

ICT in Education: The Threat of Implementation Failure

2011-06-23T11:16:00.000+02:00

I am evaluating a few projects looking at the application of ICTs in Education.

Although my job is to measure the learning outcomes of the projects, it seems that implementation failure is a very real risk. Projects break down even before they can be logically expected to make a difference in learning outcomes. Infrastructure problems and limited skills are some of the big threats. It seems that my projects aren't the only ones dealing with these kind of implementation challenges.

Greta Björk Gudmundsdottir wrote an interesting article in the open access journal: Internationl Journal of Education And Development: Using Information and Communication Technology. The article is titled:From digital divide to digital equity: Learners’ ICT competence in four primary schools in Cape Town, South Africa. It speaks to specifically computer skills which would be necessary for ICT solutions. She says:

The potential of Information Communication Technologies (ICT) to enhance curriculum delivery can only be realised when the technologies have been well-appropriated in the school. This belief has led to an increase in government- or donor-funded projects aimed at providing ICTs to schools in disadvantaged communities. Previous research shows that even in cases where the technology is provided, educators are not effectively integrating such technologies in their pedagogical practices. This study aims at investigating the factors that affect the integration of ICTs in teaching and learning. The focus of this paper is on the domestication of ICTs in schools serving the disadvantaged communities in a developing country context. We employed a qualitative research approach to investigate domestication of ICT in the schools. Data for the study was gathered using in-depth interviews. Participants were drawn from randomly sampled schools in disadvantaged communities in the Western Cape. Results show that even though schools and educators appreciate the benefits of ICTs in their teaching and even though they are willing to adopt the technology, there are a number of factors that impede the integration of ICTs in teaching and learning.

It would make sense to build teachers' and learners' skills to work with ICT while they are required to use ICT for learning, but this may require that the projects deliberately look at ICT skills building as part of delivering the learning solutions.

How Many Days Does it Take for Respondents to Respond to Your Survey?

2011-06-20T10:38:00.000+02:00

At my consultancy we use SurveyMonkey for all our online survey needs. It is simple to use, reliable, and they are very responsive.

Their research and found that

The majority of responses to surveys using an email collector were gathered in the first few days after email invitations were sent, and
•41% of responses were collected within 1 day
•66% of responses were collected within 3 days
•80% of responses were collected within 7 days

The graph below maps the response rate against time.

The findings suggest that, under most circumstances, it would be best to wait at least seven days before starting to analyze survey responses. Sending out a reminder email after a week would probably boost the response rate somewhat.
SurveyMonkey also did some interesting analysis to answer questions like:

How Much Time are Respondents Willing to Spend on Your Survey?

Does Adding One More Question Impact Survey Completion Rate?

Go check it out!

Weekly Funny and Free Resources

2011-06-15T13:32:00.000+02:00

Today is Wednesday, but it is the end of my work week, hence the "funny" posted today. Tomorrow, 16 June, is youth day and since I am classified as youth (i.e. under 35 years of age) by the SA government, I am taking my youthful self to a destination slightly South and West of Pretoria for a couple of days. I will celebrate my freedom and remember those who sacrificed much. I will also watch the sun set over the sea, and eat lobster... and fish, and prawns...

Below is an illustration of how many "black box evaluations" are developed.It comes from a website dedicated to theory of change tools. Check it out!

http://www.researchtoaction.org/theory-of-change-useful-resources/

The DBE's Annual National Assessments

2011-06-13T15:39:00.000+02:00

The Department of Basic Education has started the implementation of the Annual National Assessments.

The biggest advantage of implementing the ANA, is that it supplements the information about education outcomes and quality currently in place in the Education SystemIn the DBE notice to all parents, the purpose of the ANA was explained as follow:

1) Teachers will use the individual results to inform their lessons plans and to give them a clear picture of where each individual child needs more attention, helping to build a more solid foundation for future learning. 2) The ANA will assist the Department to identify where the short comings are and intervene if a particular class or school does not perform to the national levels

It is unlikely that a single short test, administered at the beginning of each school year, will be more effective at providing feedback to teachers about the individual needs of learners, than the current assessments mandated by the DBE’s assessment policies. Continuous assessment policies already require teachers to test learners for this purpose, and if this information has not been used up and ‘til now, it is unlikely that instituting another assessment will make an impact in the school system. Rapid assessments have been shown[1] to be a very cost effective strategy for learner performance, but this requires frequent assessments and teachers with the capacity to analyse and use the results.

Assessments like these have been shown to be a useful accountability tool, depending on how the results are used[2] . It is unclear at this stage how exactly schools and teachers will be held accountable. The results will be shared with parents – which may or may not start a process where parents become more informed and involved in school quality issues. But, these results will have to be interpreted very carefully. A great teacher might produce poor literacy results because the learners in the school only started speaking the language of learning and teaching a year before. This is not a fault of the teacher… yet it might be very tempting to use it as a tool for blame. On the other hand, if the learner results show that there is a problem with a specific teacher or a specific school – How exactly will the DBE intervene? Will they have the support and the necessary information to intervene positively? Is it fair to only target maths and language teachers for “intervention” if poor numeracy and literacy results are found? Certainly, it will not benefit the Education system if the ANAs serve to antagonise the educators.

[1] Yeh, S.S. (2011). The Cost-Effectiveness of 22 Approaches for Raising Student Achievement. Information Age Publishing.

[2] Bruns, B.; Filmer D. and Patrinos, H.A. (2011). Making Schools Work. New Evidence on Accountability Reforms. Washington D.C, World Bank. Accessed online on 13 June 2011 at http://siteresources.worldbank.org/EDUCATION/Resources/278200-1298568319076/makingschoolswork.pdf

The weekly funny: Calculate the probability of a polar bear stealing your car

2011-06-10T11:07:00.000+02:00

(Click on pic for a larger image)

Or... if you are not into bears or thieves, calculate the probability of these three intersecting:

Some people have faulty logic, some have statistical skills, and some have no life.

The theory behind Sensemaker

2011-06-09T12:21:00.000+02:00

Yesterday I posted about Sensemaker. A discussion on the SAMEA listserve ensued. Kevin Kelly posted this:

The software (sense maker) is founded on a conceptual framework grounded in the work of Cognitive Edge (David Snowden). The software is very innovative, but not something that one can simply upload and start using. One really needs to grasp the conceptual background first. It should also be noted that the undergirding conceptual framework (Cynefin) is not specifically oriented to evaluation practice, and is developed more as a set of organisational and information management practices. I am hoping to run a one-day workshop at the SAMEA conference which looks at the use of complexity and systems concepts, and which will outline the Cynefin framework and explore its relevance and value for M&E.

I think I'll sign up for Kevin's course. I have been reading a little bit about Complexity and evaultion lately.

In case someone else is interested in reading up about specifically cynefin and more general complexity concepts I share some resource (with descriptions from publisher's websites)

Bob Williams and Hummelbrunner (Authors of the book Systems Concepts in Action: A practitioner’s Toolkit ) presented a work session at the November 2010 AEA conference where he introduced some systems tools as it relates to the evaluator’s practice

Systems Concepts in Action: A Practitioner's Toolkit explores the application of systems ideas to investigate, evaluate, and intervene in complex and messy situations. The text serves as a field guide, with each chapter representing a method for describing and analyzing; learning about; or changing and managing a challenge or set of problems. The book is the first to cover in detail such a wide range of methods from so many different parts of the systems field. The book's Introduction gives an overview of systems thinking, its origins, and its major subfields. In addition, the introductory text to each of the book's three parts provides background information on the selected methods. Systems Concepts in Action may serve as a workbook, offering a selection of tools that readers can use immediately. The approaches presented can also be investigated more profoundly, using the recommended readings provided. While these methods are not intended to serve as "recipes," they do serve as a menu of options from which to choose. Readers are invited to combine these instruments in a creative manner in order to assemble a mix that is appropriate for their own strategic needs.

Another good reference about Systems concepts I found was Johnny Morrell’s Book – Evaluation in the Face of Uncertainty.

Unexpected events during an evaluation all too often send evaluators into crisis mode. This insightful book provides a systematic framework for diagnosing, anticipating, accommodating, and reining in costs of evaluation surprises. The result is evaluation that is better from a methodological point of view, and more responsive to stakeholders. Jonathan A. Morell identifies the types of surprises that arise at different stages of a program's life cycle and that may affect different aspects of the evaluation, from stakeholder relationships to data quality, methodology, funding, deadlines, information use, and program outcomes. His analysis draws on 18 concise cases from well-known researchers in a variety of evaluation settings. Morell offers guidelines for responding effectively to surprises and for determining the risks and benefits of potential solutions.

His description about the book is here

And then Patton’s latest text (Developmental Evaluation – Applying Complexity Concepts to Enhance Innovation) also touches on complexity issues and Cynefin .

Developmental evaluation (DE) offers a powerful approach to monitoring and supporting social innovations by working in partnership with program decision makers. In this book, eminent authority Michael Quinn Patton shows how to conduct evaluations within a DE framework. Patton draws on insights about complex dynamic systems, uncertainty, nonlinearity, and emergence. He illustrates how DE can be used for a range of purposes: ongoing program development, adapting effective principles of practice to local contexts, generating innovations and taking them to scale, and facilitating rapid response in crisis situations. Students and practicing evaluators will appreciate the book's extensive case examples and stories, cartoons, clear writing style, "closer look" sidebars, and summary tables. Provided is essential guidance for making evaluations useful, practical, and credible in support of social change.

Rogers also published a nice article in 2008 in the Journal Evaluation about this

This article proposes ways to use programme theory for evaluating aspects of programmes that are complicated or complex. It argues that there are useful distinctions to be drawn between aspects that are complicated and those that are complex, and provides examples of programme theory evaluations that have usefully represented and address both of these. While complexity has been defined in varied ways in previous discussions of evaluation theory and practice, this article draws on Glouberman and Zimmerman's conceptualization of the differences between what is complicated (multiple components) and what is complex (emergent). Complicated programme theory may be used to represent interventions with multiple components, multiple agencies, multiple simultaneous causal strands and/or multiple alternative causal strands. Complex programme theory may be used to represent recursive causality (with reinforcing loops), disproportionate relationships (where at critical levels, a small change can make a big difference — a `tipping point') and emergent outcomes.

For more resources, try AEA 365

Sense Maker

2011-06-08T17:35:00.000+02:00

In a previous post, I ventured that we should start questioning the archaic. Our methods and our ways of communicating results haven't changed much over the past 10 years or so, despite new technologies and preferences.

I have posted a number of examples of interesting data visualizations, but the clip below introduces a new way of collecting information, with the help of a product called Sensemaker.

Here Irene Guijt talks about Sensemaker in the context of Evaluation.

Here is an article in the Stanford Social Innovation Review about a real life application done by Global Giving.

Data Visualization: Museum of Me

2011-06-06T11:46:00.000+02:00

Presentation of information is important to anyone that wants to make an impact with what they say. Intel dreamed up another interesting way of presenting different kinds of information.

If you have a facebook profile (and you don't mind intel punting their product a bit), why not take a walk in your own museum of you? The "Museum of Me" compiles all your Facebook information and creates a three-minute long expose about you. It could be scary... In the same way as listening to your own recorded voice could be scary. Gizmodo says that this is a reminder why you should'nt be using facebook!

Survey answers when you ask people to state the obvious

2011-06-03T08:04:00.000+02:00

You run a survey and you ask two questions which should have fairly straightforward answers:

Question 1: Are you Male / Female?

Question 2: What Colour is this?

The following comic from doghouse diaries, and the results of an actual colour survey at xkcd tells you a little about the validity of surveys...

The write-up about the "male / female" categories and the controls they tried to implement for color blindness at the xkcd blog is also something worth reading.

Evaluation Tasks

2011-06-02T11:07:00.000+02:00

I found this graphical representation of Evaluation Tasks from Better Evaluation very useful for thinking about the evaluation process.

(Click on the pic for a larger version).

In my experience the "synthesize findings across evaluations"-bit gets neglected. In my work as an evaluator contracted to many corporate donors, I am usually required to submit an evaluation report for use by the client. I often have to sign a confidentiality agreement that prohibits me from doing any formal synthesis and sharing, even if I am doing similar work for different clients. Informally, I do share from my experience, but the communication is based on my anecdotal retellings of evidence that has been integrated in a very patchy manner.I try to push and prod clients into talking to each other about common issues, but this rarely results in a formal synthesis.

It is not always feasible for the clients who commission evaluations to do this kind of synthesis. Their in-house evaluation capacity rarely includes the meta-analysis skill, and even if they contract a consultant to conduct a meta-analysis based on a variety of their own evaluations, there are some problems: Aggregating findings from a range of evaluations that do not pay attention to the possibility that a meta-analysis will be done somewhere in the future, requires a bit of a “fruit-salad approach” where apples and oranges, and even some peas and radishes, are thrown together. Another obvious problem is that donors who do not care to share the good, bad and ugly of their programs with the entire world, would be hesitant to make their evaluations available for a meta-analysis conducted by another donor.

Perhaps we require a “harmonization” effort among the corporate donors working in the same area?

Dunning-Kruger Effect and Evaluation

2011-06-01T10:31:00.000+02:00

Justin Kruger and David Dunning published a paper in the Journal of Personality and Social Psychology (1999, Vol 77, No.6, 1121 -1134) and the term "Dunning Kruger effect" was coined. This is the abstract:

People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.

Errol Morris described how the following sad story about a guy called McArthur Wheeler, inspired Dunning's scientific inquiry:

Wheeler had walked into two Pittsburgh banks and attempted to rob them in broad daylight. What made the case peculiar is that he made no visible attempt at disguise. The surveillance tapes were key to his arrest. There he is with a gun, standing in front of a teller demanding money. Yet, when arrested, Wheeler was completely disbelieving. “But I wore the juice,” he said. Apparently, he was under the deeply misguided impression that rubbing one’s face with lemon juice rendered it invisible to video cameras. If Wheeler was too stupid to be a bank robber, perhaps he was also too stupid to know that he was too stupid to be a bank robber — that is, his stupidity protected him from an awareness of his own stupidity.

What does this have to do with evaluators? All I suggest is that you should think a little about the Dunning-Kruger effect next time you ask people to rate their own competence level in a survey. You would not want to design such a survey without knowing that it is not a very smart thing to do, right?

Alternatively you might want to read an earlier post I did about it here.

Data Quality - An Evaluator's Job?

2011-05-31T10:38:00.000+02:00

Recently, P.Allison Minugh posted this question on the AEA group on LinkedIn:

I find there isn't much interest in data management, so I am curious: How important is data management to your evaluation studies, and why or why not?

My Response was:

In South Africa, the issue of data management has been consistently handled under what we call "data quality" or "information quality" specialization fields. It has become increasingly more visible at our evaluation conferences, and we are starting to develop a framework for the training and certification of information quality professionals.

Recently there was a Data Quality Conference in Pretoria, and my impression was that Data Management seems to be an IT function in the USA (with a push towards standards like ISO 8000). Here, In South Africa, it is often part of the M&E officer's job. It really is a grassroots concern - How to capture clinical data from paper records, how to make data available across clinics, how to reduce double counting, how to ensure that data collection tools are designed to enhance VRIPT (Validity, Reliability, Integrity, Precision and Timeliness), how to set up your Data Management System (Collection, Collation and Capturing, Reporting and Use) to ensure optimal quality and use.

Data Quality Assessments and Audits have become increasingly more pervasive - in especially the Health Sector (where District Health Information Systems need to produce all kinds of data for reporting on development initiatives, also to major donors like USAID) and the Education Sector (where the Educational Management Information System is used).

Some of my colleagues at FeedbackRA have recently done the "Information Quality Certified Professional" course. More info on this at: http://www.feedbackra.co.za/data-quality-qualifications/

A good book on the topic is titled "Data Quality Assessment" by Arkady Maydanchik

Values and Evaluation

2011-05-30T09:13:00.000+02:00

The AEA’s Annual Conference (Wednesday, November 2, through Saturday, November 5, 2011 in Anaheim, California) will focus on Values. eVALUation was also the topic of the last SAMEA conference in 2009.

Jennifer Greene says about this theme:

Like culture, evaluation is inherently imbued with values. Our work as evaluators intrinsically involves the process of valuing, as our charge is to make judgments about the “goodness” or the quality, merit or worth of a program. Judgments rest on criteria, which in turn reflect priorities and beliefs about what is most important. At Evaluation 2011, I would like us to take up the challenges of values and valuing in evaluation, particularly the plurality of values represented by different evaluation purposes and audiences, key evaluation questions, and quality criteria. I anticipate that greater attention to and openness in the value dimensions of our work can improve our practice, offer voice to diverse stakeholder interests, and enhance our capacity to make a difference in society.

Last week, as we celebrated Africa Day, I thought a little about what it means to be an African. This was my FB status update for the day:

I dream in a language that grew up on the African continent, my forebears shed blood, sweat and tears to help tame the land that is my home, and the spirit of Ubuntu directs my choices. In the words of Mbeki: “I am an African”.

This made me think about the philosophy of Ubuntu and how it translates into values which affect my dealings as an evaluator. Ubuntu means “I am what I am because of who we all are”

The Arch, Desmond Tutu, explained it so:

A person with Ubuntu is open and available to others, affirming of others, does not feel threatened that others are able and good, for he or she has a proper self-assurance that comes from knowing that he or she belongs in a greater whole and is diminished when others are humiliated or diminished, when others are tortured or oppressed.Ubuntu speaks particularly about the fact that you can't exist as a human being in isolation. It speaks about our interconnectedness. You can't be human all by yourself, and when you have this quality - Ubuntu - you are known for your generosity.

There is a Zulu saying: “umuntu ngumuntu ngabantu” which means: a person is a person through (other) persons” which is very different from “Cogito ergu sum” or "I think, therefore I am".

I could immediately think of five implications that Ubuntu has for evaluators:

You need to be very aware of your role and the role of others as representatives of a bigger collective. Mutual respect is of the utmost importance. This “respect” will affect the way in which you ask questions, and you must interpret people’s answers in this context. Do not be surprised if you have to go to great lengths to get people to provide constructive criticism.
When you share evaluation feedback, affirmation is very important. When you share negative findings, it must never be humiliating for an individual or a group of people.
As an evaluator, you are part of the bigger picture. You have an important role to play in a system of interconnected people, organizations and stories. If you try to be the “know-it-all external evaluation specialist” you will hit a wall. Listening and conversing, allowing people to participate in the meaning creation process, is essential.
There are many opportunities for “being generous”: If you evaluate a community based organization that takes time to answer your questions and provide you with some of their truly South-African hospitality, you might as well provide something in return. Writing up the evaluation findings in a form that they (not only the donor) can understand and use is one way. Sharing some of your technical knowledge (e.g. how to organize data, where to find a budget template, contact details of other people who work in the same field and could assist) is another way. Sometimes you might even share your evaluation tools and templates with people who did not pay for this “intellectual property”.
You have a responsibility to give back. Taking an inexperienced evaluator under your wing or volunteering your time for a good cause shows that you recognize you are where you are because others were willing to share with you. It is not uncommon for people who stay in abject poverty to share the little that they have with each other. Those who have more, probably have a responsibility to share more.

Lessons for Evaluators

2011-05-27T08:04:00.000+02:00

This week, a vicious rumour circulated that the speech below was delivered by a mayor of a large South African city.

Barrie Bramley, writes that it is, however, an unedited clip recorded by an actor for a milk advertisement.

Both the vicous rumour, and the contents of the clips have some lessons for evaluators:

1. Using big words in your reports and presentations will not hide an incoherent argument
2. Being long-winded bores your audience and delays tea
3. Sources should be double checked ALWAYS!
4. Comments made should be based on checked facts.

Have a lovely week!

22 Seems to be the Magic Number in Solving Education problems!

2011-05-26T17:17:00.000+02:00

In a previous post, I introduced the book by Stuart S. Yeh Entitled “The Cost-Effectiveness of 22 Approaches for Raising Student Achievement”.
Now the World Bank released a report entitled “Making Schools Work – New Evidence on Accountability Reforms” which is based on 22 recent impact evaluations of accountability-focused reforms in 11 developing countries. I wonder why this fascination with the number 22?

In the book (written by Barbara Burns, Deon Filmer and Harry Anthony Patrinos) they investigate strategies to address “service delivery failures” where increased spending does not lead to a concomitant change in education output (completion) or outcomes (learning). The idea is that if people in the schooling system are held accountable, things will improve.

This book focuses specifically on

three key strategies to strengthen accountability relationships in school systems—information for accountability, school-based management, and teacher incentives

and looks into how these can affect school enrolment, completion, and student learning.

Main findings about the three strategies include:

Information for accountability (for example – providing “school report cards”) seems to work, but it isn’t a solution to all the problems. Which information is shared, who it is shared with and how it is shared are important considerations which could help parents, communities and other role players identify where the weaknesses in the system is.

School based management reforms (e.g. implementing effective school governance, and school based management) are effective, but these “reforms need at least five years to bring about fundamental changes at the school level and about eight years to yield significant changes in test scores”

Teacher incentives of two kinds have been investigated: Contract teachers (where teachers are contracted on condition that they deliver certain results), and pay for performance reforms (bonuses from meeting targets) seem to be successful too, but perverse behaviours (Such as gaming, cheating or teaching to the test) are likely to abound and eventually negate the overall success of this strategy.

We’ve seen some progress in this regard in the South African schooling system: School Management and Governance training remains an important component of “whole school” development, and the implementation of the Annual National Assessments (ANA) is likely to evolve into an “information for accountability” initiative. (Also see this article about the ANA’s in the local press). Perhaps its time to take the hand of the labour unions and see how incentivising teachers can be implemented?

Cohen's d and Effect Size

2011-05-25T17:28:00.000+02:00

In my previous posting I explained the idea of significance testing. A statistically significant result does not necessarily mean that the result is practically significant. The “effect size” usually gives an indication of whether something is practically significant.

There are a couple of different ways of calculating an effect size.

r which is the correlation coefficient or R² which is the coefficient of determination
Eta squared ή²

Cohen’s d

This time, I will focus on Cohen’s d.

If you did a t-test, it’s usually a good idea to calculate cohen’s d.

Cohen's d is an appropriate effect size for the comparison between two means. It indicates the standardized difference between two means, and expresses this difference in standard deviation units. The formula for calculating d when you did a paired sample t test is:

Cohen’s d = Mean difference

Standard deviation

If you have two separate groups (in other words you conducted an independent sample t test), you use the pooled standard deviation instead of the standard deviation.

If Cohen’s d is bigger than 1, the difference between the two means is larger than one standard deviation, anything larger than 2 means that the difference is larger than two standard deviations. It is seldom that we get such big effect sizes with the kinds of programmes that I evaluate, so the following rule of thumb applies:

A d value between 0 to 0.3 is a small effect size, if it is between 0.3 and 0.6 it is a moderate effect size, and an effect size bigger than 0.6 is a large effect size.

Here is an example:

Kids wrote a grade 12 exam, then completed a programme that provides additional compensatory education, and then they rewrite the grade 12 exam. Below is a table that compares the Maths mark prior to the programme, to the Maths mark after the programme.

The result is statistically significant (see the last column, p < .000). The learners' results, on average, improved with about 9.9% (Mean difference is indicated in the “mean” column. Usually such a result is indicated as follow:

t (54) = 6.852; p < .000

To calculate Cohen’s d, we divide the mean difference by the standard deviation

d = mean difference/ standard deviation = 9.98148 / 10.70442 = 0.932

0.932 is larger than 0.6 so this can be classified as a large difference. In fact it is close to 1, which means that this programme probably helped the learners, on average, to improve their marks with about 1 standard deviation. That is amazing!

Means and p values.

2011-05-23T17:43:00.000+02:00

On comparing two groups’ means (or averages), it’s not sufficient to only compare the means –Because an average is just one statistic that summarises a whole distribution of scores.

In the picture below, the mean age at which these kids first drank alcohol, was around age 14. But there are kids who started earlier, and some who started later.

When comparing two means, it is important to determine whether the two distributions differ so much, that it is unlikely that they are both from the same bigger population.

If they differ, the null hypothesis is rejected. If they don’t differ, the alternative hypothesis is rejected.

Notice: although the means differ in B, the overlap in distributions is quite large.

Depending on the scale of the data (nominal, ordinal, interval or ratio) the properties of the distributions (normally distributed or not) and the kind of comparison that’s required (i.e. two independent groups e.g. boys and girls; or two measures for the same group e.g. average for boys before the programme, and after the programme) different statistics may be used.

Usually, we do a t test which yields a t statistic or an ANOVA which yields an F statistic, or their non-paramatric equivalents – the Mann Whitney or Kruskall Wallis test. Because it isn’t very easy to off- hand know if a t of 112 is good or bad, these statistics are converted to a p value (probability value) which indicates how probable it is that the null hypothesis is true.

If the p value is smaller than <0.05, the null hypothesis is rejected – there is only a 5% chance that the two distributions are the same.

Just look carefully at that criterion: p values of 0.5 (50%) and 0.06 (6%) are bigger than 0.05, the null hypothesis will be accepted. A p value 0.045 (4.5%), or any value such as p < 0.000, is smaller than 0.05 and would therefore mean the null hypothesis should be rejected - in other words, the two means differ statistically significantly.

A cut off of p = 0.05 is conventional, but a p of 0.1 (10%) or 0.001 (1%) is sometimes used as a cut-off criterion (depending on the likelihood of Type I and Type II errors)

A result like the one below means:
t (163) = -2.68, p < .05
The t statistic for the means calculated from two groups with 163 cases is -2.68, and is statistically significant at the 5% level.

F (2, 1015) = 111.286, < .001
The F statistic, for a sample of 1015 cases with 2 degrees of freedom (i.e. three groups) is 111.286 and is statistically significant at the 1% level.

The smaller the p value is, the happier you should be – because it means that you will have something interesting to report on!

The Evaluator and Statistics

2011-05-20T16:13:00.000+02:00

I had to calculate Cohen's d, Eta Squared and r today, but thought that would be too dreary a topic for a Friday blog post.

Instead, I found some statistics quotes here

Some of my favourites:
Torture numbers, and they'll confess to anything. ~Gregg Easterbrook

He uses statistics as a drunken man uses lampposts - for support rather than for illumination. ~Andrew Lang

Statistics can be made to prove anything - even the truth. ~Author Unknown

Evaluation of iPad for Education

2011-05-17T11:12:00.000+02:00

Reed College, in Portland Oregon, reports on their evaluation of the use of the iPad in class here. Reports on the use by students and faculty are available.

Whereas a previous evaluation was very critical of the usefulness of the Kindle DX in the class context, this report seems to support the adoption of tablets in the classroom.

The report particularly commented positively on the legibility of material on the iPad, the usability of the touch screen and the size and weight of the tablet. It was also found to be particularly useful if students wanted to switch between texts in class, and the search ability and navigation within texts was also positively evaluated.

They commented that PDF transferability was somewhat difficult, the filing system was not optimally user friendly and that the on-screen keyboard of the iPad did not efficiently support more than short comment typing. Some other concerns related to cost factors and accessibility

Better Evaluation Virtual Writeshop

2011-05-16T14:56:00.000+02:00

Irene Guijt posted this on the Pelican List serv today

Perhaps you have undertaken an evaluation on a program to mitigate climate change effects on rural people living in poverty, or one on capacity development in value chains. Or worked on participatory ways to make sense of evaluation data, or developed simple ways to integrate numbers and stories. We’d like to bring unknown experiences to the global stage for wider use.

Do you have an experience that covers many different aspects of evaluation – design, collection, sensemaking, and reporting? Did you look at different options to develop a context-sensitive approach? And has your evaluation process not yet been shared widely? If your answer is yes to these questions, then our virtual writeshop on evaluation may be of interest.

We will facilitate a virtual writeshop between May and September 2011 that will lead to around 10 focused documents to be shared globally. Participating in the writeshop will give you structured editorial support and peer review to develop a publication for the BetterEvaluation site.

For more information, including how to submit a proposal, go here.

Media and Science

2011-05-14T10:39:00.000+02:00

XKCD says it best

22 Approaches for Raising Student Achievement

2011-05-11T09:47:00.001+02:00

I’m working my way through a book by Stuart S. Yeh Entitled “The Cost-Effectiveness of 22 Approaches for Raising Student Achievement” (Also available as an ebook http://ow.ly/4R62s or on paper from www.loot.co.za see: http://ow.ly/4RRll)

The book (based on Studies in the States) concludes that:

The review of cost-effectiveness studies suggests that rapid assessment is more cost effective with regard to student achievement than comprehensive school reform (CSR), cross-age tutoring, computer-assisted instruction, a longer school day, increases in teacher education, teacher experience or teacher salaries, summer school, more rigorous math classes, value-added teacher assessment, class size reduction, a 10% increase in per pupil expenditure, full-day kindergarten, Head Start (preschool), high-standards exit exams, National Board for Professional Teaching Standards (NBPTS) certification, higher teacher licensure test scores, high-quality preschool, an additional school year, voucher programs, or charter schools

.

I find this interesting, because this makes a very compelling argument for using computer based learning in schools. The first chapter of the book presents a nice theoretical overview that indicates how kids become disheartened if they don’t consistently have mastery experiences. If assessment and teaching can be individualized so that each learner progressively improves compared to their previous performance (rather than a comparison with peers) they are likely to feel that they are in control of their learning, and they would be more likely to stay engaged in the learning process. The chapter states:

A theory of learning may be deduced: Individualization of assessment, task difficulty and performance expectations for each student on a daily basis, in combination with performance feedback, autonomy in task execution and an accelerating standard of performance, ensures that students achieve success and feel successful on a daily basis, fostering student engagement, increased effort, and further improvements in achievement in a virtuous cycle.

Software can automate the provision of corrective feedback, and assigning of content (on a daily basis) and this has been shown to have powerful effects on learning and achievement. Yeh reports that in a study of Math Assessment, (involving 1,880 students in grades 2 through 8, 80 classrooms and seven states) they found an effect size of 0.324 Standard deviations over a 7 month period. This is a huge increase in learner performance.

Of course we need to remember that this is based on the American school system that is very different from ours. And note the study is about "Cost Effectiveness". It is not saying that the other strategies are not effective - In their context the other strategies were effective but at a higher cost than Rapid Assessment

Of course there are a few other assumptions that will have to be checked if a similar intervention is implemented in South Africa:
1) Computer infrastructure, technical support, and teacher’s abilities must be supportive of successful implementation. It is a lot harder to implement an ICT based project than one might imagine at the outset.
2) Rapid assessment cannot replace the role of the teacher – It can help learners improve, but they still have to get quality tuition from a qualified teacher.
3) Learners’ ability to interact with the software must not be blocked by poor reading / language capabilities.

GIS Data and Maps to Find South African Health Facilities

2011-05-10T12:01:00.000+02:00

Here is a neat resource I came across at the ESI Data Quality Conference held in March.

It contains some basic data and various layers of health facility data in South Africa, and allows easy map sharing. Check it out at:

http://www.mapsharing.org.za:8008/mapguide/fusion/templates/mapguide/slate/

Some Educational ICT solutions I've come across

2011-05-09T18:07:00.000+02:00

VITALmaths which is a collection of video clips that's accessible from cellphone. They demonstrate basic maths concepts.

LearnThings which provides online learning resources for Maths and Science teaching.

Crocodile Clips a variety of maths games.

Connexions a repository of online teaching and learning resources.

Master Maths M2 Computer Based Training proprietary software for self paced Maths Learning

NovaNet and Success Maker proprietary software from the USA for self paced learning available through a South African distributor

Inspiring Read about South African Issues

2011-05-09T16:03:00.000+02:00

FAIL!!!!!

2011-05-06T09:54:00.000+02:00

Tools to create infographics.

2011-04-29T12:14:00.000+02:00

This post from Fast company provides an overview of some tools for creating info graphics.

They discuss:
Many Eyes
Which allows you to visually represent some data sets that they have available, or allows you to upload your own to play with.

Google Public Data Explorer
Which is a public version of one of Google's research tools.

Hohli
Which helps you to create and customzie Venn Diagrams. Hohli also allows you to create other charts, including scatter plots and other line charts.

Wordle
Although this tool describes itself as a "toy" for generating word clouds, it can be an effective service to spruce up your work.

Visual.ly
Is a new tool (still being tested), that will allow you to create and share infographics. From a first look on YouTube, this new service will be a great resource to create a compelling storytelling visualization. A youtube clip explains it all

Groups of Teachers building their own capacity?

2011-04-29T12:13:00.000+02:00

A client requires an evaluation of a teacher development initiative. This initiative aims to establish a teacher network which brings teachers from different schools together. It is hoped that the teachers will develop their own capacity with the help of a facilitator that has some content to share.

Rogers & Funnell (2010 - Purposeful Programme Theory, Jossey Bass) introduced me to some network theory, and they also provide a representation of a community capacity building programme "archetype". This archetype sets out the steps which must occur for this kind of programme to work. The steps are: (Not necessarily in a linear order).

1. Community develops a better understanding of issues,
opportunities, and challenges that it can address and potential
projects, activities, or processes through which to address them.

They then mobilise their human capital, social capital, institutional capital, economic capital and natural capital, which may result in:

2. Community developing an awareness and understanding of one or
more elements of its existing capacity.

3. Community develops a better understanding of the relevance of
its existing capacity to take up opportunities, projects, and
challenges, what further capacity is required, and who requires it.
write-up of Network Theory that I found particularly useful

4. Community identifies and undertakes activities, processes, and
projects that successfully develop required capacity

5. Community taps into and applies existing and/or newly
developed capacity to address challenges and seize opportunities

6. Community identifies how it can sustain and enhance its
capacity and looks for new opportunities to apply capacity

7. Stronger Communities:
Enhanced and maintained well-being of communities

The evaluation sets out to look for proof that the teacher development worked: by checking if the teachers' practices have changed, and looking for evidence that the learners have benefited. I'm making the argument that the result of establishing a sustainable network should also be looked at separately - as a different category of results.

To evaluate the network, we'll have to use social network analysis methods. An AEA LinkedIn discussion alerted me to a programme called NODEXL (A free plug-in for Excel)
which I intend to try out.

This tutorial from the "Findings group" explains some basic concepts in social network analysis.

All that's left to do, is to apply these new things and to explain to colleagues that Social Network Analysis has nothing to do with Facebook or Twitter!

Using Dashboards to communicate findings

2011-04-11T10:49:00.000+02:00

Shaku Atre says:
"The fundamental premise of business intelligence has traditionally been “to provide the right information to the right people at the right time and at the right cost.” While this statement is irrefutable, it would be more accurate if we changed the word “information” to “actionable information.”

This sounds quite similar to the UFE focus of ensuring that evaluation findings are available to the intended users for the intended use!

In communicating results in a useful manner, this resource http://ow.ly/4xlQn from Information Management shares some interesting ideas about Dashboards.

Hypothetical Crucial Confrontations

2011-04-08T19:16:00.000+02:00

Every Utilization Focused Evaluator knows: If an evaluation deliverable is submitted late, it can potentially totally negate the reason for doing the evaluation in the first place. If the intended user does not have the information at his/her disposal when the time for the intended use comes, you have a disaster.

Late delivery and a whole range of unpleasant consequences can be prevented, if you have the skill of holding people accountable for broken promises along the way. I find holding my team accountable is relatively easy, but holding a client accountable for broken commitments, is quite a different ballgame- A very unpleasant and daunting one!

Imagine the following totally hypothetical* example: You are working with a client who commits to giving feedback on deliverables by a certain date, and by the time you get to that date, there is no feedback. Or no consequential feedback. Then you try to get sign-off on deliverables, but a second and third round of comments follow, and it takes forever to move along. Hypothetically speaking, it could take you 8 months to get sign off on one deliverable!

It could be really difficult to work effectively with such a (hypotehtical) client, and you may start to doubt your own ability to execute. I cringe if I look at it from the hypothetical client's perspective - and I'm not sure what this must look like to a hypothetical innocent bystander.

Evaluation team leaders need to be able to hold their clients accountable for broken commitments, else the team may be heading for a deep and dark place where everyone just ends up hating working together.

I'm reading a book called "Crucial Confrontations" on exactly this topic. It is packed with so many interesting concepts. I can highly recommend it. See: http://ow.ly/4waEW (Also available on Kindle!)

The book has taught me to be a little bit more discerning about the actual problem that needs to be addressed (amongst other things). Who would've thought that what seems like a simple (hypothetical) problem can actually be so very complex?

1) If the problem is a single instance of not receiving feedback all you need to do is follow up with the client until he / she has met his / her commitment. Getting the feedback once, means the problem is resolved.

2) If the problem is a pattern of broken commitments, just dogging the client until he/ she provides feedback will probably only resolve the problem until the next round of feedback is due. Perhaps the process of feedback and signoff should be changed, to break this pattern?

3) But the problem may be a relationship problem. If the client is in a difficult position in his / her organization he / she may be unable to act quickly / definitively / at all. No amount of process change will solve this quandary. If a pattern of broken commitments have lead to tensions in the relationship between the client and the evaluator, interacting is likely to become more difficult as time goes by, and more commitments will be broken. The solution that is required, is a fix for the relationship!

The book has so many other good suggestions, which would be very handy in planning interactions with a hypothetical problem client. May you never have the need of applying these skills!

**********************************************************************
*My husband says that when people say "theoretically" they mean "not really". When I say hypothetically, I mean exactly that!

AEA 365

2011-03-31T15:10:00.000+02:00

The American Evaluation Association has this AEA365 service, which is really cool. I love what they have done with it. I only today managed to look at it in detail and its now my new favourite resource to recommend to new evaluators.

See www.eval.org aea365 Tip-a-Day Email Alerts: We're highlighting hot tips, cool tricks, rad resources, and lessons learned for and from evaluators Evaluation Tip-a-Day Emails

In 2007 I actually wrote a concept note for a service like this when I was at an AfrEA workshop in Niamey, Niger. Subsequently I mentioned it at the SAMEA conference as a potential service and everyboy thought it was a good idea. But I just did not have the resources at my disposal to make it happen. During that time I was invovled in running the operations of SAMEA and later AFREA so all my volunteer time and inspiration was swallowed up there. And although I thought that graduate students might be best placed to start compiling the content, I just never could find someone that would be willing to drive such an idea. So I am soooooooo happy that AEA had the same smart idea and were actually able to pull together the resources to make it a reality.

Categorizing Educational Qualifications

2011-03-31T14:46:00.000+02:00

I had to look at a survey for a colleague. She was interested in getting information about the education and qualifications of survey respondents and just put in an open-ended question.

Bad idea! Why? Have you ever tried to afterwards classify people's qualifications into coding categories? I'm sure it can be done but its much better to give the people your classification categories, and have them select the appropriate options.

Before we get to the classification question, decide whether you only want to know what a person's highest qualification is, or whether you want to know which combination of qualifications a person holds. Usually we ask for highest only, but this depends on your research interest.

In South Africa we have SAQA (the South African Qualifications Authority) that provides the qualification framework and they are the best people to consult if you want a detailed qualifications framework applicable in South Africa. We seldom want this, because most people don't know what the difference between and NQF level 3 and 4 qualification is. A cool trick is to see what Stats SA uses. This is from an old General Household Survey questionnaire:

Note to users
This question is applicable to all household members. The enumerators are instructed that it is only those qualifications already obtained which must be entered. That means the current level, whereby a person is still busy with is not applicable. It is very important to complete each record even if the person has not attended school. Moreover the enumerators are instructed that diploma and certificates must be of at least six months duration.

Universe
All members of the household in the selected dwelling.

Final code list
0 = NO SCHOOLING
1 = GRADE R/0
2 = SUB A/GRADE 1
3 = SUB B/GRADE 2
4 = GRADE 3/STANDARD 1
5 = GRADE 4/STANDARD 2
6 = GRADE 5/STANDARD 3
7 = GRADE 6/STANDARD 4
8 = GRADE 7/STANDARD 5
9= GRADE 8/STANDARD 6/FORM 1
10 = GRADE 9/STANDARD 7/FORM 2
11 = GRADE 10/STANDARD 8/FORM 3
12 = GRADE 11/STANDARD 9/FORM 4
13 = GRADE 12/STANDARD 10/FORM 5/MATRIC
14 = NTC l
15 = NTC II
16 = NTC III
17 = DIPLOMA/CERTIFICATE WITH LESS THAN GRADE 12/STD 10
18 = DIPLOMA/CERTIFICATE WITH GRADE 12/STD 10
19 = DEGREE
20 = POSTGRADUATE DEGREE OR DIPLOMA
21 = OTHER (specify in column)
22 = DON'T KNOW
99 =UNSPECIFIED

This would unfortunately not work for my colleague because she is doing a survey with respondents who trained all over the world.

The international Equivalent of the SAQA standards is ISCED - The International Standard Classification of Education. It is an UNESCO standard and the last revision was in 1997.

ISCED provides an integrated and consistent statistical framework for the collection and reporting of internationally comparable education statistics. It contains two components:

a statistical framework for the comprehensive statistical description of national education and learning systems along a set of variables that are of key interest to policy makers in international educational comparisons; and

a methodology that translates national educational programmes into an internationally comparable set of categories for (i) the levels of education; and (ii) the fields of education.

Find ISCED here:
http://www.unesco.org/education/information/nfsunesco/doc/isced_1997.htm

Communicating Data

2011-03-25T11:24:00.000+02:00

This week, I sat in a two day information quality conference. One of the key points was that data is not necessarily being used to improve service delivery, because (very simplistically)

* data quality is problematic, and
* we don't have people who can tell the story of the data so that others can relate to it.

I've already decided that my new thing for this year will be to think about communicating data. So Im seraching for good examples of visualization methods.

A friend sent this interesting example along:
http://www.guardian.co.uk/world/interactive/2011/mar/22/middle-east-protest-interactive-timeline

And another friend alerted me to this website:
http://www.informationisbeautiful.net/

"Development "

2011-03-25T10:09:00.000+02:00

Aid on the edge posted a little satirical cartoon that made me reflect on the value of "development" as we do it these days.
See: http://aidontheedge.info/2011/03/23/there-you-go/

But that is only one side of the coin:

A friend once told me that he worked in development in Africa, and admitted to forcing a Western idea of education into a local village many years ago. The contemporary wisdom is that one should rather embrace indigenous knowledge systems and community structures. When he went back many years later, he wanted to "repent" of his youthful folly.

When he got there, there wasn't much left of the village. Sadly climate change has wreaked havoc and the village's agricultural lifestyle could not longer be sustained. If it wasn't for the "out of place" education initiative that my friend started many years earlier, there would have been no livelihood for the villagers. At least the kids who got educated, got out. They were now earning a living in another way, sending back some money.

I'm not suggesting that forcing education was the right thing to do, but we can also not blame the education and local development for climate change that is happening on a global scale and would have had an effect irrespective of whether development came to that corner of Africa or not.

Specificity and Sensitivity in tests

2011-03-07T15:53:00.000+02:00

You are required to identify kids in need of remediation using a scholastic ability test.

If your test is highly specific, a low score will be able to identify everyone that requires remediation. - A lack of specificity indicates that some kids who require remediation are not identified.

If your test is highly sensitive, then a high score will clearly exclude anyone that does not need remediation.

SPIN and SNOUT are commonly used mnemonics which helps to remind us ofthe disticntion: A highly SPecific test, when Positive, rules IN disease (SP-P-IN), and a highly 'SeNsitive' test, when Negative rules OUT disease (SN-N-OUT)

Questioning the archaic...

2011-03-04T17:06:00.000+02:00

We had a debate in our office the other day as to the proper use of spacing after a fullstop. Some of my colleagues insisted that double spacing after a fullstop was the proper way to type whilst others insisted on single spacing. A couple of opinion polls later, I started checking some style manuals and the opinion of the typographers. The jury is not out on this anymore - If you type on a modern computer, single space is what you should use.

Apparently people who type two spaces, were taught by people who were taught by people who learned to type on typewriters that only allowed monospacing - i.e. an "l" and an "m", despite being different in size, was given the same amount of space, because typewriters couldn't work differently. This resulted in lots of white space in the middle of words, hence the need for double spacing between senences. With the introduction of computers, almost all texts are now created in proportional fonts - so the narrower characters take less space, and you don't need a double space after a full stop.

But this got me thinking - How much of what we do as evaluators and researchers do we do just because we were taught by people who had to make use of old archaic technology to get the job done? I mean, think about it - Why do we still insist that the primary output from an evaluation should be a report? Or... hold on to your seat... a PowerPoint presentation?

If use of evaluations (or information) depends on the degree to which the findings are communicated concisely, then we should be building our communications capability and get creative. If you look at the capabilities that simple Mac Software like Keynote offers (and I'm no expert) then really! There is so much more that we should be doing

I share with you three examples of what I would like to see more of:

Hans Roslin inspires with visualization of statistics (This guy rocks!):
http://ow.ly/47WDV

This "Story of Stuff" clip combines presentation, story telling and animation in really interesting ways
http://ow.ly/47WIh

And Here is how a normal presentation with some voice can help to get the message across, a lot better than just a PowerPoint presentation
http://ow.ly/47WOA

Survey Design

2011-02-10T11:41:00.000+02:00

I'm working on a retrospective pre-post competency survey, and needed to be reminded of some basics of survey design again.

I find Neuman's chapter about survey design a good foundation: http://www.amazon.com/gp/product/0205457932.

Jane Davidson Makes some compelling arguments that suggest that we do need to think twice when we decide to make use of Likert type scales in surveys. http://genuineevaluation.com/breaking-out-of-the-likert-scale-trap/
She sugggests that rather than use the "strongly agree / disagree" type anchors, one could use evaluative terms like "inadequate / good" that might make the data easier to interpret.

Ive also found a list of possible Likert Scale Anchors that are most useful:
http://www.hehd.clemson.edu/prtm/trmcenter/scale.pdf

Why Competency Self Assessments are essentially flawed beyond redemption

2010-07-02T13:25:00.000+02:00

I'm working on an evaluation to determine if a training programme for senior government managers makes a difference - In the competence level of the managers, and in the service delivery they are able to produce within their work context. We are tracking a wide evidence base about all of the participants, but the client is insistent that a competency self-assessment be included. We agreed, on the condition that this one piece of evidence will be used together with all of the other evidence we will be collecting throughout the study. The value that the competency self-assessment will add, is something we have debated in the team. The following entertaining post by Errol Morris, however, pretty much sums it all up:

http://opinionator.blogs.nytimes.com/2010/06/20/the-anosognosics-dilemma-1/

David Dunning, a Cornell professor of social psychology... wondered whether it was possible to measure one’s self-assessed level of competence against something a little more objective — say, actual competence. Within weeks, he and his graduate student, Justin Kruger, had organized a program of research. Their paper, “Unskilled and Unaware of It: How Difficulties of Recognizing One’s Own Incompetence Lead to Inflated Self-assessments,” was published in 1999.

Dunning and Kruger argued in their paper, “When people are incompetent in the strategies they adopt to achieve success and satisfaction, they suffer a dual burden: Not only do they reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the ability to realize it. Instead, …they are left with the erroneous impression they are doing just fine.”
It became known as the Dunning-Kruger Effect — our incompetence masks our ability to recognize our incompetence.

There have been many psychological studies that tell us what we see and what we hear is shaped by our preferences, our wishes, our fears, our desires and so forth. We literally see the world the way we want to see it. But the Dunning-Kruger effect suggests that there is a problem beyond that. Even if you are just the most honest, impartial person that you could be, you would still have a problem — namely, when your knowledge or expertise is imperfect, you really don’t know it. Left to your own devices, you just don’t know it. We’re not very good at knowing what we don’t know

In logical reasoning, in parenting, in management, problem solving, the skills you use to produce the right answer are exactly the same skills you use to evaluate the answer

Can a song cause someone's death?

2010-04-08T10:28:00.000+02:00

A quantitative evaluator, a qualitative evaluator, and a normal person are waiting for a bus. The normal person suddenly shouts, "Watch out, the bus is out of control and heading right for us! We will surely be killed!"
Without looking up from his newspaper, the quantitative evaluator calmly responds, "That is an awfully strong causal claim you are making. There is anecdotal evidence to suggest that buses can kill people, but the research does not bear this out. People ride buses all the time and they are rarely killed by them. The correlation between riding buses and being killed by them is very nearly zero. I defy you to produce any credible evidence that buses pose a significant danger. It would really be an extraordinary thing if we were killed by a bus. I wouldn't worry."

Dismayed, the normal person starts gesticulating and shouting, "But there is a bus! A particular bus! That bus! And it is heading directly toward some particular people! Us! And I am quite certain that it will hit us, and if it hits us it will undoubtedly kill us!" At this point the qualitative evaluator, who was observing this exchange from a safe distance, interjects, "What exactly do you mean by bus? After all, we all construct our own understanding of that very fluid concept. For some, the bus is a mere machine, for others it is what connects them to their work, their school, the ones they love. I mean, have you ever sat down and really considered the bus-ness of it all? It is quite immense, I assure you. I hope I am not being too forward, but may I be a critical friend for just a moment? I don't think you've really thought this whole bus thing out. It would be a pity to go about pushing the sort of simple linear logic that connects something as conceptually complex as a bus to an outcome as one dimensional as death."

Very dismayed, the normal person runs away screaming, the bus collides with the quantitative and qualitative evaluators, and it kills both instantly...Very, very dismayed, the normal person begins pleading with a bystander, "I told them the bus would kill them. The bus did kill them. I feel awful. To which the bystander replies, "Tut tut, my good man. I am a statistician and I can tell you for a fact that with a sample size of 2 and no proper control group, how could we possibly conclude that it was the bus that did them in?"

* On that note, however, I reject all the talk that the singing of a song CAUSED the death of a man. It may have been a contributor, but I doubt it was either a necessary or sufficient circumstance for this specific death!

http://www.news24.com/SouthAfrica/News/Spike-in-race-tensions-institute-20100406

A lonely brainstorm... Or many minds?

2009-11-26T23:04:00.000+02:00

A grantmaking organization (our client) is interested in evaluating the level of their service delivery and relationship management – as perceived by the grantees that they disburse funds to. So here is the question – What are the evaluation standards that we should use?

Grantee perceptions?
The terms of reference indicates that the client expects that the evaluators will interact with the grantees to answer their questions. But if we ask grantees what they think of the grant maker’s processes, approach, involvement, communication etc. we might get senseless data because the wide range of grantees will have very different expectations about what qualifies as good service delivery / relationship management. It will probably be easy to collect data about their perceptions, but that won’t be very useful. And then there is also the issue of possible bias: Those grantees that experienced difficulty in submitting reports etc for monitoring purposes, might actually be slightly more negative than the rest of the grantees that would probably be eager to be complement the people that will dish out their next pay check.

The Grantmakers’ own standards?
It might make sense to determine whether the grantmaker has any implicit or explicit service delivery standards or contracted agreements that could be used as the standard to evaluate their performance against. But if the grantmaker has a standard that says: “All applications must be acknowledged in writing within 6 months from the date of receipt” that would be easy to check, but surely that service standard seems a little odd? Does it really take six months to respond to a submission?

Industry standards and benchmarks?
The alternative would be to look at service delivery standards and benchmarks as set by other industry players. There’s lots of literature about grantmaking internationally, but information about South African grantmakers are limited – There is the CSI handbook, but it doesn’t contain the level of detail that may be required to develop an extensive set of evaluation standards and benchmarks. And grant makers are notoriously secretive about their approach, systems and quality standards, so we will probably not be able to get detailed information from more than a handful of players in the field that we have established past relationships with.

Room for a participatory agreement on what exactly should be measured?
It is possible that a rigorous engagement of grantees and grant makers at the outset of the evaluation could provide the most satisfactory solution to the “which standards should we use” question. And that is probably just what we will do! Background research about all of the above will probably provide a good basis to start the workshop, but it will be interesting to see what the final consensus will dictate!

Visualization Methods - Really really interesting

2009-10-26T10:23:00.000+02:00

Previously I wrote about Edward Tufte's Book on presenting graphs. Well, it seems that data visualization has been taken to a whole new level.

Ralph Lengler & Martin J. Eppler form the Institute of Corporate Communication compiled a "Periodic table" of visualization methods that categorizes and shows examples of about 100 visualization methods.

The table can be downloaded in pdf format at:
http://www.visual-literacy.org/periodic_table/periodic_table_as_pdf.pdf

But try the online version - As you mouse over the various "elements" an example pops up to demonstrate what it looks like.
http://ow.ly/v9RI

The full article explaining the table can be found at
http://ow.ly/wk7d

Wow - It takes people specializing in visualization methods to think of such an innovative way to present their concept.

PS. I heard about this on the American Evaluation Assocation's Linked in Group.This and other useful information gets shared from time to time.

Get involved - SAMEA is preparing a submission

2009-10-12T17:13:00.000+02:00

Media statement by the Minister in the Presidency T Manuel for National Planning on the release of the Green Paper on National Strategic Planning
4 September 2009

Today government is releasing two discussion documents, one a Green Paper on National Strategic Planning and the other a Policy Document on Performance Monitoring and Evaluation. The decision by President Zuma to appoint Ministers in the Presidency responsible for National Planning and Performance Monitoring and Evaluation is designed to improve the overall effectiveness of government, enabling government to better meets its development objectives in both the short- and longer-term. These two discussion documents must be seen in the context of wider efforts led by the President to improve the performance of government through enhancing coherence and co-ordination in government, managing the performance of the state and communicating better with the public.
The Green Paper on National Strategic Planning is a discussion document that outlines the tasks of the national planning function, broadly defined. It deals with the concept of national strategic planning, as well as processes and structures. Once consultations on these issues have been completed, the process to set up the high-level structures will commence; and this will be followed by intense work to develop South Africa's long-term vision and other outputs. In other words, the Green Paper does not deal with these substantive issues of content.
The rationale for planning is that government (and indeed the nation at large) requires a longer-term perspective to enhance policy coherence and to help guide shorter term policy trade-offs. The development of a long-term plan for the country will help government departments and entities across all the spheres of government to develop programmes and operational plans to meet society’s broader developmental objectives. Such a plan must articulate the type of society we seek to create and outline the path towards a more inclusive society where the fruits of development benefit all South Africans, particularly the poor.
The planning function is to be coordinated by the Minister in The Presidency for National Planning. There are four key outputs of the planning function. Firstly, to develop a long term vision for South Africa, Vision 2025, which would be an articulation of our national aspirations regarding the society we seek and which would help us confront the key challenges and trade-offs required to achieve those goals. A National Planning Commission comprising of external commissioners who are experts in relevant fields would play a key role in developing this plan. The development of a National Plan would require broader societal consultation and existing forums would be used for this purpose. The Minister in The Presidency will co-ordinate these engagements. A National Plan has to be adopted by Cabinet for it to have the force of a government plan. The Minister would serve as a link between the Commission and Government, feeding the work of the Commission into government.
The next set of outputs cover the five-yearly Medium Term Strategic Framework (MTSF) and the National Programme of Action. These are documents of national government, adopted by Cabinet, drawing on the electoral mandate of the government of the day. The Minister in The Presidency for National Planning, supported by a Ministerial Committee on Planning, would coordinate the development of these documents with input from Ministers, departments, provinces, organised local government, public entities and coordinating clusters.
Further, it is envisaged that the planning function in The Presidency will undertake research and release discussion papers on a range of topics that impact on long-term development. These include topics such as demographic trends, global climate change, human resource development, and future energy mix and food security. The Presidency would also release and process baseline data on critical such as demographics, biodiversity as well as migratory and economic trends. This work will be undertaken by the Minister, working with the National Planning Commission (NPC) and the Minister, working with the NPC would, from time to time, advise government on progress in implementing the national plan, including the identification of institutional and other blockages to its implementation.
One of the functions of The Presidency in respect of national planning is to develop frameworks for spatial planning that seek to undo the damage that apartheid's spatial development patterns have wrought on our society. This includes the development of high level frameworks to guide regional planning and infrastructure investment.
The national planning function will provide guidance on the allocation of resources and in the development of departmental, sectoral, provincial and municipal plans.
The Minister in The Presidency responsible for national planning will be supported by a Planning Secretariat, which will also provide administrative, research and other support to the National Planning Commission. National Strategic Planning is an iterative process involving extensive consultation and engagement within government and with broader society.
It is envisaged that Parliament will play a key role in guiding the planning function through its oversight role but also through facilitating broader stakeholder input into the planning process. For this reason, it is appropriate that Parliament should lead the discussion process on the Green Paper.
This Green Paper is a discussion document. Government welcomes comment, advice, criticisms and suggestions from all in society.
Please address all comments on the Green Paper on National Strategic Planning to the Minister in the Presidency for National Planning c/o:
Hassen Mohamed
E-mail: hassen@po.gov.za
Tel: 012 300 5455
Fax: 086 683 5455
Issued by: The Presidency
4 September 2009

Please see http://www.info.gov.za/speeches/2009/09090414151003.htm for the actual green paper and Policy document on performance monitoring and evaluation.

Check out the GOODs

2009-10-05T12:25:00.000+02:00

A colleague referred me to a refreshing website that might be interesting to do-gooders the world over. It is called GOOD.

Maybe they called it GOOD because it can be found at the following url: http://www.good.is/. Apparently "GOOD is a collaboration of individuals, businesses, and nonprofits pushing the world forward"

Maybe they called it GOOD because it is good. I remind you, dear reader, that I am an evaluator so I should - according to the Scrivenian* wisdom I sometimes subscribe to - be particularly well placed to pass judgements about merit and worth. However, I will reserve judgement about the Goodness of GOOD for now. Except for saying what I have already said about it.

There is an interesting blog about Innovation and Evaluation in philanthropy. See

http://www.good.is/post/innovation-and-evaluation-are-inseparable/

*OK, that only sounded GOOD in my head, but the meaning I'm hoping to convey is that Michael Scriven's writings are relevant here.

ELDIS RESOURCE

2009-09-29T15:24:00.000+02:00

http://www.eldis.org/go/topics/resource-guides/manuals-and-toolkits/monitoring-and-evaluation

The Eldis Community site enables development professionals across the world to debate, discuss and exchange ideas and information. This community group, focusing on results-based M&E, is composed of development evaluation practitioners committed to evaluation capacity building at all levels of human development activities - global, country or community level; policy, programme or project level - with the aim of bringing about an equitable, accountable and progressive society for everyone.

Social Capital is a fundamental requirement for associations to work.

2009-03-18T17:40:00.005+02:00

The IOCE has an EvaLeaders listserve which aims to connect key people across the worlds' evaluation associations. I took up the task of trying to think of something to do to get the discussion going. We settled for a "monthly discussion question" and after posting the first of the questions, we were met with a resounding silence.

A variety of hypotheses were shared in order to explain the silence, the most interesting one:

"Our first question assumed that those on the EvaLeaders list share a sense of community with leaders of other IOCE member evaluation associations, and thus would be willing to take the time to write something about what their group is up to... the reality check is that there is a long-term process involved".

Concepts like "Evaluation Community" and "Community of Practice" are frequently used when speaking about Evaluation Associations, but I certainly have not sat down to think of what this actually means in practice. I have not really come to terms with the fact that social capital is inherent in working networks... capital in all shapes and sizes are requried for a network to work. In a working network, more social capital is also easily created.

Evaluation Associations are social networks, and although we typically evaluate an association's effectiveness by the number of activities they present and by the size of their membership, the true value of an association is actually in the strength of the links between members. Its these links that make shared values and common activities possible. If something as abstract as "hapiness" can dynamically spread through social networks*, then surely values, knowledge and a whole host of other fuzzy, yet potentially important evaluation-aligned attributes can be transferred too.

The question is: How do you get the minimum social capital together to start a vibrant network? Are there social-capital loans available from the World Bank? How many in-kind donations would be required? :)

I'm afraid I have more questions than answers to ponder...

*"Dynamic spread of happiness in a large social network: longitudinal analysis ver 20 years in the Framingham Heart Study" written by James Fowled and Nicholas Christakis. (BMJ 2008;337:a2338 doi:10.1136/bmj.a2338)

Participatory Evaluation Design

2009-03-09T16:03:00.004+02:00

I'm planning an evaluation planning meeting during which the intended evaluation users will design an organizational capacity evaluation. The organizations under scrutiny deliver services to the disabled (Or is the correct term "differently Abled"?). We will start with "drawing the road" (Ross Connor recently did a presentation on this at the Lisbon EES Conference) followed by the development of a stakeholder map, clarification of evaluation questions and the development of an evaluation matrix.

The evaluation matrix will outline the final evaluation questions, indicate which stakeholder need it addresses, and will also identify the data collection method and source. As a quality control exercise I'm planning to give the team a checklist that would ask the members whether the planned data collection meets some basic evaluation principles.

Some of the principles that I will try to incorporate:
• Independence: You cannot ask the same person in whose compliance you are interested, whether they are complying. The incentive to provide false information might be very high. You can ask school principals about the degree to which the Province has met their commitments, and you can ask parents whether the school charges money, but you cannot ask the school principal whether they are charging school fees if they have been declared a no-fee school.
• Relevance: Appropriate questions must be asked. You cannot expect a member of the general public (e.g. a parent) if the school is complying with the school funding norms – He / she is unlikely to know what these entail.
• Consider Systemic Impacts. Look broader than just the cases directly affected. No fee schools are not the only ones likely to be impacted by this specific policy provision. The schools in the area are also likely to be affected in some way.
• Appropriate Samples need to be selected. The sampling approach, sample size are all related to the question that needs to be answered.
• Appropriate methods need to be selected. Although certain designs are likely to results in easy answers, they might not be appropriate
• Implementation Phase: Take into account the level of implementation when you do the assessment. It is well known that after initial implementation an implementation dip might occur. Do not try to do an impact assessment when the level of implementation has not yet stabilised in the system.
• Fidelity: Take into account the fidelity of implementation, i.e to what degree the policy was implemented as it was intended.
• Quality Focus: Although a specific funding policy might have as a major aim to improve access to services, quality should always be a consideration. It is no use you have increased access to a service that never before delivered quality outputs, outcomes and impacts. Similarly it is no use that access to a good quality service improved, but due to the increased up-take of the service, the quality were negatively impacted.

I'll provide some feedback after the workshop

GDE Colloquium on their M&E Framework

2008-11-28T12:52:00.002+02:00

Recently the Gauteng Department of Education held a colloquium on their Monitoring and Evaluation Framework. As one of the speakers, I reflected on the fact that M&E frameworks often erroneously assume that the evaluand is a stable system. I argued that there are multiple triggers that leads to the evolution of the evaluand and that this has implications for M&E.

Triggers for evolving systems, organizations, policies, programmes & interventions
(Morell, J.A. (2005). Why are there unintended consequences of program action, and what are the implications for doing evaluation? In American Journal of Evaluation 2005 (26) p 444 - 463 )
• Unforeseen consequences
– Weak application of analytical frameworks, failure to capture experience of past research
• Unforeseeable consequences
– Changing environments
• Overlooked consequences
– Known consequences are ignored for practical, political or ideological reasons
• Learning & Adapting
– As implementation happens, the learning is used to adapt
• Selection Effects
– If different approaches are tried, those that are successful are likely to be replicated and those that are unsuccessful are unlikely to be replicated.

Implications for M&E
• M&E needs to work in aid of evolution (not just change)
– The M&E framework should be key in allowing the GDE to adapt, learn, respond to changes
• Not just by ensuring that the right information is tracked, but to ensure that the right people have access to it at the right time.
• M&E needs to respond to evolution
– As the Evaluand changes, some indicators will be incorrectly focused or missing, so the framework will have to be updated periodically
– It might be necessary to implement measures that go beyond checking “whether the Dept makes progress towards reaching its goals and objectives”
• Diversity of input into the design of the framework
• Using appropriate evaluation methods
– Consider expected impacts of change in planning for roll-out of M&E

Critical analysis of an M&E framework
• Does it ask the right questions in order for us to judge the merit, worth or value” of that which we are monitoring / evaluating?
• Does it allow for credible & reliable evidence to be used?

Types of Questions to ask
(Chelimsky, E. (2007). Factors Influencing the Choice of Methods in Federal Evaluation Practice. New Directions for Evaluation 113. p 13 - 33)

• Descriptive questions: Questions that focus on determining how many, what proportion etc. for the purposes of describing some aspect of the education context. (e.g. if you were interested in finding out what the drop out rate for no-fee schools is)
• Normative questions: Questions that compare outcomes of an intervention (such as the implementation of new policies) against a pre-existing standard or norm. Norm referenced questions can use various standards to compare against:
– Previous measures for the group that’s exposed to the policy intervention (e.g. if you compare the current drop-out rate to the previous drop-out rate for a specific set of schools affected by the policy)
– A widely negotiated and accepted standard (e.g. if it was accepted that a 5% drop out rate is acceptable, you can check whether the schools currently have that drop-out rate or not)
– Measure from another similar group (e.g. if you compare the drop-out rate for different types of schools)
• Attributive questions: Questions that attempt to attribute outcomes directly to an intervention like a policy change or a programme (Is the change in the drop-out rate in no-fee schools due to the implementation of the no-fee school policy)
• Analytic-Interpretive questions that builds our Knowledge base: Questions that ask about the state of the debate issues important for decision making about specific policies. (e.g. What is known about the relationship between drop-out rate and the per-learner education spend of the Department of Education)

Questions at different Time Periods
• Prior to implementation:
– Q1.1: What does available baseline data tell us about the current situation in the entities that will be affected? (Descriptive)
– Q1.2: Given what we know about existing circumstances and the changes proposed when the new policy / programme is implemented, what are the likely impacts/ effects likely to be? (Analytic-Interpretive, Normative)
• Evidence based policy making requires some sort of ex-ante assessment of the likely changes. This assessment can then later be referred to again when the final impact evaluation is conducted.

• Directly after implementation, and continued until full compliance is reached:
– Q2.1: To what degree is there compliance to the policy / fidelity to the programme design? (Descriptive)
– Q2.2: What are the short term positive and negative effects of the policy change / programme? (Descriptive, Normative and Attributive)
– Q2.3: How can the implementation and compliance be improved? (Analytic-Interpretive)
– Q2.4: How can the negative short term effects be mitigated? (Analytic-Interpretive)
– Q2.5: How can the positive short term effects be bolstered? (Analytic-Interpretive)
• This is important because no impact assessment can be done if the policy / programme has not been implemented properly, if there are significant barriers to the implementation of the policy / programme an intervention to remove these barriers would be necessary or the policy / programme should be changed.

• After compliance has been reached and the longer term effects of the policy are able to be discerned:
– Q3.1: To what degree did the policy achieve what it set out to do? (Normative)
– Q3.2: What has been the longer term and systemic effects attributable to the policy change? (Descriptive, Normative, Attributive)
– Q3.3: How can the implementation be improved / negative effects be mitigated / positive effects be bolstered? (Analytic-Interpretive)
• This is important to demonstrate that policy change was effective in addressing the underlying issues initially requiring the policy change, and to check that no unintended perversions of the policy became implemented.

• Designs appropriate to Descriptive questions:
– CASE STUDY DESIGNS
– RAPID APPRAISAL DESIGNS
– GROUNDED THEORY DESIGNS
• Designs Appropriate to Analytic-Interpretive questions
– LITERATURE REVIEW
– MIXED METHOD DESIGNS
• Designs Appropriate to Normative questions
– TIME SERIES RESEARCH DESIGNS
• Designs Appropriate to Attributive questions
– EXPERIMENTAL DESIGNS
– QUASI-EXPERIMENTAL DESIGNS

Principles for Evidence Collection
• Independence: You cannot ask the same person in whose compliance you are interested, whether they are complying. The incentive to provide false information might be very high.
• Relevance: Appropriate questions must be asked of the right persons..
• Consider Systemic Impacts. Look broader than just the cases directly affected.
• Appropriate Samples need to be selected. The sampling approach, sample size are all related to the question that needs to be answered.
• Appropriate methods need to be selected. Although certain designs are likely to results in easy answers, they might not be appropriate
• Implementation Phase: Take into account the level of implementation when you do the assessment. It is well known that after initial implementation an implementation dip might occur. Do not try to do an impact assessment when the level of implementation has not yet stabilised in the system.
• Fidelity: Take into account the fidelity of implementation, i.e to what degree the policy was implemented as it was intended.

Metrics for Social Entrepreneurs

2008-05-08T13:56:00.002+02:00

I found this website aimed at social entrepreneurs quite useful.
http://www.socialedge.org/discussions/success-metrics/new-metrics-for-today-s-social-entrepreneurs/

It lists some approaches for measurement:
"Social entrepreneurs now have a smorgasbord of measurement methodologies to choose from in addition to developing project-specific metrics (i.e., families served, reduction in arrests, units built, jobs created). They include:

• Balanced Scorecard Methodology (New Profit Inc.)
• The Acumen-Mckinsey Scorecard (Acumen Fund)
• Social Return Assessment Scorecard (Pacific Community Ventures)
• AtKisson Compass Assessment for Investors (AtKisson)
• Poverty and Social Impact Analysis (World Bank)
• OASIS: Ongoing Assessment of Social Impacts (REDF)"

They also extract Five principles of metrics that are often mentioned in discussions

1. Do have a set of success metrics
Funders and investors want to know that you have a way of measuring your success.
2. Tailor your metrics to your mission
If you are running a non-profit, then focus on social impact; if you are running a for-profit, you need the third bottom line - ROI.
3. Measure what you can in real time, but understand that social change is often measurable only over a longer period.
Try to find polling and survey organizations that are measuring the long-term trends and use their free published data.
4. Learn about established methodologies for social measurements
Applying them will save you work, get better results, and signal investors that you are serious about metrics.
5. Look at the cost-benefit of your metrics
Determine what percentage of your operations should be reasonably dedicated to success measurement and set it aside in your proposal and operating budgets.

Personally, I think that the issue of Return on Investment is crucial for any social entrepreneur. You need to be able to prove to your donors that they are getting value for money - Too many times teachers are trained at the cost of training and astronaut.

Graphical Presentations

2008-04-02T16:00:00.005+02:00

I previously commented on Tufte's book about using graphical presentations properly. Here are some examples that you would not find in his book.

The Global Classroom - Direct from Claremont Graduate University

2008-02-14T09:44:00.003+02:00

This very useful resource was brought under my attention via the AEA and SAMEA listservs.

The panel discussions from Claremont Graduate University 's recent "What Works?" workshops are available for viewing online as part of our video library. Evaluators, foundation directors, academics, and leaders of successful for-profits and non-profits came together to discuss what really works when tackling important social problems.

To view this footage, visit us at: http://www.cgu.edu/pages/5243.asp.

To view many other talks on evaluation and topics in applied psychology, visit our full video library at http://www.cgu.edu/pages/4435.asp.

Taxonomy of Evaluation

2008-02-06T11:47:00.000+02:00

I found the Evaluation Webring's Taxonomy of Types, Approaches and Fields of Evaluation Quite Useful.

Category 1: Types of evaluation
Internal evaluation or self- evaluation
An evaluation carried out by members of the organisation(s) who are associated with the programme, intervention or activity to be evaluated.

Ex-ante evaluation or impact assessment
An assessment which seeks to predict the likelihood of achieving the intended results of a programme or intervention or to forecast its unintended effects. This is conducted before the programme or intervention is formally adopted or started. Common examples of ex-ante evaluation are environmental and/or social impact assessments and feasibility studies.

Mid-term or interim evaluation
An evaluation conducted half-way through the lifecycle of the programme or intervention to be evaluated. Monitoring An ongoing activity aimed at assessing whether the programme or intervention is implemented in a way that is consistent with its design and plan and is achieving its intended results.

Ex-post or summative evaluation
An evaluation which usually is conducted some time after the programme or intervention has been completed or fully implemented. Generally its purpose is to study how well the intervention served its aims, and to draw lessons for similar
interventions in the future.

Meta-evaluation
Two processes are often referred to as meta-evaluation: (1) the assessment by a third evaluator of evaluation reports prepared by other evaluators; and (2) the assessment of the performance of systems and processes of evaluation.

Formative evaluation
An evaluation which is designed to provide some early insights into a programme or intervention to inform management and staff about the components that are working and those that need to be changed in order to achieve the intended objectives.

Category 2: Evaluative approaches
Outcome evaluation
An evaluation which is focused on the change brought about by the programme or intervention to be evaluated or its results regarding the intended beneficiaries.
Impact evaluation An evaluation that focuses on the broad, longer-term impact or effects, whether intended or unintended, of a programme or intervention. It is usually done some time after the programme or intervention has been completed.

Performance evaluation
An analysis undertaken at a given point in time to compare actual performance with that planned in terms of both resource utilization and achievement of objectives. This is generally used to redirect efforts and resources and to redesign structures.

Participatory evaluation
An evaluation that actively involves all or selected stakeholders in the evaluation process. Different approaches involve varying degrees of participation, inclusion, capacity-building, ownership, etc.

Empowerment evaluation
An approach that aims to improve programs through using specific tools for assessing the planning, implementation and self-evaluation of programs, and by incorporating evaluation into a program or organization’s planning and management. It involves a high level of participation by stakeholders in the evaluation process and is guided by ten key principles.
Collaborative evaluation An evaluation which aims for a significant degree of collaboration or cooperation between evaluators and stakeholders.

Utilization-focused evaluation
A process that assists the primary intended users of an evaluation to select the most appropriate content, model, methods, and theory for the evaluation, focusing on their intended use of the evaluation. Use refers to how people apply evaluation findings and experience the evaluation process.

Feminist evaluation
An evaluation that commonly involves adapting or redesigning relevant evaluation theories and methodologies so that they are compatible with feminist theories and methodologies. Feminist evaluations aim to be inclusive and empowering for women in particular.

Theory-based evaluation
An evaluation based on the theories of change that underlie a given programme or intervention. Its major aim is to examine the extent to which these theories hold and to validate their underlying assumptions.

Most Significant Change
A form of participatory monitoring and evaluation which involves the collection and systematic review and analysis of change stories by panels of designated stakeholders or staff. It is mainly used to assess intermediate program impacts and outcomes.

Category 3) Fields of evaluation

Programme or project evaluation
The evaluation of a programme or project.

Policy evaluation
The evaluation of policies and procedures.

Evaluation of legislation
The evaluation of a piece of legislation.

Evaluation of technical assistance
The evaluation of technical assistance provided by international, bilateral or multilateral donors.

Organisation or institutional evaluation
An evaluation of an organization’s or other institution’s capacity for innovation and change. It involves examining its decision-making processes and organisational structures.

Proposal assessment
The assessment of bids presented by tenderers following a specific call for tenders/bids.
Financial audit The scrutiny of accounts of an organization or other institution against a set of standards.

Personnel evaluation
A systematic method of evaluating an employee’s or staff member’s performance. This involves tracking, evaluating and providing feedback in relation to specific predetermined standards which are consistent with the organization’s overall

Surveys - Should we believe them?

2008-02-05T00:05:00.000+02:00

There is a lot written about survey methodology as a tool in evaluation, but despite the easy and neat stats that they deliver, one should regard them with a little bit of skepticism, it seems.

Two stories to demonstrate the point:

According to a speaker on 702 talk radio I heard earlier this week, Volkskas bank still receives votes for one of the best brands in South Africa (in the Annual Markinor survey), despite the fact that it has ceased existence now for more than just a couple of years. At least in this survey, you can identify problematic answers because survey respondents had the option of giving an open-ended answer. I shudder to think what people actually do when they get one of those tick box multiple choice surveys...

In the next example, it is just so clear that one should question even the most basic assumptions people make when they complete a survey.

http://news.yahoo.com/s/afp/20080204/wl_uk_afp/britainpeoplehistoryoffbeat_080204001239
LONDON (AFP) - Britons are losing their grip on reality, according to a poll out Monday which showed that nearly a quarter think Winston Churchill was a myth while the majority reckon Sherlock Holmes was real. The survey found that 47 percent thought the 12th century English king Richard the Lionheart was a myth. And 23 percent thought World War II prime minister Churchill was made up. The same percentage thought Crimean War nurse Florence Nightingale did not actually exist.Three percent thought Charles Dickens, one of Britain's most famous writers, is a work of fiction himself. Indian political leader Mahatma Gandhi and Battle of Waterloo victor the Duke of Wellington also appeared in the top 10 of people thought to be myths. Meanwhile, 58 percent thought Sir Arthur Conan Doyle's fictional detective Holmes actually existed; 33 percent thought the same of W. E. Johns' fictional pilot and adventurer Biggles.

competencies/capabilities of an evaluator

2008-01-09T04:49:00.000+02:00

Q: Do you know of a document that articulates competencies/capabilities of an evaluator? If you do please send me a reference or copy.

A: Lots of work has been done on this topic by various Evaluation Associations across the world. Some useful references:

King, Jean, Stevahn, Laurie, Ghere, Gail, & Minnema, Jane (2001). Toward a taxonomy of essential evaluator competencies. American Journal of Evaluation, 22, 229-247.

Mertens, Donna M. (1994). Training evaluators: Unique skills and knowledge. New Directions for Program Evaluation, 62, 17-27.

Treasure Board of Canada Competency Profile for Federal Public Service Evaluation Professionalshttp://www.tbs-sct.gc.ca/eval/dev/Professionalism/profession_e.asp

Here is the list:

Essential Competencies for Program Evaluators (ECPE)

(Stevahn and King, Ghere, & Minnema, American Journal of Evaluation, March

2005)

1.0 Professional Practice

1.1 Applies professional evaluation standards

1.2 Acts ethically and strives for integrity and honesty in conducting evaluations

1.3 Conveys personal evaluation approaches and skills to potential clients

1.4 Respects clients, respondents, program participants, and other stakeholders

1.5 Considers the general and public welfare in evaluation practice

1.6 Contributes to the knowledge base of evaluation

2.0 Systematic Inquiry

2.1 Understands the knowledge base of evaluation (terms, concepts, theories, assumptions)

2.2 Knowledgeable about quantitative methods

2.3 Knowledgeable about qualitative methods

2.4 Knowledgeable about mixed methods

2.5 Conducts literature reviews

2.6 Specifies program theory

2.7 Frames evaluation questions

2.8 Develops evaluation designs

2.9 Identifies data sources

2.10 Collects data

2.11 Assesses validity of data

2.12 Assesses reliability of data

2.13 Analyzes data

2.14 Interprets data

2.15 Makes judgments

2.16 Develops recommendations

2.17 Provides rationales for decisions throughout the evaluation

2.18 Reports evaluation procedures and results

2.19 Notes strengths and limitations of the evaluation

2.20 Conducts meta-evaluations

3.0 Situational Analysis

3.1 Describes the program

3.2 Determines program evaluability

3.3 Identifies the interests of relevant stakeholders

3.4 Serves the information needs of intended users

3.5 Addresses conflicts

3.6 Examines the organizational context of the evaluation

3.7 Analyzes the political considerations relevant to the evaluation

3.8 Attends to issues of evaluation use

3.9 Attends to issues of organizational change

3.10 Respects the uniqueness of the evaluation site and client

3.11 Remains open to input from others

3.12 Modifies the study as needed

4.0 Project Management

4.1 Responds to requests for proposals

4.2 Negotiates with clients before the evaluation begins

4.3 Writes formal agreements

4.4 Communicates with clients throughout the evaluation process

4.5 Budgets an evaluation

4.6 Justifies cost given information needs

4.7 Identifies needed resources for evaluation, such as information, expertise, personnel, instruments

4.8 Uses appropriate technology

4.9 Supervises others involved in conducting the evaluation

4.10 Trains others involved in conducting the evaluation

4.11 Conducts the evaluation in a nondisruptive manner

4.12 Presents work in a timely manner

5.0 Reflective Practice

5.1 Aware of self as an evaluator (knowledge, skills, dispositions)

5.2 Reflects on personal evaluation practice (competencies and areas for growth)

5.3 Pursues professional development in evaluation

5.4 Pursues professional development in relevant content areas

5.5 Builds professional relationships to enhance evaluation practice

6.0 Interpersonal Competence

6.1 Uses written communication skills

6.2 Uses verbal/listening communication skills

6.3 Uses negotiation skills

6.4 Uses conflict resolution skills

6.5 Facilitates constructive interpersonal interaction (teamwork, group facilitation, processing)

6.6 Demonstrates cross-cultural competence

M&E for Philanthropy

2007-12-21T00:40:00.000+02:00

The following shows what makes it so difficult to work with charities!

2 Young Hedge-Fund Veterans Stir Up the World of Philanthropy

By STEPHANIE STROM

Published: December 20, 2007

Holden Karnofsky and Elie Hassenfeld rank charities by analyzing the numbers in much the same way they did at their investment management company
http://www.nytimes.com/2007/12/20/us/20charity.html?ex=1355893200&en=a0d9a701ad60ffd0&ei=5124&partner=permalink&exprod=permalink

In the fall of 2006, they and six colleagues created what Mr. Karnofsky calls a “charity club.” Each member was assigned to research charities working in a specific field and report back on those that achieved the best results. They were stunned by the paucity of information they could collect.

“I got lots of marketing materials from the charities, which look nice, you know, pictures of sheep looking happy and children looking happy, but otherwise are pretty useless,” said Jason Rotenberg, a former member of the club and now a $50,000 donor to the Clear Fund. “It didn’t seem like a reasonable way of deciding between one charity and another.”

GiveWell’s findings are available on the Internet, without charge, at www.givewell.net. In evaluating charities, Mr. Karnofsky and Mr. Hassenfeld press them for information, analyzing the numbers in much the same way they did at Bridgewater. The Smile Train, for instance, a charity that repairs cleft palates, was asked how much it spent in each region and each country to treat how many patients in each.Many in the field question how long GiveWell can survive. While 34 percent of wealthy donors who responded to a survey sponsored by the Bank of America said they wanted more information on nonprofits, almost three-quarters said they would give more if charities spent less on administration. And collecting information is costly.

As a result, most philanthropic advisory services like GiveWell have a hard time raising money. The Clear Fund has raised $300,000 since its inception this year, about half of which has gone to operating GiveWell.

The problem is that you want charities to measure impact. Although the cost of measuring is sometimes prohibitive, the potential cost of not measuring should be enough reason to make sure that you do measure.

Probability Sampling Approaches

2007-11-12T00:31:00.000+02:00

Probability Sampling Approaches

Probability sampling approaches allow you to generalize to the full population, since it ensures that special random characteristics are likely to be distributed evenly across the units included and excluded in / from the sample. It therefore is likely to yield a less biased sample and the results could be said to apply to the full population (if the appropriate sample size was selected). Different kinds of probability sampling approaches are possible.

The figures below demonstrate the different approaches. Assume each number is a unique member of the population, assume that each group consists of discreet mutually exclusive members of the population (In columns) and assume that each cluster (delineated by a block) is a group of members in the same geographic area.

With simple random sampling the sample is selected from the whole population using a table of numbers. Note that this does not necessarily ensure balanced representation amongst different groups.

With stratified random sampling, a set number of participants from each group can be selected. Note that this does not necessarily ensure that the most economical approach is used. In the example some cases from almost all of the geographic clusters are included.

With cluster sampling, a set number of clusters are randomly selected (in this case 4) with a set number of randomly selected units within each cluster (in this case 5). Although this will be more economical in terms of fieldwork costs because travel to different clusters have been limited, it does not necessarily guarantee equal representation of groups.

With systematic sampling, a set pattern is systematically applied to select participants. In the case of the example above, every 11^th member of the population were selected. Note that it did not require a random table of numbers, but were still subject to the same limitations as the simple random sample.

Type of Probability Samples	When is it applicable	Drawbacks
Simple Random Sampling (I.e.randomly select 50 schools off a list with all schools in the country)	It is ideal for statistical purposes	· It may be difficult to achieve in practice · It requires a precise list of the whole population · It is costly to conduct as those sampled may be spread over a wide area.
Stratified Random Sampling (I.e. Randomly select 50 schools per strata such as province)	· It ensures better coverage of the population than simple random sampling. · It is administratively more convenient to stratify a sample – interviewers can be specifically trained to manage particular strata (e.g. age, gender, ethnic or language groups).	· Difficulty in identifying appropriate strata. · More complex to organize and analyse results
Cluster Sampling (I.e. split the schools in a province up in geographical clusters, select 10 clusters randomly, and then proceed to visit 20 schools within each cluster)	More cost effective in terms of travel, thereby producing a reduction in the overall cost	· Units in a cluster may be very similar and therefore are less likely to represent the whole population · Cluster sampling has a larger sampling error than simple random sampling.
Systematic Sampling (i.e. a set pattern is applied to the data set, e.g. every 11^th member is selected)	It spreads the sample more uniformly over the population and is easier to conduct than simple random sampling.	The system may interact with a concealed pattern in the population.

Local monitoring of Public Service Delivery

2007-11-08T11:48:00.000+02:00

I find the readings on the pelican listserve (Pelican Initiative: Platform for Evidence-based Learning & Communications for Social Change) always interesting. Today the message below was posted.

I find the description of Social Auditing quite interesting. At first I thought it might be similar to Social Accounting - the move of private companies to also account on the triple bottom line of Environmental, Social and Economical impacts (costs and benefits) in order to promote corporate responsibility. We've actually done a couple of sustainability reports using this framework. The focus, however is more on accountability than it is on changing things for improvement. Hope it is not the same for this kind of local government monitoring.

I have also previously posted something on the MSC techniques described here. I think it is quite useful if there are no clearly defined objectives to start off with, or if the social reality is very complex and requires more of a systems look at the effects. Any case. You decide for yourself!

PS. the AEA conference is on in Baltimore this week, and although I am not able to attend, my business partner is. It sounds like it is interesting and stimulating as always!

Ciao

B

**********************************************

Last month, a number of useful documents and experiences were shared by Gilles Mersadier, who is the coordinator of the FIDAfrique network (http://www.fidafrique.net/article413.html). Among the material that he shared, he referred to a methodology for capitalisation (the process of sharing experiences among and across organisations) that is being used in the context of 25 West-African rural development projects. To date, 20 of these projects have published different types of documents that are disseminated throughout the FIDAfrique network. The production of these documents, which are available on the network's website, are supported by a methodological guide on the capitalisation process (in French and
English):
http://www.fidafrique.net/article467.html?var_recherche=capitalization

In the past few weeks, different contributions have been sent in the context of the discussion around local monitoring of public service delivery. One of the questions which we posed in the beginning of this discussion focused on the types of approaches that are being used for local monitoring purposes. In this message, I would like to briefly describe two of such approaches:

1: Social Auditing
2: Most Significant Change

**** 1: Social Auditing

The Social Auditing method has been developed and used for participatory monitoring of public service delivery by the organisation CIET (for more info on the organisation, which started in 1985 in Mexico and developed into an international network, please visit http://www.ciet.org/en/aboutciet/ ).

The method's primary aim is to 'increase the informed interaction between communities and public services'. The impact, coverage and costs of public services are examined through a combination of quantitative (survey) and qualitative (key informant and focus groups) evidence. The civil society plays a central role in interpreting this evidence, and through this process contributes to the creation of local solutions. The use of both 'hard' and 'soft' evidence helps to provide a strong and accurate underpinning to the locally defined ideas and solutions, and as such strengthens the legitimacy of the solutions that are developed.

A report that was published in 2005 on the use of the method for assessing 'governance and delivery of public services' in Pakistan lists the following seven stages of a Social Audit cycle:
(1) Clarify the strategic focus;
(2) Design sample and instruments, pilot testing;
(3) Collect information from households on use and perception of public services;
(4) Link this with information from the public services;
(5) analyse the findings in a way that points to action;
(6) Take findings back to the communities for their views about how the improve the situation;
(7) Bring evidence and community voice into discussions between service providers, planners and community representatives to plan and implement changes.

The CIET website features an extensive library section where the reports of previous social audits can be accessed, together with other experiences relating to the network's central focus on the 'socialisation of
evidence':
http://www.ciet.org/en/browse/librarydocs/

**** 2: Most Significant Change

Central in the Most Significant Change (MSC) technique is - as the name suggests - the collection of significant change stories that emerge from the field level. Following the collection of these stories, those stories which are considered most significant are selected by panels of designated stakeholders or staff. The collection, discussion and further selection of the stories revolves around 'domains': the areas that the stakeholders collectively decide on as the focus of the monitoring. Provided that it is clear who is involved at selecting the stories at the different levels, the use of the domains makes for a transparent monitoring process. Given these characteristics, and the fact that it is relatively easy to learn how to use it, the technique can be useful in the context of local monitoring of public service delivery.

While an English guide about the technique has been available since 2005
(see: http://www.mande.co.uk/docs/MSCGuide.htm ), efforts have also been made to translate the guide into different international and local languages, including Spanish, French, Russian, Tamil and Indonesian. Rick Davies has set up a specific Web Log to make available these translations, and to allow users to share suggestions to further improve the quality of the translations. You can access this website here:

http://mscguide-translations.blogspot.com/

Our current discussion on the topic of local monitoring of public service delivery is moving towards an end, so it would be great if some of you could still share some experiences and ideas on this topic in case you did not yet have the time to do so. We will aim to send around a summary of the key points that have been contributed sometime next week.

Best wishes,
Niels

What is an Evaluator?

2007-08-16T09:20:00.000+02:00

Do you find it difficult to explain to people what you do? Those magical two sentences that will get people to go "Aaaaaah, now I get what you do?" Unfortunately I have not been able to come up with something concrete yet. But I am still trying.

In a television interview to publicise the SAMEA conference, the DDG from the PSC Mr. Mash Dipofu tried to explain it with an example. He asked the TV presenter if he knew what the viewers thought of his programme, how it can be improved and how many people actually watches it. He explained that by answering these questions, you are doing what an evaluator would be doing and answering the underlying question: Does what I am doing have value?

Which got me thinking. So much of what we do as evaluators are also done by other professionals.

We are a little like investigative journalists: We talk to people and ask questions and gather information to make an argument for or argainst something. Sometimes to inform readers of some wrong doing... Sometimes we celebrate what has been achieved.
Then we are also a little like the weather guy. We collect numbers over a long period of time and by applying some statistical techniques we can start predicting what will happen in future.
Another way of looking at our job is to compare it to that of teachers. We guide people to learn from their environments - assuming that they need to be taught how to use the information at their disposal to make intelligent choices. We check with tests whether the intended result has been achieved... much like teachers check whether their students have mastered a skill or knowledge component.
And then of course evaluators are also a little like an auditor in the way that we try to prove to people that money has been well spent.

The problem with explaining to people what we do, is probably because people tend to confuse it with research and planning and implementation and all kinds of other things. I have also spent some time to think about how being an evaluator is different from being a researcher, a planner and an implementer.

Because we use research techniques to collect evidence in order to evaluate, the difference between being a researcher (who asks questions in a specific way to gather evidence) and an evaluator (who asks questions in a specific way to gather evidence to then make a value judgment about the evaluand) is sometimes a little difficult to explain. But there is a difference!
Planning, on the other hand comes quite naturally when you are an evaluator. After delivering an evaluation, I frequently get asked to assist in planning processes - if people value what you produced in the evaluation they want to make sure that they plan to implement the recommendations made. Being a weather guy and an auditor makes it easier to plan because you can draw info together to make predictions, and you know that you will have to explain to people why you chose to spend their money in a particular way.
I think evaluators will probably make terrible implementers. As an evaluator you are constantly asking questions: Is this the best way to do things? Will we achieve results? How would we know that we added value? How do we know that this is the best way forward? To implement, however, you sometimes have to say "Well I don't know all the answers but I am making a decision to do ABC in the following way and that is the way it is!"

Despite having useful analogies to explain what evaluators do and don't do, I think that we are at risk if we, as practitioners of a scientific metadiscipline, don't understand how the ideology underlying evaluation is different from other those informing other jobs. Evaluators might be sharing some commonalities with teachers, weather people, auditors and journalists, but we have different values and assumptions guiding our work. We cannot forget that our work probably has a deeply political nature because we have to choose at some stage whose questions we will have to answer.

I found the following useful bit about evaluation approaches and the underlying philosophy, epistemology and ontology at http://www.recipeland.com/facts/Evaluation

Classification of approaches

Two classifications of evaluation approaches by House House, E. R. (1978). Assumptions underlying evaluation models. Educational Researcher. 7(3), 4-12. and Stufflebeam & Webster Stufflebeam, D. L., & Webster, W. J. (1980). An analysis of alternative approaches to evaluation. Educational Evaluation and Policy Analysis. 2(3), 5-19. can be combined into a manageable number of approaches in terms of their unique and important underlying principles.

House considers all major evaluation approaches to be based on a common ideology, liberal democracy. Important principles of this ideology include freedom of choice, the uniqueness of the individual, and empirical inquiry grounded in objectivity. He also contends they all are based on subjectivist ethics, in which ethical conduct is based on the subjective or intuitive experience of an individual or group. One form of subjectivist ethics is utilitarian, in which â€œthe goodâ€ is determined by what maximizes some single, explicit interpretation of happiness for society as a whole. Another form of subjectivist ethics is intuitionist / pluralist, in which no single interpretation of â€œthe goodâ€ is assumed and these interpretations need not be explicitly stated nor justified.

These ethical positions have corresponding epistemologiesâ€”philosophies of obtaining knowledge. The objectivist epistemology is associated with the utilitarian ethic. In general, it is used to acquire knowledge capable of external verification (intersubjective agreement) through publicly inspectable methods and data. The subjectivist epistemology is associated with the intuitionist/pluralist ethic. It is used to acquire new knowledge based on existing personal knowledge and experiences that are (explicit) or are not (tacit) available for public inspection.

House further divides each epistemological approach by two main political perspectives. Approaches can take an elite perspective, focusing on the interests of managers and professionals. They also can take a mass perspective, focusing on consumers and participatory approaches.

Stufflebeam and Webster place approaches into one of three groups according to their orientation toward the role of values, an ethical consideration. The political orientation promotes a positive or negative view of an object regardless of what its value actually might be. They call this pseudo-evaluation. The questions orientation includes approaches that might or might not provide answers specifically related to the value of an object. They call this quasi-evaluation. The values orientation includes approaches primarily intended to determine the value of some object. They call this true evaluation.

http://www.recipeland.com/facts/Evaluation

Rant for today*

2007-07-11T03:34:00.000+02:00

A potential client sends out Terms of Reference requesting potential service providers to submit a quotation for an (5 maybe 10 person day) evaluation engagement. Note, they did not ask for a proposal, they asked for a quotation. Given the limited scope of the project, a quotation makes sense. So that is what I submit.

Then potential client reads through the proposals they received and *Horror* *shock* discovers that there isn’t enough information in the quotations to make a transparent decision. (Maybe there was a little problem with the Terms of Reference?) They then set up a meeting in which they expect potential service providers to present to a panel. “For the purposes of meeting with the potential evaluators and ensuring the selection process is fair and transparent”. (Have I mentioned that this is a small job?)

So I go and have a wonderful meeting with the potential client, but in the end do not get the job. My heart isn’t broken or anything. It would’ve been nice to get the job but… ah well, I think they looked for a content specialist rather than an evaluation specialist in the first place.

But hey, I am an evaluator, and evaluative thinking requires me to find out why I was not successful. So I write the “Thanks for the notification, we are disappointed, could you please tell us where our submission was weak… bla-di-blah” email. To which I don’t receive a response. I get my office manager to follow up and get a response of the kind: “We liked your presentation / Sorry we don’t have time to provide feedback / now please just leave us alone”.

What happened to the transparency and fairness thing? Am I unreasonable to think that a two line explanatory email is not much to ask after all of the trouble they put me through?

Ahem… The UK evaluation Society has some guidelines for persons when they commission evaluations at:

http://www.evaluation.org.uk/Pub_library/PRO2907%20-%20UKE.%20A5%20Guideline.pdf

Maybe more people should read that?

*Because this is my blog I get to complain here every now and again. I promise that this will not turn into those rant-upon-rant blogs, but I really need to get the following off my chest.

Whew! Its been like how long?

2007-07-11T03:23:00.001+02:00

I see I haven't posted anything here since April. That is probably because I have nothing to write about when I have time, and when I have time to write, I can't think of anything to write about. So here are a list of things I would like to address at some stage:

* What people think a focus group is and what it really is
* Rapid assessment methods - differences in approaches (I need to draw up a table based on that AJE article I read yesterday)
* When people should consider appointing an evaluation specialist rather than a content specialist for certain evaluations.
* If evaluators do strategic planning, what is it that they should know about planning AND if planners or anybody else does evaluations what is it they need to know about evaluations
* People say evaluation is a meta-discipline. Why do they say that.
*All this talk about accreditation is just confusing everybody. What does it mean and what about the international body of work that has been done on this aspect?

Handy Publication: New Trends in Evaluation

2007-04-25T09:21:00.000+02:00

You know how you always come back with a stack of stuff to read when you've attended an evaluation conference? It most cases the material just gets added to my ever growing "to read" pile. It is only once I start searching for something or decide to spring-clean, that I actually sit down and read some of the stuff. This morning I came across a copy of UNICEF / CEE/CIS and IPEN's New Trends in Evaluation.

What a delightfully simple straightforward publication - yet it packs so much relevant information between its two covers. I wish I had remembered about it last week when I lectured to students at UJ. Before I was able to get on with the lecture on Participatory M&E, I first had to explain how M&E is different and similar to Social Impact Assessments (In the sense of ex-ante Environmental Impact Assessment type assessment). I think it would have been a very handy introductory source to have.

The table of contents looks as follow:

1. Why Evaluate?
The evolution of the evaluation function
The status of the evaluation function worldwide
The importance of Evaluation Associations and Networks
THe oversight and M&E function
2. How to Evaluate?
Evaluation culture: a new approach to learning and change
Democracy and Evaluation
Democratic Approach to Evalution
3. Programme Evaluation Development in the CEE/CIS

But what is really useful is the Annexures:
Annex 1: Internet Bades Discussion Groups Relevant to Evaluation
Annex 2: Internet Websites Relevant to Evaluation
Annex 3: Evaluation Training and Reference Sources Available Online
Annex 4-1: UNEG Standards for Evaluation in the UN System
Annex 4-2: UNEG Norms for Evaluation in the UN System
Annex 5: What goes into a Temrs of Reference; UNICEF evaluation Technical Notes, Issue No.2

The good thing about this publication is that you can download it for free off the internet at

http://www.unicef.org/ceecis/New_trends_Dev_EValuation.pdf

An introductory blurb and a presentation is also available from the IOCE website.

http://ioce.net/news/news_articles/061023_unicef-ipen.shtml

Presentation Presented at the SAMEA conference 26 - 30 March 2007

2007-04-10T00:27:00.000+02:00

Based on some of the ideas in previous blog entries, I presented the following presentation at the recent SAMEA conference.

Setting Indicators and Targets for Evaluation of Education Initiatives

Introduction

Good Evaluation Indicators and Targets are usually an important part of a robust Monitoring and Evaluation system.
Although evaluation indicators are usually considered as important, all evaluations do not have to make use of a set of pre-determined indicators and targets.
The most significant change (MSC) technique, for example, looks for stories of significant change amongst the beneficiaries of a programme, and after the fact uses a team of people to determine which of these stories represent MSC and real impact.
You have to include the story around the indicators in your evaluation reports in order to learn from the findings.

What do we mean?

The definition of an Indicator is: “A qualitative or quantitative reflection of a specific dimension of programme performance that is used to demonstrate performance / change”
It is distinguished from a Target which: “Specifies the milestones / benchmarks or extent to which the programme results must be achieved”
And it also different from a Measure which is: “The Tool / Protocol / Instrument / Gauge you use to assess performance”

Types of Indicators

The reason for using indicators is to feel the pulse of a project as it moves towards meeting its objectives or to see the extent to which it has been achieved. There are different types of indicators:
Risk/enabling indicators – external factors that contribute to a project’s success or failure. They include socio-economic and environmental factors, the operation and functioning of institutions, the legal system and socio-cultural practices.
Input indicators – also called ‘resource’ indicators, they relate to the resources devoted to a project or programme. Whilst they can flag potential challenges, they cannot, on their own determine whether a project will be a success or not.
Process indicators – also called ‘throughput’ or ‘activity’ indicators. They reflect delivery of resources devoted to a programme or project on an ongoing basis. They are the best indicators of implementation and are used for project monitoring.
Output indicators –indicates whether activities have taken place by considering the outputs from the activities.
Outcome indicators - indicates whether your activities delivered a positive outcome of some kind.
Impact indicators – Concerns the effectiveness, usually long term, of a programme or project as judged by the measurable achieved in improving the quality of life of beneficiaries or other similar impact level result.

Good Indicators

Good Performance Indicators should be
Direct (Does it measure Intended Result?)
Objective (Is it ambiguous?)
Adequate (Are you measuring enough?)
Quantitative (Numerical comparisons are less open to interpretation)
Disaggregated (Split up by gender, age, location etc.)
Practical (Can you measure it timeously and at reasonable cost?)
Reliable (How confidently can you make decisions about it?) (USAID, 1996)

SMART Indicators
Most people have also heard about SMART indicators:

Specific
Measurable
Action Oriented
Realistic
Timed

How we use indicators

For many of the evaluation initiatives that we help to plan M&E systems for, we usually work with the managers to set indicators that they understand and can use.
Although the issue of data availability and data quality is usually a big concern, it is often the indicators and targets that are set that could make or break an evaluation.

Case Study

Implementers of a teacher training initiative wants to know if their project is making a difference in the maths and science performance of learners.

Pitfalls

Alignment between Indicators & Targets (If the indicator says something about a number, then the target must also be couched in terms of a number, and not a percentage)
Averaging out things that do not belong together (i.e. maths and science) does not make sense at all.
Not disaggregating enough (Are you interested in all learners, or is it important to look at disaggregating your data by age group, gender, educator)
Assuming that all targets should be about an increase: (Sometimes a trend in the opposite direction exists and it is expected that your programme will only mediate the effects)
Assuming that an increase from 20% to 50% is the same as an average increase of 50% to 80%. (Psychometrists have used the standardised gain statistic for a very long time. It is interesting that we don’t see more of it in our programmes.)
Ignoring the statistics you will use in analysis: (In some cases you are using a sample and averages. This means an average increase might just look like an increase, but when you test for statistical significance it is actually not an increase)
Setting indicators that require two measurements where one would be enough (Are you interested in an average increase, or just the % of people that make some minimum standard.)
Ignoring other research done on the topic (If a small effect size is generally reported for interventions of these kinds, isn’t an increase of 30% over baseline a little ambitious?)
If you don’t have other research on the topic, it should be allowable to adjust the indicators.
Setting an indicator and target that assumes direct causality between the project activity and the anticipated outcome (Even if you have brilliant teachers, how must the learners perform if learners have nowhere to do homework, School discipline is non-existent and after learners have accumulated 10 years of conceptual deficits in their education?)
Ignoring Relevance, efficiency, sustainability, and equity considerations. (Is educators training really going to solve the most pressing need?If your programme makes a difference, is it at the same cost as training an astronaut?What will happen if the trained educator leaves?Does the educator training benefit rural learners in the same way in which it would benefit urban learners?)

Ways to address the pitfalls

Do a mock data exercise to see how your indicator and target could play out.
This will help you think through the data sources, the statistics, and the meaning of the indicator
Read extensively about similar projects to determine what the usual effect size is.
When you do your problem analysis, be sure to include other possible contributing factors, and don’t try to attribute change if it is not justifiable.
Look at examples of other indicators for similar programmes
Keep at it and work with someone who would be able to check your proposed indicators with a fresh eye.

Where to look
Example Indicators can be found in:

Project / Programme Evaluation reports from multi-lateral donor agencies
UNESCO Education For All Indicators
Long term donor-funded projects such as DDSP, QIP.
StatsSA publications and statistical extracts about the education sector.
Government M&E indicators.

Announcement from AEA about benefits

2007-03-07T09:14:00.000+02:00

The AEA announced that two more Journals are available to members!

Announcement: American Evaluation Association Expands Online Journal Access

AEA Members now receive electronic access to two additional journals - Evaluation and the Health Professions and Evaluation Review - in addition to continued access to AEA's own American Journal of Evaluation and New Directions for Evaluation, as part of membership benefits.

The journals' content is searchable, and archived online content goes back multiple years.

Individual subscriptions to each journal are over $100 each, making AEA membership more of a value then ever at only $80. Members also receive AJE and NDE in hardcopy, discounts on conference and training registration, regular communications about news from all corners of the evaluation community, discounts on books, and the opportunity to participate in the life of the association and the field.

Learn more about AEA and join online at: www.eval.org

Centre for Global Development

2007-02-01T04:42:00.000+02:00

This is the Centre for Global Development Report that upset so many people again. I mean really! Didn't we agree that mixed methods are the way to go? RCTs Cannot possibly be the answer for all our impact evaluation questions.

From the CGD website at:

http://www.cgdev.org/content/publications/detail/7973

When Will We Ever Learn? Improving Lives Through Impact Evaluation

05/31/2006

Each year billions of dollars are spent on thousands of programs to improve health, education and other social sector outcomes in the developing world. But very few programs benefit from studies that could determine whether or not they actually made a difference. This absence of evidence is an urgent problem: it not only wastes money but denies poor people crucial support to improve their lives.

This report by the Evaluation Gap Working Group provides a strategic solution to this problem addressing this gap, and systematically building evidence about what works in social development, proving it is possible to improve the effectiveness of domestic spending and development assistance by bringing vital knowledge into the service of policymaking and program design.

In 2004 the Center for Global Development, with support from the Bill & Melinda Gates Foundation and The William and Flora Hewlett Foundation, convened the Evaluation Gap Working Group. The group was asked to investigate why rigorous impact evaluations of social development programs, whether financed directly by developing country governments or supported by international aid, are relatively rare. The Working Group was charged with developing proposals to stimulate more and better impact evaluations. This report, the final report of the working group, contains specific recommendations for addressing this urgent problem.

Monitoring without Indicators - Most Significant Change

2007-01-31T01:11:00.000+02:00

On the Pelican list today, they sent through this handy reference to something that I think is infinitely useful for gathering proof and evidence when you don't have indicators and stacks of pre-developed evaluation mechanisms.

Check it out at:
http://www.mande.co.uk/docs/MSCGuide.htm. and http://www.mande.co.uk/MSC.htm

The guide (Prepared by Rick Davies and Jess Dart) explains the MSC technique as follows:

"The most significant change (MSC) technique is a form of participatory monitoring and evaluation. It is participatory because many project stakeholders are involved both in deciding the sorts of change to be recorded and in analysing the data. It is a form of monitoring because it occurs throughout the program cycle and provides information to help people manage the program. It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole.
Essentially, the process involves the collection of significant change (SC) stories emanating from the field level, and the systematic selection of the most significant of these stories by panels of designated stakeholders or staff. The designated staff and stakeholders are initially involved by ‘searching’ for project impact. Once changes have been captured, various people sit down together, read the stories aloud and have regular and often in-depth discussions about the value of these reported changes. When the technique is implemented successfully, whole teams of people begin to focus their attention on program impact."

Certainly this looks like a very promising technique!

Report Back: Making Evaluation Our Own

2007-01-30T10:00:00.000+02:00

A special stream was held on making Evaluation our own at the AfrEA conference. After the conference a small committee of African volunteers worked to capture some of the key points of the discussion. Thanks to Mine Pabari from Kenya for forwarding a copy!

What do you think of this?

Making Evaluation Our Own: Strengthening the Foundations for Africa-Rooted and Africa Led M&E

Overview & Recommendations to AfrEA

Niamey, 18^th January, 2007

Discussion Overview

On 18 January 2007 a special stream was held to discuss the topic

Making Evaluation our own: Strengthening the Foundations for Africa-Rooted and Africa-Led M&E. It was designed to bring African and other international experiences in evaluation and in development evaluation to help stimulate debate on how M&E , which has generally been imposed from outside, can become Africa led and owned.

The introductory session aimed to set the scene for the discussion by considering i) What the African evaluation challenges are (Zenda Ofir) ii) The Trends Shaping M&E in the Developing World (Robert Piccioto) iii) The African Mosaic and Global Interactions: The Multiple Roles of and Approaches to Evaluation (Michael Patton & Donna Mertens). The last presentations explained, among others, the theoretical underpinnings of evaluation as it is practiced in the world today.

The next session briefly touched on some of the current evaluation methodologies used internationally in order to highlight the variety of methods that exist. It also stimulated debate over the controversial initiative on impact evaluation launched by the Center for Global Development in Washington. The discussion then moved to consider some of the international approaches that are currently useful or likely to become prominent in finding evidence about development in Africa (Jim Rugh, Bill Savedoff, Rob van den Berg, Fred Carden, Nancy MacPherson & Ross Conner)

The final session aimed to consider some possibilities for developing an evaluation culture rooted in Africa. (Bagele Chilisa). In this session some examples of how the African culture leans itself towards evaluation was given and also some examples that demonstrated that the currently used evaluation methodologies could be enriched if it considered an African world view.

Key issues emerging from the presentations and discussion formed the basis for the motions presented below:

Currently much of the evaluation practice in Africa is based on external values and contexts, is donor driven and the accountability mechanisms tend to be directed towards recipients of aid rather than both recipients and the providers of aim
For evaluation to have a greater contribution to development in Africa it needs to address challenges including those related to country ownership; the macro-micro disconnect; attribution; ethics and values; and power-relations.
A variety of methods and approaches are available and valuable to contributing to frame our questions and methods of collecting evidence. However, we first need to reexamine our own preconceived assumptions; underpinning values, paradigms (e.g. transformative v/s pragmatic); what is acknowledged as being evidence; and by whom before we can select any particular methodology/approach.

The lively discussion that ensued led towards the appointment of a small group of African evaluators to note down suggested actions that AfrEA could spearhead in order to fill the gap related to Africa-Rooted and Africa-Led M&E.

The stream acknowledges and extends its gratitude to the presenters for contributing their time to share their experiences and wealth of knowledge. Also, many thanks to NORAD for its contribution to the stream; and the generous offer to support an evaluation that may be used as a test case for an African-rooted approach – an important opportunity to contribute to evaluation in Africa.

In particular, the stream also extends much gratitude to Zenda Ofir and Dr. Sully Gariba for their enormous effort and dedication to ensure that AfrEA had the opportunity to discuss this important topic with the support of highly skilled and knowledgeable evaluation professionals.

Motions

In order for evaluation to contribute more meaningfully to development in Africa, there is a need to re-examine the paradigms that guide evaluation practice on the continent. Africa rooted and Africa led M&E requires ensuring that African values and ways of constructing knowledge are considered as valid. This, in turn, implies that:

§ African evaluation standards and practices should be based on African values & world views

§ The existing body of knowledge on African values & worldviews should be central to guiding and shaping evaluation in Africa

§ There is a need to foster and develop the intellectual leadership and capacity within Africa and ensure that it plays a greater role in guiding and developing evaluation theories and practices.

We therefore recommend the following for consideration by AfrEA:

o AfrEA guides and supports the development of African guidelines to operationalize the African evaluation standards and; in doing so, ensure that both the standards and operational guidelines are based on the existing body of knowledge on African values & worldviews

o AfrEA works with its networks to support and develop institutions, such as Universities, to enable them to establish evaluation as a profession and meta discipline within Africa

o AfrEA identifies mechanisms in which African evaluation practitioners can be mentored and supported by experienced African evaluation professionals

o AfrEA engages with funding agencies to explore opportunities for developing and adopting evaluation methodologies and practices that are based on African values and worldviews and advocate for their inclusion in future evaluations

o AfrEA encourages and supports knowledge generated from evaluation practice within Africa to be published and profiled in scholarly publications. This may include;

§ Supporting the inclusion of peer reviewed publications on African evaluation in international journals on evaluation (for example, the publication of a special issue on African evaluation)

§ The development of scholarly publications specifically related to evaluation theories and practices in Africa (e.g. a journal of the AfrEA)

Contributors

§ Benita van Wyk – South Africa

§ Bagele Chlisa – Botswana

§ Abigail Abandoh-Sam – Ghana

§ Albert Eneas Gakusi – AfDB

§ Ngegne Mbao – Senegal

§ Mine Pabari - Kenya

More Evaluation Checklists

2007-01-30T09:46:00.000+02:00

Last week I put a reference to the UFE Check-list on my blog, and today I received a very useful link with all kinds of other evaluation check-lists on the AfrEA listserv . Try it out at:

http://www.wmich.edu/evalctr/checklists/checklistmenu.htm

It has Check-lists for

*Evaluation Management

*Evaluation Models

*Evaluation Values & Criteria

* Check-lists are useful for practitioners because it helps you to develop and test your methodology with view of improving it for the future.
* They are useful for those who commission evaluations because it reminds you what should be taken into account at all stages of the evaluation process.
* I think, however, that check-lists like these can be particularly powerful if they become institutionalised in practice - If an organisation requires the check-list to be considered as part of a day-to-day business process.

Making Evaluation our Own

2007-01-29T16:26:00.000+02:00

At the AfrEA conference, there was a special stream on: 'Making Evaluation our Own'. It aimed to investigate where we are in terms of having Africa rooted, Africa lead evaluations.

I found it particularly useful because it became patently obvious that there are African world views and African methods of knowing that are not yet exploited for Evaluation in Africa. This of course brings the whole debate about "African" Evaluation theories to bear, and asks which kinds of evaluation theories are currently influencing our practice as evaluators in Africa.

Marvin C. Alkin and Christina A. Christie developed what they call the EVALUATION THEORY TREE. It splits the prominent (North-American) evaluation theorists into three big branches: Theories that focus on the use of evaluation, theories that focus on the methods of evaluation and theories that focus on how we value when evaluating. You can find more information about this at http://www.sagepub.com/upm-data/5074_Alkin_Chapter_2.pdf

The second tree is a slightly updated version. It was interesting to note that most of my reading about evaluation has been on "Methods" and "Use".

I think that if we are serious about developing our own African evaluation theories, we might need to develop our own African tree. Bob Piccioto mentioned that the African tree might use the branches of the above tree as roots, and grow its own unique branches.

A small commission from the conference put together a call for Action that outlines some key steps that should be taken if we hope to make progress soon. Hopefully I can post this at a later stage.

Keep well!

UFE & The difference between Evaluation and Research

2007-01-25T14:21:00.000+02:00

At the recent AFREA conference I was again reminded of what we are supposed to be doing in evaluation. Consider the word evaluation: It is about valuing something. Valuing for the purposes of accountability and for learning and improvement.

It is not just research, and although some people have indicated that they get irritated with our attempts at distinguishing evaluation from research, I think it is critically important to distinguish between research and evaluation.

Depending on which paradigm you come from, one might argue that research can be the same as evaluation. I don’t argue with that. What I do have a problem with is people approaching evaluations like research projects where the focus is all on “How do we collect evidence?” The methodology is critically important, agreed, and there is nothing that grates me more than seeing how people use poorly designed evaluation methodologies to collect “evidence”.

But evaluation is not just about how we collect information. Evaluation is supposed to take it a step further and make some evaluative judgments based on the data that was collected. Just describing your evaluation findings without saying what it means is senseless.

It is good and well if you find information about the level of maths capacity in rural schools interesting, but an evaluation will also go further and indicate whether the project is relevant, effective, efficient, has an impact and is sustainable or creates sustainable results. Without this additional “Valuing” judgments, an evaluation is only a research project that may increase our knowledge, but don’t help us to make decisions.

Something that may help more evaluations to be true evaluations is the Utilization Focused Evaluation approach of Michael Quinn Patton. It is all about how to ensure that an evaluation serves its intended purpose for the intended users. Go ahead – google Utilization Focused Evaluation and see how many hits come up. It literally is the biggest thing that has hit the Evaluation community in the past 30 years, yet many people are blissfully ignorant of this.

For those who commission evaluations, Patton specifically created a checklist that may be of value in making sure that evaluations are useful. www.wmich.edu/evalctr/checklists/ufe.pdf It might need to be adapted for use in your specific setting, but it definitely asks a couple of pretty critical questions about our evaluations.

Go ahead… I dare you to read up more about UFE (Utilization Focused Evaluation) and not be excited about the possibilities that evaluation has!

Have A good day!

PS. I hope to post some more of my thoughts on the AfrEA conference over the next month or so!

IOCE

2007-01-04T11:53:00.000+02:00

The IOCE is an international organisation for cooperation in evaluation and they have a couple of neat resources on their website:

http://www.ioce.net/resources/reports.shtml

•	The World Bank's Independent Evaluation Group Finds Progress On Growth, But Stronger Actions Needed For Sustainable Poverty Reduction

•	The World Bank's Independent Evaluation Group (IEG) is releasing its 2006 Annual Report on Operations Evaluation (AROE)

•	Joint UNICEF/IPEN Evaluation Working Paper on "New trends in development evaluation"

•	Resources for Evaluation and Social Research Methods

•	What Constitutes Credible Evidence in Evaluation and Applied Research?	>>

•	When Will We Ever Learn: Recommendations to Improve Social Development through Enhanced Impact Evaluation

Very very usable Evaluation Journal - AJE

2007-01-04T10:04:00.000+02:00

I’ve just paged through the December 2006 issue of the American Journal of Evaluation, and once again I am impressed.

It is such a usable journal for practitioners like myself, whilst still balancing it with the academic requirements that a journal should have. They do this by including

Articles – That deal with topics applicable to the broad field of program evaluation
Forum Pieces – A section were people get to present opinions and professional judgments relating to the philosophical, ethical and practical dilemmas of our profession.
Exemplars - Interviews with practitioners whose work can demonstrate in a specific evaluation study, the application of different models, theories and priciples described in evaluation literature.
Historical Record - Important turning points within the profession is analyzed, or historically significant evaluation works are discussed.
Method Notes – Which includes shorter papers describing methods and techniques that can improve evaluation practice.
Book Reviews – Recent books applicable to the broad field of program evaluation are reviewed.

I receive this journal as part of my membership to the American Evaluation Association – at a fraction of the costs that buying the publication on its own would have.

Go ahead – try it out – Here is a link to its archive:

http://aje.sagepub.com/archive/

Log Frame Training - Common Challenges

2006-12-05T13:55:00.000+02:00

I recently again facilitated a logframe workshop where I oriented the managers of intervention programmes towards basic log frame concepts and the idea of indicators, targets and means of verification.

Most people seem to intuitively grasp what we are trying to achieve when we present the workshop, and most take well to the assumption that it allows for better planning, monitoring and evaluation, but a set of common challenges seem to arise. In this entry I highlight three of these challenges and give an idea of how I try to get around them. I would be interested to hear from anyone how they approach these.

CHALLENGE 1:
Participants find it difficult to distinguish outputs, outcomes and impacts from one another. Even if we give them plenty of examples, clear definitions and an opportunity to practice their identification of the different kinds of results, they still find it difficult to correctly place these in the results chain. It is not absolutely crucial that they are placed correctly, but it definitely helps when you are later developing and indicators matrix. To help participants, I often give the following explanation

Outputs are what your programme delivers and are often a tangible indication that some activity was completed.
Outcomes are the changes you hope to see in the behaviour / skill / knowledge / values / attitudes of those you interact with in the shorter term.
Impacts are the other organisational and longer term changes you hope to see as a result of changed behaviour / skills / knowledge / values / attitudes of the participants.

And then I demonstrate it with a tomato plant example:

Planting tomato seeds, fertilising them and watering the soil are likely to result in a number of green sprouts emerging as a direct result of your "intervention". These aren't yet the tomatoes, but they tell you that you that some activity was completed and that you are possibly on your way to some sort of meaningful result (Output).
Having big fat red juicy tomatoes harvested tells you that you have achieved something - the seeds changed into something more useful (Outcome).
If you are able to eat your tomatoes and enhance your nutrition or if you sell the tomatoes to supplement your income, these are impact level results (Impact).

CHALLENGE 2:

When participants do a problem analysis they tend to accurately identify the level of intervention required to actually solve a problem. But when it comes to planning the intervention, they loose sight of the magnitude of the problem (despite being encouraged to go back to the problem analysis) and rather focus on the practicalities as it relates to their current organisational strength. So they agree to do two three hour workshops per term because that is all that they can manage. I often have to point them to the example again to get them to understand the effect of this kind of programming:

In the tomato plant example this equates to agreeing to water the tomatoes only once a month because that is all you have the time and staff for. And it obviously could lead to a reduction in the benefits gained.

CHALLENGE 3:

When developing a Log Frame Indicators Matrix, people have great difficulty in ensuring that the indicator , target and means of verification align well.

They might talk about the number of something in the indicator and put a percentage in the target. E.g. Indicator: The number of indicators that complete the course. Target: 90% of all Educators
They might talk about an increase in performance when they phrase the indicators, but only refer to a single measurement opportunity in the means of verification without any baseline data available. E.g. Indicator: Increase in learner performance on literacy test. Target: 80% of learners must pass. Means of Verification: End of year test

I have used the the following "recipe" with some success.

Appropriate targets if your indicator says something about an
INCREASE / IMPROVEMENT IN
–Number of people with skill / knowledge / appropriate behaviour
*e.g. 20% more people ….
–The knowledge / skill / quality level at which your participants can do something
*e.g. Average knowledge score increases with x%
- The number of people achieving a certain standard increases (e.x. pass, expemption)

* e.g. Number of persons passing increases with 20% over baseline

NOTE If you speak about an increase / improvement in your indicator / target your means of verification presupposes that knowledge about the baseline conditions and at least one other period in time will be required.

Appropriate targets if your indicator says something about achieving a
MINIMUM STANDARD
–Number of people achieving the minimum standard

*e.g. 80% of people must at least pass / get 80%
NOTE: This could be measured at a single instance only

Appropriate targets if your indicator says something about establishing SOMETHING NEW

–Number of people doing / showing something new

* e.g. 125 people must submit a business plan to COMSA
–The frequency with which people do something new

*e.g. Teachers to include open-ended questioning at least once in all observed lessons

Note: This could be measured at a single instance only.

My Impressions: UKES / EES Conference 2006

2006-10-18T15:36:00.000+02:00

The first joint UKES (United Kingdom Evaluation Society) and EES (European Evaluation Society) evaluation conference was held at the beginning of October in London. It was attended by approximately 550 participants from over 50 countries – Also a number of the prominent thinkers in M&E from North America. Approximately 15 South Africans attended the conference and approximately 300 papers were presented in the 6 streams of the conference. The official conference website is at: http://www.profbriefings.co.uk/EISCC2006/

Although it is impossible to summarise even a representative selection of what was said at the conference, I was struck by particularly discussions around the following:

How North-South and West-East evaluation relationships can be improved.
A panel discussion was held on this topic where a representative from the UKES, IOCE (International Organisation for Cooperation in Evaluation), and IDEAS (International Development Evaluation Association) gave some input followed by a vigorous discussion about what should be done to improve relationships. The international organizations used this as an opportunity to find out what could be done in terms of capacity building, advocacy, sharing of experiences and representation in major evaluation dialogues (e.g. the Paris declaration on Aid effectiveness[1]) etc. Like the South African Association, these associations also run on the resources made available by volunteers so the scope of activities that can be started is limited. The need for finding high-yield, quick gains was explored.
Filling the empty chairs around the evaluation table
Elliott Stern (Past president of UKES / EES and Editor of “Evaluation”) made the point that many of the evaluations done are not done by people that typically identify with the identity of an evaluator - Think specifically of Economists. Not having them represented when we talk about evaluation and how it should be improved means that they miss out on the current dialogue, and we don’t get an opportunity to learn from their perspectives.

Importance of developing a programme theory regarding evaluations
When we evaluate programmes and policies we recognize that clarifying the programme theory can help to clarify what exactly we expect to happen. One of the biggest challenges in the evaluation field is making sure that evaluations are used in the decision-making processes. Developing a programme theory regarding evaluations can help us to clarify what the actions are that’s required to ensure that change happens after the evaluation is completed. When we think of evaluation in this way, it is emphasized once again that delivering and presenting a report only cannot reasonably be expected to impact the way in which a programme is implemented. More research is required to establish exactly under which conditions a set of specific activities will lead to evaluation use.

Research about Evaluation is required so that we can have better theories on Evaluation
Steward Donaldson & Christina Christie (Claremont Graduate University) and a couple of other speakers were quite adamant that if “Evaluation” wants to be taken seriously as a field, we need more research to develop theories that go beyond only telling us how to do evaluations. Internationally evaluation is being recognized as a Meta-Discipline and a Profession, but as a field of science we really don’t have a lot of research about evaluation. Our theories are more likely to tell us how to do evaluations and what tool sets to use, but we have very little objective evidence that one way of doing evaluations is better or produce better results than another.

Theory of Evaluation might develop in some interesting ways
There was also talk about some likely future advances in evaluation theory. Melvin Mark (Current AEA President) said that looking for one comprehensive theory of evaluation is probably not going to deliver results. Different theories are useful under different circumstance. What we should aim for are more contingency theories that tell us when to do what. Current examples of contingency theories include Patton’s Utilization Focused Evaluation Approach - The intended use by intended users determines what kind of evaluation will be done. Theories that take into account the phase of implementation is also critically important. More theories on specific content areas are likely to be very useful e.g. evaluation influence, stakeholder engagement etc. Bill Trochim (President-Elect of AEA) presented a paper on Evolutionary Evaluation that was quite thought provoking and continued from thinking of Donald Campbell etc.

Evaluation for Accountability
Baronness Onora O’Neill (President of the British Academy) expanded what accountability through evaluation means by expanding on the question “Who should be held accountable and by whom?” She indicated that evaluation is but one of a range of activities that’s required to keep governments and their agencies accountable, yet a very critical one. The issue of evaluation for accountability was also echoed by other speakers like Sulley Gariba (From Ghana, previous president of IDEAS) with vivid descriptions of how the African Peer Review Mechanism could be seen as one such type of evaluation that delivers results when communicated to the critical audience.

Evidence Based Policy Making / Programming
Since the European Commission an the OECD was well represented, many of the presentations focused on / touched on topics relating to evidence based policy making. The DAC principles of Evaluation for development assistance (namely Relevance, Effectiveness, Efficiency, Impact, Sustainability) seems to be quite entrenched in evaluation systems, but innovative and useful ways of measuring impact level results was explored by quite some speakers.

Interesting Resources
Some interesting resources that I learned of during the conference include:
www.evalsed.com An online resource of the European Union for the evaluation of Socio-economic development.
The SAGE Handbook of Evaluation Edited by Ian Shaw, Jennifer Green, and Melvin Mark. More information at: http://www.sagepub.com/booksProdDesc.nav?prodId=Book217583
Encyclopedia for Evaluation edited by Sandra Mathisson. More info at: http://www.sagepub.com/booksProdDesc.nav?prodId=Book220777 Other Guidelines for good practice: http://www.evaluation.org.uk/Pub_library/Good_Practice.htm
[1] For more info about the Paris Declaration look at http://www.oecd.org/document/18/0,2340,en_2649_3236398_35401554_1_1_1_1,00.html

Social Entrepreneurship

2006-10-18T14:52:00.000+02:00

I’ve got a bee in my bonnet. And I must admit, I don’t quite know what to do with it. It probably has something to do with all of those systems-theory lectures I had at university. Here it is: We know the world and what happens in it cannot necessarily be explained in a linear fashion. So why, oh why do we plan and evaluate ALL our projects according to the logic model (where the combination of A, B and C under conditions D and E will produce F, G and H)? – then again… maybe it is just me and other people (Maybe I should Ask Bob Williams… he’s a real systems guy!) already have very nicely functioning alternative toolsets and methods to evaluate the non-linear world. (If you happen to be one of them, please come and save me from my ignorance and leave a comment so that I can learn from you)

I’m not proposing that we throw out that approach totally. But really! Given the scope of the developmental challenges we have here in SA, we must really hope for a miracle if we think that our logically planned out projects are going to solve all of our problems. If we have a little faith in the fact that we live in a chaotic system that has the capacity for self-organisation, we might actually want to start planning our interventions in a way that empowers key agents in the system to go out and do a number of unexpected and hopefully amazing things.

A related question: Why do we ONLY fund and evaluate projects and organizations, when it is people that make the difference? Let me clarify, I’m not saying projects and organizations don’t make a difference… But it is the 79 year old lady that decides to do something for the kids of her community on one special day. It is the social worker who thinks of a way to take the extra food off our tables and find a way to distribute it to those who need it… It is the guy who drives past the men on the side of the road that suddenly thinks of a way to provide tools and job opportunities to them.

These special people – “Social Entrepreneurs” I think they are called - Should be funded to do what they do best – think of ideas, implement them and set up structures. Because lo and behold they start worrying about how to put dinner on the table and abandon their potentially brilliant idea to take a desk job somewhere! This is apparently exactly what Ashoka does. See their website for more information: http://www.ashoka.org/africa

When venture capital investors want to invest in a new and innovative idea the majority of their pre-assessment work is around the individual that is pitching the idea. Some people just have the diversity of networks, skills and resources at their disposal to make things happen. Maybe there is some lesson in this for us!