M&E Blog

A lonely brainstorm... Or many minds?

2009-11-26T23:04:00.000+02:00

A grantmaking organization (our client) is interested in evaluating the level of their service delivery and relationship management – as perceived by the grantees that they disburse funds to. So here is the question – What are the evaluation standards that we should use?

Grantee perceptions?
The terms of reference indicates that the client expects that the evaluators will interact with the grantees to answer their questions. But if we ask grantees what they think of the grant maker’s processes, approach, involvement, communication etc. we might get senseless data because the wide range of grantees will have very different expectations about what qualifies as good service delivery / relationship management. It will probably be easy to collect data about their perceptions, but that won’t be very useful. And then there is also the issue of possible bias: Those grantees that experienced difficulty in submitting reports etc for monitoring purposes, might actually be slightly more negative than the rest of the grantees that would probably be eager to be complement the people that will dish out their next pay check.

The Grantmakers’ own standards?
It might make sense to determine whether the grantmaker has any implicit or explicit service delivery standards or contracted agreements that could be used as the standard to evaluate their performance against. But if the grantmaker has a standard that says: “All applications must be acknowledged in writing within 6 months from the date of receipt” that would be easy to check, but surely that service standard seems a little odd? Does it really take six months to respond to a submission?

Industry standards and benchmarks?
The alternative would be to look at service delivery standards and benchmarks as set by other industry players. There’s lots of literature about grantmaking internationally, but information about South African grantmakers are limited – There is the CSI handbook, but it doesn’t contain the level of detail that may be required to develop an extensive set of evaluation standards and benchmarks. And grant makers are notoriously secretive about their approach, systems and quality standards, so we will probably not be able to get detailed information from more than a handful of players in the field that we have established past relationships with.

Room for a participatory agreement on what exactly should be measured?
It is possible that a rigorous engagement of grantees and grant makers at the outset of the evaluation could provide the most satisfactory solution to the “which standards should we use” question. And that is probably just what we will do! Background research about all of the above will probably provide a good basis to start the workshop, but it will be interesting to see what the final consensus will dictate!

Visualization Methods - Really really interesting

2009-10-26T10:23:00.000+02:00

Previously I wrote about Edward Tufte's Book on presenting graphs. Well, it seems that data visualization has been taken to a whole new level.

Ralph Lengler & Martin J. Eppler form the Institute of Corporate Communication compiled a "Periodic table" of visualization methods that categorizes and shows examples of about 100 visualization methods.

The table can be downloaded in pdf format at:
http://www.visual-literacy.org/periodic_table/periodic_table_as_pdf.pdf

But try the online version - As you mouse over the various "elements" an example pops up to demonstrate what it looks like.
http://ow.ly/v9RI

The full article explaining the table can be found at
http://ow.ly/wk7d

Wow - It takes people specializing in visualization methods to think of such an innovative way to present their concept.

PS. I heard about this on the American Evaluation Assocation's Linked in Group.This and other useful information gets shared from time to time.

Get involved - SAMEA is preparing a submission

2009-10-12T17:13:00.000+02:00

Media statement by the Minister in the Presidency T Manuel for National Planning on the release of the Green Paper on National Strategic Planning
4 September 2009

Today government is releasing two discussion documents, one a Green Paper on National Strategic Planning and the other a Policy Document on Performance Monitoring and Evaluation. The decision by President Zuma to appoint Ministers in the Presidency responsible for National Planning and Performance Monitoring and Evaluation is designed to improve the overall effectiveness of government, enabling government to better meets its development objectives in both the short- and longer-term. These two discussion documents must be seen in the context of wider efforts led by the President to improve the performance of government through enhancing coherence and co-ordination in government, managing the performance of the state and communicating better with the public.
The Green Paper on National Strategic Planning is a discussion document that outlines the tasks of the national planning function, broadly defined. It deals with the concept of national strategic planning, as well as processes and structures. Once consultations on these issues have been completed, the process to set up the high-level structures will commence; and this will be followed by intense work to develop South Africa's long-term vision and other outputs. In other words, the Green Paper does not deal with these substantive issues of content.
The rationale for planning is that government (and indeed the nation at large) requires a longer-term perspective to enhance policy coherence and to help guide shorter term policy trade-offs. The development of a long-term plan for the country will help government departments and entities across all the spheres of government to develop programmes and operational plans to meet society’s broader developmental objectives. Such a plan must articulate the type of society we seek to create and outline the path towards a more inclusive society where the fruits of development benefit all South Africans, particularly the poor.
The planning function is to be coordinated by the Minister in The Presidency for National Planning. There are four key outputs of the planning function. Firstly, to develop a long term vision for South Africa, Vision 2025, which would be an articulation of our national aspirations regarding the society we seek and which would help us confront the key challenges and trade-offs required to achieve those goals. A National Planning Commission comprising of external commissioners who are experts in relevant fields would play a key role in developing this plan. The development of a National Plan would require broader societal consultation and existing forums would be used for this purpose. The Minister in The Presidency will co-ordinate these engagements. A National Plan has to be adopted by Cabinet for it to have the force of a government plan. The Minister would serve as a link between the Commission and Government, feeding the work of the Commission into government.
The next set of outputs cover the five-yearly Medium Term Strategic Framework (MTSF) and the National Programme of Action. These are documents of national government, adopted by Cabinet, drawing on the electoral mandate of the government of the day. The Minister in The Presidency for National Planning, supported by a Ministerial Committee on Planning, would coordinate the development of these documents with input from Ministers, departments, provinces, organised local government, public entities and coordinating clusters.
Further, it is envisaged that the planning function in The Presidency will undertake research and release discussion papers on a range of topics that impact on long-term development. These include topics such as demographic trends, global climate change, human resource development, and future energy mix and food security. The Presidency would also release and process baseline data on critical such as demographics, biodiversity as well as migratory and economic trends. This work will be undertaken by the Minister, working with the National Planning Commission (NPC) and the Minister, working with the NPC would, from time to time, advise government on progress in implementing the national plan, including the identification of institutional and other blockages to its implementation.
One of the functions of The Presidency in respect of national planning is to develop frameworks for spatial planning that seek to undo the damage that apartheid's spatial development patterns have wrought on our society. This includes the development of high level frameworks to guide regional planning and infrastructure investment.
The national planning function will provide guidance on the allocation of resources and in the development of departmental, sectoral, provincial and municipal plans.
The Minister in The Presidency responsible for national planning will be supported by a Planning Secretariat, which will also provide administrative, research and other support to the National Planning Commission. National Strategic Planning is an iterative process involving extensive consultation and engagement within government and with broader society.
It is envisaged that Parliament will play a key role in guiding the planning function through its oversight role but also through facilitating broader stakeholder input into the planning process. For this reason, it is appropriate that Parliament should lead the discussion process on the Green Paper.
This Green Paper is a discussion document. Government welcomes comment, advice, criticisms and suggestions from all in society.
Please address all comments on the Green Paper on National Strategic Planning to the Minister in the Presidency for National Planning c/o:
Hassen Mohamed
E-mail: hassen@po.gov.za
Tel: 012 300 5455
Fax: 086 683 5455
Issued by: The Presidency
4 September 2009

Please see http://www.info.gov.za/speeches/2009/09090414151003.htm for the actual green paper and Policy document on performance monitoring and evaluation.

Check out the GOODs

2009-10-05T12:25:00.000+02:00

A colleague referred me to a refreshing website that might be interesting to do-gooders the world over. It is called GOOD.

Maybe they called it GOOD because it can be found at the following url: http://www.good.is/. Apparently "GOOD is a collaboration of individuals, businesses, and nonprofits pushing the world forward"

Maybe they called it GOOD because it is good. I remind you, dear reader, that I am an evaluator so I should - according to the Scrivenian* wisdom I sometimes subscribe to - be particularly well placed to pass judgements about merit and worth. However, I will reserve judgement about the Goodness of GOOD for now. Except for saying what I have already said about it.

There is an interesting blog about Innovation and Evaluation in philanthropy. See

http://www.good.is/post/innovation-and-evaluation-are-inseparable/

*OK, that only sounded GOOD in my head, but the meaning I'm hoping to convey is that Michael Scriven's writings are relevant here.

ELDIS RESOURCE

2009-09-29T15:24:00.000+02:00

http://www.eldis.org/go/topics/resource-guides/manuals-and-toolkits/monitoring-and-evaluation

The Eldis Community site enables development professionals across the world to debate, discuss and exchange ideas and information. This community group, focusing on results-based M&E, is composed of development evaluation practitioners committed to evaluation capacity building at all levels of human development activities - global, country or community level; policy, programme or project level - with the aim of bringing about an equitable, accountable and progressive society for everyone.

Social Capital is a fundamental requirement for associations to work.

2009-03-18T17:40:00.005+02:00

The IOCE has an EvaLeaders listserve which aims to connect key people across the worlds' evaluation associations. I took up the task of trying to think of something to do to get the discussion going. We settled for a "monthly discussion question" and after posting the first of the questions, we were met with a resounding silence.

A variety of hypotheses were shared in order to explain the silence, the most interesting one:

"Our first question assumed that those on the EvaLeaders list share a sense of community with leaders of other IOCE member evaluation associations, and thus would be willing to take the time to write something about what their group is up to... the reality check is that there is a long-term process involved".

Concepts like "Evaluation Community" and "Community of Practice" are frequently used when speaking about Evaluation Associations, but I certainly have not sat down to think of what this actually means in practice. I have not really come to terms with the fact that social capital is inherent in working networks... capital in all shapes and sizes are requried for a network to work. In a working network, more social capital is also easily created.

Evaluation Associations are social networks, and although we typically evaluate an association's effectiveness by the number of activities they present and by the size of their membership, the true value of an association is actually in the strength of the links between members. Its these links that make shared values and common activities possible. If something as abstract as "hapiness" can dynamically spread through social networks*, then surely values, knowledge and a whole host of other fuzzy, yet potentially important evaluation-aligned attributes can be transferred too.

The question is: How do you get the minimum social capital together to start a vibrant network? Are there social-capital loans available from the World Bank? How many in-kind donations would be required? :)

I'm afraid I have more questions than answers to ponder...

*"Dynamic spread of happiness in a large social network: longitudinal analysis ver 20 years in the Framingham Heart Study" written by James Fowled and Nicholas Christakis. (BMJ 2008;337:a2338 doi:10.1136/bmj.a2338)

Participatory Evaluation Design

2009-03-09T16:03:00.004+02:00

I'm planning an evaluation planning meeting during which the intended evaluation users will design an organizational capacity evaluation. The organizations under scrutiny deliver services to the disabled (Or is the correct term "differently Abled"?). We will start with "drawing the road" (Ross Connor recently did a presentation on this at the Lisbon EES Conference) followed by the development of a stakeholder map, clarification of evaluation questions and the development of an evaluation matrix.

The evaluation matrix will outline the final evaluation questions, indicate which stakeholder need it addresses, and will also identify the data collection method and source. As a quality control exercise I'm planning to give the team a checklist that would ask the members whether the planned data collection meets some basic evaluation principles.

Some of the principles that I will try to incorporate:
• Independence: You cannot ask the same person in whose compliance you are interested, whether they are complying. The incentive to provide false information might be very high. You can ask school principals about the degree to which the Province has met their commitments, and you can ask parents whether the school charges money, but you cannot ask the school principal whether they are charging school fees if they have been declared a no-fee school.
• Relevance: Appropriate questions must be asked. You cannot expect a member of the general public (e.g. a parent) if the school is complying with the school funding norms – He / she is unlikely to know what these entail.
• Consider Systemic Impacts. Look broader than just the cases directly affected. No fee schools are not the only ones likely to be impacted by this specific policy provision. The schools in the area are also likely to be affected in some way.
• Appropriate Samples need to be selected. The sampling approach, sample size are all related to the question that needs to be answered.
• Appropriate methods need to be selected. Although certain designs are likely to results in easy answers, they might not be appropriate
• Implementation Phase: Take into account the level of implementation when you do the assessment. It is well known that after initial implementation an implementation dip might occur. Do not try to do an impact assessment when the level of implementation has not yet stabilised in the system.
• Fidelity: Take into account the fidelity of implementation, i.e to what degree the policy was implemented as it was intended.
• Quality Focus: Although a specific funding policy might have as a major aim to improve access to services, quality should always be a consideration. It is no use you have increased access to a service that never before delivered quality outputs, outcomes and impacts. Similarly it is no use that access to a good quality service improved, but due to the increased up-take of the service, the quality were negatively impacted.

I'll provide some feedback after the workshop

GDE Colloquium on their M&E Framework

2008-11-28T12:52:00.002+02:00

Recently the Gauteng Department of Education held a colloquium on their Monitoring and Evaluation Framework. As one of the speakers, I reflected on the fact that M&E frameworks often erroneously assume that the evaluand is a stable system. I argued that there are multiple triggers that leads to the evolution of the evaluand and that this has implications for M&E.

Triggers for evolving systems, organizations, policies, programmes & interventions
(Morell, J.A. (2005). Why are there unintended consequences of program action, and what are the implications for doing evaluation? In American Journal of Evaluation 2005 (26) p 444 - 463 )
• Unforeseen consequences
– Weak application of analytical frameworks, failure to capture experience of past research
• Unforeseeable consequences
– Changing environments
• Overlooked consequences
– Known consequences are ignored for practical, political or ideological reasons
• Learning & Adapting
– As implementation happens, the learning is used to adapt
• Selection Effects
– If different approaches are tried, those that are successful are likely to be replicated and those that are unsuccessful are unlikely to be replicated.

Implications for M&E
• M&E needs to work in aid of evolution (not just change)
– The M&E framework should be key in allowing the GDE to adapt, learn, respond to changes
• Not just by ensuring that the right information is tracked, but to ensure that the right people have access to it at the right time.
• M&E needs to respond to evolution
– As the Evaluand changes, some indicators will be incorrectly focused or missing, so the framework will have to be updated periodically
– It might be necessary to implement measures that go beyond checking “whether the Dept makes progress towards reaching its goals and objectives”
• Diversity of input into the design of the framework
• Using appropriate evaluation methods
– Consider expected impacts of change in planning for roll-out of M&E

Critical analysis of an M&E framework
• Does it ask the right questions in order for us to judge the merit, worth or value” of that which we are monitoring / evaluating?
• Does it allow for credible & reliable evidence to be used?

Types of Questions to ask
(Chelimsky, E. (2007). Factors Influencing the Choice of Methods in Federal Evaluation Practice. New Directions for Evaluation 113. p 13 - 33)

• Descriptive questions: Questions that focus on determining how many, what proportion etc. for the purposes of describing some aspect of the education context. (e.g. if you were interested in finding out what the drop out rate for no-fee schools is)
• Normative questions: Questions that compare outcomes of an intervention (such as the implementation of new policies) against a pre-existing standard or norm. Norm referenced questions can use various standards to compare against:
– Previous measures for the group that’s exposed to the policy intervention (e.g. if you compare the current drop-out rate to the previous drop-out rate for a specific set of schools affected by the policy)
– A widely negotiated and accepted standard (e.g. if it was accepted that a 5% drop out rate is acceptable, you can check whether the schools currently have that drop-out rate or not)
– Measure from another similar group (e.g. if you compare the drop-out rate for different types of schools)
• Attributive questions: Questions that attempt to attribute outcomes directly to an intervention like a policy change or a programme (Is the change in the drop-out rate in no-fee schools due to the implementation of the no-fee school policy)
• Analytic-Interpretive questions that builds our Knowledge base: Questions that ask about the state of the debate issues important for decision making about specific policies. (e.g. What is known about the relationship between drop-out rate and the per-learner education spend of the Department of Education)

Questions at different Time Periods
• Prior to implementation:
– Q1.1: What does available baseline data tell us about the current situation in the entities that will be affected? (Descriptive)
– Q1.2: Given what we know about existing circumstances and the changes proposed when the new policy / programme is implemented, what are the likely impacts/ effects likely to be? (Analytic-Interpretive, Normative)
• Evidence based policy making requires some sort of ex-ante assessment of the likely changes. This assessment can then later be referred to again when the final impact evaluation is conducted.

• Directly after implementation, and continued until full compliance is reached:
– Q2.1: To what degree is there compliance to the policy / fidelity to the programme design? (Descriptive)
– Q2.2: What are the short term positive and negative effects of the policy change / programme? (Descriptive, Normative and Attributive)
– Q2.3: How can the implementation and compliance be improved? (Analytic-Interpretive)
– Q2.4: How can the negative short term effects be mitigated? (Analytic-Interpretive)
– Q2.5: How can the positive short term effects be bolstered? (Analytic-Interpretive)
• This is important because no impact assessment can be done if the policy / programme has not been implemented properly, if there are significant barriers to the implementation of the policy / programme an intervention to remove these barriers would be necessary or the policy / programme should be changed.

• After compliance has been reached and the longer term effects of the policy are able to be discerned:
– Q3.1: To what degree did the policy achieve what it set out to do? (Normative)
– Q3.2: What has been the longer term and systemic effects attributable to the policy change? (Descriptive, Normative, Attributive)
– Q3.3: How can the implementation be improved / negative effects be mitigated / positive effects be bolstered? (Analytic-Interpretive)
• This is important to demonstrate that policy change was effective in addressing the underlying issues initially requiring the policy change, and to check that no unintended perversions of the policy became implemented.

• Designs appropriate to Descriptive questions:
– CASE STUDY DESIGNS
– RAPID APPRAISAL DESIGNS
– GROUNDED THEORY DESIGNS
• Designs Appropriate to Analytic-Interpretive questions
– LITERATURE REVIEW
– MIXED METHOD DESIGNS
• Designs Appropriate to Normative questions
– TIME SERIES RESEARCH DESIGNS
• Designs Appropriate to Attributive questions
– EXPERIMENTAL DESIGNS
– QUASI-EXPERIMENTAL DESIGNS

Principles for Evidence Collection
• Independence: You cannot ask the same person in whose compliance you are interested, whether they are complying. The incentive to provide false information might be very high.
• Relevance: Appropriate questions must be asked of the right persons..
• Consider Systemic Impacts. Look broader than just the cases directly affected.
• Appropriate Samples need to be selected. The sampling approach, sample size are all related to the question that needs to be answered.
• Appropriate methods need to be selected. Although certain designs are likely to results in easy answers, they might not be appropriate
• Implementation Phase: Take into account the level of implementation when you do the assessment. It is well known that after initial implementation an implementation dip might occur. Do not try to do an impact assessment when the level of implementation has not yet stabilised in the system.
• Fidelity: Take into account the fidelity of implementation, i.e to what degree the policy was implemented as it was intended.

Metrics for Social Entrepreneurs

2008-05-08T13:56:00.002+02:00

I found this website aimed at social entrepreneurs quite useful.
http://www.socialedge.org/discussions/success-metrics/new-metrics-for-today-s-social-entrepreneurs/

It lists some approaches for measurement:
"Social entrepreneurs now have a smorgasbord of measurement methodologies to choose from in addition to developing project-specific metrics (i.e., families served, reduction in arrests, units built, jobs created). They include:

• Balanced Scorecard Methodology (New Profit Inc.)
• The Acumen-Mckinsey Scorecard (Acumen Fund)
• Social Return Assessment Scorecard (Pacific Community Ventures)
• AtKisson Compass Assessment for Investors (AtKisson)
• Poverty and Social Impact Analysis (World Bank)
• OASIS: Ongoing Assessment of Social Impacts (REDF)"

They also extract Five principles of metrics that are often mentioned in discussions

1. Do have a set of success metrics
Funders and investors want to know that you have a way of measuring your success.
2. Tailor your metrics to your mission
If you are running a non-profit, then focus on social impact; if you are running a for-profit, you need the third bottom line - ROI.
3. Measure what you can in real time, but understand that social change is often measurable only over a longer period.
Try to find polling and survey organizations that are measuring the long-term trends and use their free published data.
4. Learn about established methodologies for social measurements
Applying them will save you work, get better results, and signal investors that you are serious about metrics.
5. Look at the cost-benefit of your metrics
Determine what percentage of your operations should be reasonably dedicated to success measurement and set it aside in your proposal and operating budgets.

Personally, I think that the issue of Return on Investment is crucial for any social entrepreneur. You need to be able to prove to your donors that they are getting value for money - Too many times teachers are trained at the cost of training and astronaut.

Graphical Presentations

2008-04-02T16:00:00.005+02:00

I previously commented on Tufte's book about using graphical presentations properly. Here are some examples that you would not find in his book.

The Global Classroom - Direct from Claremont Graduate University

2008-02-14T09:44:00.003+02:00

This very useful resource was brought under my attention via the AEA and SAMEA listservs.

The panel discussions from Claremont Graduate University 's recent "What Works?" workshops are available for viewing online as part of our video library. Evaluators, foundation directors, academics, and leaders of successful for-profits and non-profits came together to discuss what really works when tackling important social problems.

To view this footage, visit us at: http://www.cgu.edu/pages/5243.asp.

To view many other talks on evaluation and topics in applied psychology, visit our full video library at http://www.cgu.edu/pages/4435.asp.

Taxonomy of Evaluation

2008-02-06T11:47:00.000+02:00

I found the Evaluation Webring's Taxonomy of Types, Approaches and Fields of Evaluation Quite Useful.

Category 1: Types of evaluation
Internal evaluation or self- evaluation
An evaluation carried out by members of the organisation(s) who are associated with the programme, intervention or activity to be evaluated.

Ex-ante evaluation or impact assessment
An assessment which seeks to predict the likelihood of achieving the intended results of a programme or intervention or to forecast its unintended effects. This is conducted before the programme or intervention is formally adopted or started. Common examples of ex-ante evaluation are environmental and/or social impact assessments and feasibility studies.

Mid-term or interim evaluation
An evaluation conducted half-way through the lifecycle of the programme or intervention to be evaluated. Monitoring An ongoing activity aimed at assessing whether the programme or intervention is implemented in a way that is consistent with its design and plan and is achieving its intended results.

Ex-post or summative evaluation
An evaluation which usually is conducted some time after the programme or intervention has been completed or fully implemented. Generally its purpose is to study how well the intervention served its aims, and to draw lessons for similar
interventions in the future.

Meta-evaluation
Two processes are often referred to as meta-evaluation: (1) the assessment by a third evaluator of evaluation reports prepared by other evaluators; and (2) the assessment of the performance of systems and processes of evaluation.

Formative evaluation
An evaluation which is designed to provide some early insights into a programme or intervention to inform management and staff about the components that are working and those that need to be changed in order to achieve the intended objectives.

Category 2: Evaluative approaches
Outcome evaluation
An evaluation which is focused on the change brought about by the programme or intervention to be evaluated or its results regarding the intended beneficiaries.
Impact evaluation An evaluation that focuses on the broad, longer-term impact or effects, whether intended or unintended, of a programme or intervention. It is usually done some time after the programme or intervention has been completed.

Performance evaluation
An analysis undertaken at a given point in time to compare actual performance with that planned in terms of both resource utilization and achievement of objectives. This is generally used to redirect efforts and resources and to redesign structures.

Participatory evaluation
An evaluation that actively involves all or selected stakeholders in the evaluation process. Different approaches involve varying degrees of participation, inclusion, capacity-building, ownership, etc.

Empowerment evaluation
An approach that aims to improve programs through using specific tools for assessing the planning, implementation and self-evaluation of programs, and by incorporating evaluation into a program or organization’s planning and management. It involves a high level of participation by stakeholders in the evaluation process and is guided by ten key principles.
Collaborative evaluation An evaluation which aims for a significant degree of collaboration or cooperation between evaluators and stakeholders.

Utilization-focused evaluation
A process that assists the primary intended users of an evaluation to select the most appropriate content, model, methods, and theory for the evaluation, focusing on their intended use of the evaluation. Use refers to how people apply evaluation findings and experience the evaluation process.

Feminist evaluation
An evaluation that commonly involves adapting or redesigning relevant evaluation theories and methodologies so that they are compatible with feminist theories and methodologies. Feminist evaluations aim to be inclusive and empowering for women in particular.

Theory-based evaluation
An evaluation based on the theories of change that underlie a given programme or intervention. Its major aim is to examine the extent to which these theories hold and to validate their underlying assumptions.

Most Significant Change
A form of participatory monitoring and evaluation which involves the collection and systematic review and analysis of change stories by panels of designated stakeholders or staff. It is mainly used to assess intermediate program impacts and outcomes.

Category 3) Fields of evaluation

Programme or project evaluation
The evaluation of a programme or project.

Policy evaluation
The evaluation of policies and procedures.

Evaluation of legislation
The evaluation of a piece of legislation.

Evaluation of technical assistance
The evaluation of technical assistance provided by international, bilateral or multilateral donors.

Organisation or institutional evaluation
An evaluation of an organization’s or other institution’s capacity for innovation and change. It involves examining its decision-making processes and organisational structures.

Proposal assessment
The assessment of bids presented by tenderers following a specific call for tenders/bids.
Financial audit The scrutiny of accounts of an organization or other institution against a set of standards.

Personnel evaluation
A systematic method of evaluating an employee’s or staff member’s performance. This involves tracking, evaluating and providing feedback in relation to specific predetermined standards which are consistent with the organization’s overall

Surveys - Should we believe them?

2008-02-05T00:05:00.000+02:00

There is a lot written about survey methodology as a tool in evaluation, but despite the easy and neat stats that they deliver, one should regard them with a little bit of skepticism, it seems.

Two stories to demonstrate the point:

According to a speaker on 702 talk radio I heard earlier this week, Volkskas bank still receives votes for one of the best brands in South Africa (in the Annual Markinor survey), despite the fact that it has ceased existence now for more than just a couple of years. At least in this survey, you can identify problematic answers because survey respondents had the option of giving an open-ended answer. I shudder to think what people actually do when they get one of those tick box multiple choice surveys...

In the next example, it is just so clear that one should question even the most basic assumptions people make when they complete a survey.

http://news.yahoo.com/s/afp/20080204/wl_uk_afp/britainpeoplehistoryoffbeat_080204001239
LONDON (AFP) - Britons are losing their grip on reality, according to a poll out Monday which showed that nearly a quarter think Winston Churchill was a myth while the majority reckon Sherlock Holmes was real. The survey found that 47 percent thought the 12th century English king Richard the Lionheart was a myth. And 23 percent thought World War II prime minister Churchill was made up. The same percentage thought Crimean War nurse Florence Nightingale did not actually exist.Three percent thought Charles Dickens, one of Britain's most famous writers, is a work of fiction himself. Indian political leader Mahatma Gandhi and Battle of Waterloo victor the Duke of Wellington also appeared in the top 10 of people thought to be myths. Meanwhile, 58 percent thought Sir Arthur Conan Doyle's fictional detective Holmes actually existed; 33 percent thought the same of W. E. Johns' fictional pilot and adventurer Biggles.

competencies/capabilities of an evaluator

2008-01-09T04:49:00.000+02:00

Q: Do you know of a document that articulates competencies/capabilities of an evaluator? If you do please send me a reference or copy.

A: Lots of work has been done on this topic by various Evaluation Associations across the world. Some useful references:

King, Jean, Stevahn, Laurie, Ghere, Gail, & Minnema, Jane (2001). Toward a taxonomy of essential evaluator competencies. American Journal of Evaluation, 22, 229-247.

Mertens, Donna M. (1994). Training evaluators: Unique skills and knowledge. New Directions for Program Evaluation, 62, 17-27.

Treasure Board of Canada Competency Profile for Federal Public Service Evaluation Professionalshttp://www.tbs-sct.gc.ca/eval/dev/Professionalism/profession_e.asp

Here is the list:

Essential Competencies for Program Evaluators (ECPE)

(Stevahn and King, Ghere, & Minnema, American Journal of Evaluation, March

2005)

1.0 Professional Practice

1.1 Applies professional evaluation standards

1.2 Acts ethically and strives for integrity and honesty in conducting evaluations

1.3 Conveys personal evaluation approaches and skills to potential clients

1.4 Respects clients, respondents, program participants, and other stakeholders

1.5 Considers the general and public welfare in evaluation practice

1.6 Contributes to the knowledge base of evaluation

2.0 Systematic Inquiry

2.1 Understands the knowledge base of evaluation (terms, concepts, theories, assumptions)

2.2 Knowledgeable about quantitative methods

2.3 Knowledgeable about qualitative methods

2.4 Knowledgeable about mixed methods

2.5 Conducts literature reviews

2.6 Specifies program theory

2.7 Frames evaluation questions

2.8 Develops evaluation designs

2.9 Identifies data sources

2.10 Collects data

2.11 Assesses validity of data

2.12 Assesses reliability of data

2.13 Analyzes data

2.14 Interprets data

2.15 Makes judgments

2.16 Develops recommendations

2.17 Provides rationales for decisions throughout the evaluation

2.18 Reports evaluation procedures and results

2.19 Notes strengths and limitations of the evaluation

2.20 Conducts meta-evaluations

3.0 Situational Analysis

3.1 Describes the program

3.2 Determines program evaluability

3.3 Identifies the interests of relevant stakeholders

3.4 Serves the information needs of intended users

3.5 Addresses conflicts

3.6 Examines the organizational context of the evaluation

3.7 Analyzes the political considerations relevant to the evaluation

3.8 Attends to issues of evaluation use

3.9 Attends to issues of organizational change

3.10 Respects the uniqueness of the evaluation site and client

3.11 Remains open to input from others

3.12 Modifies the study as needed

4.0 Project Management

4.1 Responds to requests for proposals

4.2 Negotiates with clients before the evaluation begins

4.3 Writes formal agreements

4.4 Communicates with clients throughout the evaluation process

4.5 Budgets an evaluation

4.6 Justifies cost given information needs

4.7 Identifies needed resources for evaluation, such as information, expertise, personnel, instruments

4.8 Uses appropriate technology

4.9 Supervises others involved in conducting the evaluation

4.10 Trains others involved in conducting the evaluation

4.11 Conducts the evaluation in a nondisruptive manner

4.12 Presents work in a timely manner

5.0 Reflective Practice

5.1 Aware of self as an evaluator (knowledge, skills, dispositions)

5.2 Reflects on personal evaluation practice (competencies and areas for growth)

5.3 Pursues professional development in evaluation

5.4 Pursues professional development in relevant content areas

5.5 Builds professional relationships to enhance evaluation practice

6.0 Interpersonal Competence

6.1 Uses written communication skills

6.2 Uses verbal/listening communication skills

6.3 Uses negotiation skills

6.4 Uses conflict resolution skills

6.5 Facilitates constructive interpersonal interaction (teamwork, group facilitation, processing)

6.6 Demonstrates cross-cultural competence

M&E for Philanthropy

2007-12-21T00:40:00.000+02:00

The following shows what makes it so difficult to work with charities!

2 Young Hedge-Fund Veterans Stir Up the World of Philanthropy

By STEPHANIE STROM

Published: December 20, 2007

Holden Karnofsky and Elie Hassenfeld rank charities by analyzing the numbers in much the same way they did at their investment management company
http://www.nytimes.com/2007/12/20/us/20charity.html?ex=1355893200&en=a0d9a701ad60ffd0&ei=5124&partner=permalink&exprod=permalink

In the fall of 2006, they and six colleagues created what Mr. Karnofsky calls a “charity club.” Each member was assigned to research charities working in a specific field and report back on those that achieved the best results. They were stunned by the paucity of information they could collect.

“I got lots of marketing materials from the charities, which look nice, you know, pictures of sheep looking happy and children looking happy, but otherwise are pretty useless,” said Jason Rotenberg, a former member of the club and now a $50,000 donor to the Clear Fund. “It didn’t seem like a reasonable way of deciding between one charity and another.”

GiveWell’s findings are available on the Internet, without charge, at www.givewell.net. In evaluating charities, Mr. Karnofsky and Mr. Hassenfeld press them for information, analyzing the numbers in much the same way they did at Bridgewater. The Smile Train, for instance, a charity that repairs cleft palates, was asked how much it spent in each region and each country to treat how many patients in each.Many in the field question how long GiveWell can survive. While 34 percent of wealthy donors who responded to a survey sponsored by the Bank of America said they wanted more information on nonprofits, almost three-quarters said they would give more if charities spent less on administration. And collecting information is costly.

As a result, most philanthropic advisory services like GiveWell have a hard time raising money. The Clear Fund has raised $300,000 since its inception this year, about half of which has gone to operating GiveWell.

The problem is that you want charities to measure impact. Although the cost of measuring is sometimes prohibitive, the potential cost of not measuring should be enough reason to make sure that you do measure.

Probability Sampling Approaches

2007-11-12T00:31:00.000+02:00

Probability Sampling Approaches

Probability sampling approaches allow you to generalize to the full population, since it ensures that special random characteristics are likely to be distributed evenly across the units included and excluded in / from the sample. It therefore is likely to yield a less biased sample and the results could be said to apply to the full population (if the appropriate sample size was selected). Different kinds of probability sampling approaches are possible.

The figures below demonstrate the different approaches. Assume each number is a unique member of the population, assume that each group consists of discreet mutually exclusive members of the population (In columns) and assume that each cluster (delineated by a block) is a group of members in the same geographic area.

With simple random sampling the sample is selected from the whole population using a table of numbers. Note that this does not necessarily ensure balanced representation amongst different groups.

With stratified random sampling, a set number of participants from each group can be selected. Note that this does not necessarily ensure that the most economical approach is used. In the example some cases from almost all of the geographic clusters are included.

With cluster sampling, a set number of clusters are randomly selected (in this case 4) with a set number of randomly selected units within each cluster (in this case 5). Although this will be more economical in terms of fieldwork costs because travel to different clusters have been limited, it does not necessarily guarantee equal representation of groups.

With systematic sampling, a set pattern is systematically applied to select participants. In the case of the example above, every 11^th member of the population were selected. Note that it did not require a random table of numbers, but were still subject to the same limitations as the simple random sample.

Type of Probability Samples	When is it applicable	Drawbacks
Simple Random Sampling (I.e.randomly select 50 schools off a list with all schools in the country)	It is ideal for statistical purposes	· It may be difficult to achieve in practice · It requires a precise list of the whole population · It is costly to conduct as those sampled may be spread over a wide area.
Stratified Random Sampling (I.e. Randomly select 50 schools per strata such as province)	· It ensures better coverage of the population than simple random sampling. · It is administratively more convenient to stratify a sample – interviewers can be specifically trained to manage particular strata (e.g. age, gender, ethnic or language groups).	· Difficulty in identifying appropriate strata. · More complex to organize and analyse results
Cluster Sampling (I.e. split the schools in a province up in geographical clusters, select 10 clusters randomly, and then proceed to visit 20 schools within each cluster)	More cost effective in terms of travel, thereby producing a reduction in the overall cost	· Units in a cluster may be very similar and therefore are less likely to represent the whole population · Cluster sampling has a larger sampling error than simple random sampling.
Systematic Sampling (i.e. a set pattern is applied to the data set, e.g. every 11^th member is selected)	It spreads the sample more uniformly over the population and is easier to conduct than simple random sampling.	The system may interact with a concealed pattern in the population.

Local monitoring of Public Service Delivery

2007-11-08T11:48:00.000+02:00

I find the readings on the pelican listserve (Pelican Initiative: Platform for Evidence-based Learning & Communications for Social Change) always interesting. Today the message below was posted.

I find the description of Social Auditing quite interesting. At first I thought it might be similar to Social Accounting - the move of private companies to also account on the triple bottom line of Environmental, Social and Economical impacts (costs and benefits) in order to promote corporate responsibility. We've actually done a couple of sustainability reports using this framework. The focus, however is more on accountability than it is on changing things for improvement. Hope it is not the same for this kind of local government monitoring.

I have also previously posted something on the MSC techniques described here. I think it is quite useful if there are no clearly defined objectives to start off with, or if the social reality is very complex and requires more of a systems look at the effects. Any case. You decide for yourself!

PS. the AEA conference is on in Baltimore this week, and although I am not able to attend, my business partner is. It sounds like it is interesting and stimulating as always!

Ciao

B

**********************************************

Last month, a number of useful documents and experiences were shared by Gilles Mersadier, who is the coordinator of the FIDAfrique network (http://www.fidafrique.net/article413.html). Among the material that he shared, he referred to a methodology for capitalisation (the process of sharing experiences among and across organisations) that is being used in the context of 25 West-African rural development projects. To date, 20 of these projects have published different types of documents that are disseminated throughout the FIDAfrique network. The production of these documents, which are available on the network's website, are supported by a methodological guide on the capitalisation process (in French and
English):
http://www.fidafrique.net/article467.html?var_recherche=capitalization

In the past few weeks, different contributions have been sent in the context of the discussion around local monitoring of public service delivery. One of the questions which we posed in the beginning of this discussion focused on the types of approaches that are being used for local monitoring purposes. In this message, I would like to briefly describe two of such approaches:

1: Social Auditing
2: Most Significant Change

**** 1: Social Auditing

The Social Auditing method has been developed and used for participatory monitoring of public service delivery by the organisation CIET (for more info on the organisation, which started in 1985 in Mexico and developed into an international network, please visit http://www.ciet.org/en/aboutciet/ ).

The method's primary aim is to 'increase the informed interaction between communities and public services'. The impact, coverage and costs of public services are examined through a combination of quantitative (survey) and qualitative (key informant and focus groups) evidence. The civil society plays a central role in interpreting this evidence, and through this process contributes to the creation of local solutions. The use of both 'hard' and 'soft' evidence helps to provide a strong and accurate underpinning to the locally defined ideas and solutions, and as such strengthens the legitimacy of the solutions that are developed.

A report that was published in 2005 on the use of the method for assessing 'governance and delivery of public services' in Pakistan lists the following seven stages of a Social Audit cycle:
(1) Clarify the strategic focus;
(2) Design sample and instruments, pilot testing;
(3) Collect information from households on use and perception of public services;
(4) Link this with information from the public services;
(5) analyse the findings in a way that points to action;
(6) Take findings back to the communities for their views about how the improve the situation;
(7) Bring evidence and community voice into discussions between service providers, planners and community representatives to plan and implement changes.

The CIET website features an extensive library section where the reports of previous social audits can be accessed, together with other experiences relating to the network's central focus on the 'socialisation of
evidence':
http://www.ciet.org/en/browse/librarydocs/

**** 2: Most Significant Change

Central in the Most Significant Change (MSC) technique is - as the name suggests - the collection of significant change stories that emerge from the field level. Following the collection of these stories, those stories which are considered most significant are selected by panels of designated stakeholders or staff. The collection, discussion and further selection of the stories revolves around 'domains': the areas that the stakeholders collectively decide on as the focus of the monitoring. Provided that it is clear who is involved at selecting the stories at the different levels, the use of the domains makes for a transparent monitoring process. Given these characteristics, and the fact that it is relatively easy to learn how to use it, the technique can be useful in the context of local monitoring of public service delivery.

While an English guide about the technique has been available since 2005
(see: http://www.mande.co.uk/docs/MSCGuide.htm ), efforts have also been made to translate the guide into different international and local languages, including Spanish, French, Russian, Tamil and Indonesian. Rick Davies has set up a specific Web Log to make available these translations, and to allow users to share suggestions to further improve the quality of the translations. You can access this website here:

http://mscguide-translations.blogspot.com/

Our current discussion on the topic of local monitoring of public service delivery is moving towards an end, so it would be great if some of you could still share some experiences and ideas on this topic in case you did not yet have the time to do so. We will aim to send around a summary of the key points that have been contributed sometime next week.

Best wishes,
Niels

Handy Publication: New Trends in Evaluation

2007-04-25T09:21:00.000+02:00

You know how you always come back with a stack of stuff to read when you've attended an evaluation conference? It most cases the material just gets added to my ever growing "to read" pile. It is only once I start searching for something or decide to spring-clean, that I actually sit down and read some of the stuff. This morning I came across a copy of UNICEF / CEE/CIS and IPEN's New Trends in Evaluation.

What a delightfully simple straightforward publication - yet it packs so much relevant information between its two covers. I wish I had remembered about it last week when I lectured to students at UJ. Before I was able to get on with the lecture on Participatory M&E, I first had to explain how M&E is different and similar to Social Impact Assessments (In the sense of ex-ante Environmental Impact Assessment type assessment). I think it would have been a very handy introductory source to have.

The table of contents looks as follow:

1. Why Evaluate?
The evolution of the evaluation function
The status of the evaluation function worldwide
The importance of Evaluation Associations and Networks
THe oversight and M&E function
2. How to Evaluate?
Evaluation culture: a new approach to learning and change
Democracy and Evaluation
Democratic Approach to Evalution
3. Programme Evaluation Development in the CEE/CIS

But what is really useful is the Annexures:
Annex 1: Internet Bades Discussion Groups Relevant to Evaluation
Annex 2: Internet Websites Relevant to Evaluation
Annex 3: Evaluation Training and Reference Sources Available Online
Annex 4-1: UNEG Standards for Evaluation in the UN System
Annex 4-2: UNEG Norms for Evaluation in the UN System
Annex 5: What goes into a Temrs of Reference; UNICEF evaluation Technical Notes, Issue No.2

The good thing about this publication is that you can download it for free off the internet at

http://www.unicef.org/ceecis/New_trends_Dev_EValuation.pdf

An introductory blurb and a presentation is also available from the IOCE website.

http://ioce.net/news/news_articles/061023_unicef-ipen.shtml

What is an Evaluator?

2007-08-16T09:20:00.000+02:00

Do you find it difficult to explain to people what you do? Those magical two sentences that will get people to go "Aaaaaah, now I get what you do?" Unfortunately I have not been able to come up with something concrete yet. But I am still trying.

In a television interview to publicise the SAMEA conference, the DDG from the PSC Mr. Mash Dipofu tried to explain it with an example. He asked the TV presenter if he knew what the viewers thought of his programme, how it can be improved and how many people actually watches it. He explained that by answering these questions, you are doing what an evaluator would be doing and answering the underlying question: Does what I am doing have value?

Which got me thinking. So much of what we do as evaluators are also done by other professionals.

We are a little like investigative journalists: We talk to people and ask questions and gather information to make an argument for or argainst something. Sometimes to inform readers of some wrong doing... Sometimes we celebrate what has been achieved.
Then we are also a little like the weather guy. We collect numbers over a long period of time and by applying some statistical techniques we can start predicting what will happen in future.
Another way of looking at our job is to compare it to that of teachers. We guide people to learn from their environments - assuming that they need to be taught how to use the information at their disposal to make intelligent choices. We check with tests whether the intended result has been achieved... much like teachers check whether their students have mastered a skill or knowledge component.
And then of course evaluators are also a little like an auditor in the way that we try to prove to people that money has been well spent.

The problem with explaining to people what we do, is probably because people tend to confuse it with research and planning and implementation and all kinds of other things. I have also spent some time to think about how being an evaluator is different from being a researcher, a planner and an implementer.

Because we use research techniques to collect evidence in order to evaluate, the difference between being a researcher (who asks questions in a specific way to gather evidence) and an evaluator (who asks questions in a specific way to gather evidence to then make a value judgment about the evaluand) is sometimes a little difficult to explain. But there is a difference!
Planning, on the other hand comes quite naturally when you are an evaluator. After delivering an evaluation, I frequently get asked to assist in planning processes - if people value what you produced in the evaluation they want to make sure that they plan to implement the recommendations made. Being a weather guy and an auditor makes it easier to plan because you can draw info together to make predictions, and you know that you will have to explain to people why you chose to spend their money in a particular way.
I think evaluators will probably make terrible implementers. As an evaluator you are constantly asking questions: Is this the best way to do things? Will we achieve results? How would we know that we added value? How do we know that this is the best way forward? To implement, however, you sometimes have to say "Well I don't know all the answers but I am making a decision to do ABC in the following way and that is the way it is!"

Despite having useful analogies to explain what evaluators do and don't do, I think that we are at risk if we, as practitioners of a scientific metadiscipline, don't understand how the ideology underlying evaluation is different from other those informing other jobs. Evaluators might be sharing some commonalities with teachers, weather people, auditors and journalists, but we have different values and assumptions guiding our work. We cannot forget that our work probably has a deeply political nature because we have to choose at some stage whose questions we will have to answer.

I found the following useful bit about evaluation approaches and the underlying philosophy, epistemology and ontology at http://www.recipeland.com/facts/Evaluation

Classification of approaches

Two classifications of evaluation approaches by House House, E. R. (1978). Assumptions underlying evaluation models. Educational Researcher. 7(3), 4-12. and Stufflebeam & Webster Stufflebeam, D. L., & Webster, W. J. (1980). An analysis of alternative approaches to evaluation. Educational Evaluation and Policy Analysis. 2(3), 5-19. can be combined into a manageable number of approaches in terms of their unique and important underlying principles.

House considers all major evaluation approaches to be based on a common ideology, liberal democracy. Important principles of this ideology include freedom of choice, the uniqueness of the individual, and empirical inquiry grounded in objectivity. He also contends they all are based on subjectivist ethics, in which ethical conduct is based on the subjective or intuitive experience of an individual or group. One form of subjectivist ethics is utilitarian, in which â€œthe goodâ€ is determined by what maximizes some single, explicit interpretation of happiness for society as a whole. Another form of subjectivist ethics is intuitionist / pluralist, in which no single interpretation of â€œthe goodâ€ is assumed and these interpretations need not be explicitly stated nor justified.

These ethical positions have corresponding epistemologiesâ€”philosophies of obtaining knowledge. The objectivist epistemology is associated with the utilitarian ethic. In general, it is used to acquire knowledge capable of external verification (intersubjective agreement) through publicly inspectable methods and data. The subjectivist epistemology is associated with the intuitionist/pluralist ethic. It is used to acquire new knowledge based on existing personal knowledge and experiences that are (explicit) or are not (tacit) available for public inspection.

House further divides each epistemological approach by two main political perspectives. Approaches can take an elite perspective, focusing on the interests of managers and professionals. They also can take a mass perspective, focusing on consumers and participatory approaches.

Stufflebeam and Webster place approaches into one of three groups according to their orientation toward the role of values, an ethical consideration. The political orientation promotes a positive or negative view of an object regardless of what its value actually might be. They call this pseudo-evaluation. The questions orientation includes approaches that might or might not provide answers specifically related to the value of an object. They call this quasi-evaluation. The values orientation includes approaches primarily intended to determine the value of some object. They call this true evaluation.

http://www.recipeland.com/facts/Evaluation

Rant for today*

2007-07-11T03:34:00.000+02:00

A potential client sends out Terms of Reference requesting potential service providers to submit a quotation for an (5 maybe 10 person day) evaluation engagement. Note, they did not ask for a proposal, they asked for a quotation. Given the limited scope of the project, a quotation makes sense. So that is what I submit.

Then potential client reads through the proposals they received and *Horror* *shock* discovers that there isn’t enough information in the quotations to make a transparent decision. (Maybe there was a little problem with the Terms of Reference?) They then set up a meeting in which they expect potential service providers to present to a panel. “For the purposes of meeting with the potential evaluators and ensuring the selection process is fair and transparent”. (Have I mentioned that this is a small job?)

So I go and have a wonderful meeting with the potential client, but in the end do not get the job. My heart isn’t broken or anything. It would’ve been nice to get the job but… ah well, I think they looked for a content specialist rather than an evaluation specialist in the first place.

But hey, I am an evaluator, and evaluative thinking requires me to find out why I was not successful. So I write the “Thanks for the notification, we are disappointed, could you please tell us where our submission was weak… bla-di-blah” email. To which I don’t receive a response. I get my office manager to follow up and get a response of the kind: “We liked your presentation / Sorry we don’t have time to provide feedback / now please just leave us alone”.

What happened to the transparency and fairness thing? Am I unreasonable to think that a two line explanatory email is not much to ask after all of the trouble they put me through?

Ahem… The UK evaluation Society has some guidelines for persons when they commission evaluations at:

http://www.evaluation.org.uk/Pub_library/PRO2907%20-%20UKE.%20A5%20Guideline.pdf

Maybe more people should read that?

*Because this is my blog I get to complain here every now and again. I promise that this will not turn into those rant-upon-rant blogs, but I really need to get the following off my chest.

Whew! Its been like how long?

2007-07-11T03:23:00.001+02:00

I see I haven't posted anything here since April. That is probably because I have nothing to write about when I have time, and when I have time to write, I can't think of anything to write about. So here are a list of things I would like to address at some stage:

* What people think a focus group is and what it really is
* Rapid assessment methods - differences in approaches (I need to draw up a table based on that AJE article I read yesterday)
* When people should consider appointing an evaluation specialist rather than a content specialist for certain evaluations.
* If evaluators do strategic planning, what is it that they should know about planning AND if planners or anybody else does evaluations what is it they need to know about evaluations
* People say evaluation is a meta-discipline. Why do they say that.
*All this talk about accreditation is just confusing everybody. What does it mean and what about the international body of work that has been done on this aspect?

Presentation Presented at the SAMEA conference 26 - 30 March 2007

2007-04-10T00:27:00.000+02:00

Based on some of the ideas in previous blog entries, I presented the following presentation at the recent SAMEA conference.

Setting Indicators and Targets for Evaluation of Education Initiatives

Introduction

Good Evaluation Indicators and Targets are usually an important part of a robust Monitoring and Evaluation system.
Although evaluation indicators are usually considered as important, all evaluations do not have to make use of a set of pre-determined indicators and targets.
The most significant change (MSC) technique, for example, looks for stories of significant change amongst the beneficiaries of a programme, and after the fact uses a team of people to determine which of these stories represent MSC and real impact.
You have to include the story around the indicators in your evaluation reports in order to learn from the findings.

What do we mean?

The definition of an Indicator is: “A qualitative or quantitative reflection of a specific dimension of programme performance that is used to demonstrate performance / change”
It is distinguished from a Target which: “Specifies the milestones / benchmarks or extent to which the programme results must be achieved”
And it also different from a Measure which is: “The Tool / Protocol / Instrument / Gauge you use to assess performance”

Types of Indicators

The reason for using indicators is to feel the pulse of a project as it moves towards meeting its objectives or to see the extent to which it has been achieved. There are different types of indicators:
Risk/enabling indicators – external factors that contribute to a project’s success or failure. They include socio-economic and environmental factors, the operation and functioning of institutions, the legal system and socio-cultural practices.
Input indicators – also called ‘resource’ indicators, they relate to the resources devoted to a project or programme. Whilst they can flag potential challenges, they cannot, on their own determine whether a project will be a success or not.
Process indicators – also called ‘throughput’ or ‘activity’ indicators. They reflect delivery of resources devoted to a programme or project on an ongoing basis. They are the best indicators of implementation and are used for project monitoring.
Output indicators –indicates whether activities have taken place by considering the outputs from the activities.
Outcome indicators - indicates whether your activities delivered a positive outcome of some kind.
Impact indicators – Concerns the effectiveness, usually long term, of a programme or project as judged by the measurable achieved in improving the quality of life of beneficiaries or other similar impact level result.

Good Indicators

Good Performance Indicators should be
Direct (Does it measure Intended Result?)
Objective (Is it ambiguous?)
Adequate (Are you measuring enough?)
Quantitative (Numerical comparisons are less open to interpretation)
Disaggregated (Split up by gender, age, location etc.)
Practical (Can you measure it timeously and at reasonable cost?)
Reliable (How confidently can you make decisions about it?) (USAID, 1996)

SMART Indicators
Most people have also heard about SMART indicators:

Specific
Measurable
Action Oriented
Realistic
Timed

How we use indicators

For many of the evaluation initiatives that we help to plan M&E systems for, we usually work with the managers to set indicators that they understand and can use.
Although the issue of data availability and data quality is usually a big concern, it is often the indicators and targets that are set that could make or break an evaluation.

Case Study

Implementers of a teacher training initiative wants to know if their project is making a difference in the maths and science performance of learners.

Pitfalls

Alignment between Indicators & Targets (If the indicator says something about a number, then the target must also be couched in terms of a number, and not a percentage)
Averaging out things that do not belong together (i.e. maths and science) does not make sense at all.
Not disaggregating enough (Are you interested in all learners, or is it important to look at disaggregating your data by age group, gender, educator)
Assuming that all targets should be about an increase: (Sometimes a trend in the opposite direction exists and it is expected that your programme will only mediate the effects)
Assuming that an increase from 20% to 50% is the same as an average increase of 50% to 80%. (Psychometrists have used the standardised gain statistic for a very long time. It is interesting that we don’t see more of it in our programmes.)
Ignoring the statistics you will use in analysis: (In some cases you are using a sample and averages. This means an average increase might just look like an increase, but when you test for statistical significance it is actually not an increase)
Setting indicators that require two measurements where one would be enough (Are you interested in an average increase, or just the % of people that make some minimum standard.)
Ignoring other research done on the topic (If a small effect size is generally reported for interventions of these kinds, isn’t an increase of 30% over baseline a little ambitious?)
If you don’t have other research on the topic, it should be allowable to adjust the indicators.
Setting an indicator and target that assumes direct causality between the project activity and the anticipated outcome (Even if you have brilliant teachers, how must the learners perform if learners have nowhere to do homework, School discipline is non-existent and after learners have accumulated 10 years of conceptual deficits in their education?)
Ignoring Relevance, efficiency, sustainability, and equity considerations. (Is educators training really going to solve the most pressing need?If your programme makes a difference, is it at the same cost as training an astronaut?What will happen if the trained educator leaves?Does the educator training benefit rural learners in the same way in which it would benefit urban learners?)

Ways to address the pitfalls

Do a mock data exercise to see how your indicator and target could play out.
This will help you think through the data sources, the statistics, and the meaning of the indicator
Read extensively about similar projects to determine what the usual effect size is.
When you do your problem analysis, be sure to include other possible contributing factors, and don’t try to attribute change if it is not justifiable.
Look at examples of other indicators for similar programmes
Keep at it and work with someone who would be able to check your proposed indicators with a fresh eye.

Where to look
Example Indicators can be found in:

Project / Programme Evaluation reports from multi-lateral donor agencies
UNESCO Education For All Indicators
Long term donor-funded projects such as DDSP, QIP.
StatsSA publications and statistical extracts about the education sector.
Government M&E indicators.

Announcement from AEA about benefits

2007-03-07T09:14:00.000+02:00

The AEA announced that two more Journals are available to members!

Announcement: American Evaluation Association Expands Online Journal Access

AEA Members now receive electronic access to two additional journals - Evaluation and the Health Professions and Evaluation Review - in addition to continued access to AEA's own American Journal of Evaluation and New Directions for Evaluation, as part of membership benefits.

The journals' content is searchable, and archived online content goes back multiple years.

Individual subscriptions to each journal are over $100 each, making AEA membership more of a value then ever at only $80. Members also receive AJE and NDE in hardcopy, discounts on conference and training registration, regular communications about news from all corners of the evaluation community, discounts on books, and the opportunity to participate in the life of the association and the field.

Learn more about AEA and join online at: www.eval.org

Centre for Global Development

2007-02-01T04:42:00.000+02:00

This is the Centre for Global Development Report that upset so many people again. I mean really! Didn't we agree that mixed methods are the way to go? RCTs Cannot possibly be the answer for all our impact evaluation questions.

From the CGD website at:

http://www.cgdev.org/content/publications/detail/7973

When Will We Ever Learn? Improving Lives Through Impact Evaluation

05/31/2006

Each year billions of dollars are spent on thousands of programs to improve health, education and other social sector outcomes in the developing world. But very few programs benefit from studies that could determine whether or not they actually made a difference. This absence of evidence is an urgent problem: it not only wastes money but denies poor people crucial support to improve their lives.

This report by the Evaluation Gap Working Group provides a strategic solution to this problem addressing this gap, and systematically building evidence about what works in social development, proving it is possible to improve the effectiveness of domestic spending and development assistance by bringing vital knowledge into the service of policymaking and program design.

In 2004 the Center for Global Development, with support from the Bill & Melinda Gates Foundation and The William and Flora Hewlett Foundation, convened the Evaluation Gap Working Group. The group was asked to investigate why rigorous impact evaluations of social development programs, whether financed directly by developing country governments or supported by international aid, are relatively rare. The Working Group was charged with developing proposals to stimulate more and better impact evaluations. This report, the final report of the working group, contains specific recommendations for addressing this urgent problem.

Monitoring without Indicators - Most Significant Change

2007-01-31T01:11:00.000+02:00

On the Pelican list today, they sent through this handy reference to something that I think is infinitely useful for gathering proof and evidence when you don't have indicators and stacks of pre-developed evaluation mechanisms.

Check it out at:
http://www.mande.co.uk/docs/MSCGuide.htm. and http://www.mande.co.uk/MSC.htm

The guide (Prepared by Rick Davies and Jess Dart) explains the MSC technique as follows:

"The most significant change (MSC) technique is a form of participatory monitoring and evaluation. It is participatory because many project stakeholders are involved both in deciding the sorts of change to be recorded and in analysing the data. It is a form of monitoring because it occurs throughout the program cycle and provides information to help people manage the program. It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole.
Essentially, the process involves the collection of significant change (SC) stories emanating from the field level, and the systematic selection of the most significant of these stories by panels of designated stakeholders or staff. The designated staff and stakeholders are initially involved by ‘searching’ for project impact. Once changes have been captured, various people sit down together, read the stories aloud and have regular and often in-depth discussions about the value of these reported changes. When the technique is implemented successfully, whole teams of people begin to focus their attention on program impact."

Certainly this looks like a very promising technique!