Every Utilization Focused Evaluator knows: If an evaluation deliverable is submitted late, it can potentially totally negate the reason for doing the evaluation in the first place. If the intended user does not have the information at his/her disposal when the time for the intended use comes, you have a disaster.
Late delivery and a whole range of unpleasant consequences can be prevented, if you have the skill of holding people accountable for broken promises along the way. I find holding my team accountable is relatively easy, but holding a client accountable for broken commitments, is quite a different ballgame- A very unpleasant and daunting one!
Imagine the following totally hypothetical* example: You are working with a client who commits to giving feedback on deliverables by a certain date, and by the time you get to that date, there is no feedback. Or no consequential feedback. Then you try to get sign-off on deliverables, but a second and third round of comments follow, and it takes forever to move along. Hypothetically speaking, it could take you 8 months to get sign off on one deliverable!
It could be really difficult to work effectively with such a (hypotehtical) client, and you may start to doubt your own ability to execute. I cringe if I look at it from the hypothetical client's perspective - and I'm not sure what this must look like to a hypothetical innocent bystander.
Evaluation team leaders need to be able to hold their clients accountable for broken commitments, else the team may be heading for a deep and dark place where everyone just ends up hating working together.
I'm reading a book called "Crucial Confrontations" on exactly this topic. It is packed with so many interesting concepts. I can highly recommend it. See: http://ow.ly/4waEW (Also available on Kindle!)
The book has taught me to be a little bit more discerning about the actual problem that needs to be addressed (amongst other things). Who would've thought that what seems like a simple (hypothetical) problem can actually be so very complex?
1) If the problem is a single instance of not receiving feedback all you need to do is follow up with the client until he / she has met his / her commitment. Getting the feedback once, means the problem is resolved.
2) If the problem is a pattern of broken commitments, just dogging the client until he/ she provides feedback will probably only resolve the problem until the next round of feedback is due. Perhaps the process of feedback and signoff should be changed, to break this pattern?
3) But the problem may be a relationship problem. If the client is in a difficult position in his / her organization he / she may be unable to act quickly / definitively / at all. No amount of process change will solve this quandary. If a pattern of broken commitments have lead to tensions in the relationship between the client and the evaluator, interacting is likely to become more difficult as time goes by, and more commitments will be broken. The solution that is required, is a fix for the relationship!
The book has so many other good suggestions, which would be very handy in planning interactions with a hypothetical problem client. May you never have the need of applying these skills!
**********************************************************************
*My husband says that when people say "theoretically" they mean "not really". When I say hypothetically, I mean exactly that!
Friday, April 08, 2011
Thursday, March 31, 2011
AEA 365
The American Evaluation Association has this AEA365 service, which is really cool. I love what they have done with it. I only today managed to look at it in detail and its now my new favourite resource to recommend to new evaluators.
See www.eval.org aea365 Tip-a-Day Email Alerts: We're highlighting hot tips, cool tricks, rad resources, and lessons learned for and from evaluators Evaluation Tip-a-Day Emails
In 2007 I actually wrote a concept note for a service like this when I was at an AfrEA workshop in Niamey, Niger. Subsequently I mentioned it at the SAMEA conference as a potential service and everyboy thought it was a good idea. But I just did not have the resources at my disposal to make it happen. During that time I was invovled in running the operations of SAMEA and later AFREA so all my volunteer time and inspiration was swallowed up there. And although I thought that graduate students might be best placed to start compiling the content, I just never could find someone that would be willing to drive such an idea. So I am soooooooo happy that AEA had the same smart idea and were actually able to pull together the resources to make it a reality.
See www.eval.org aea365 Tip-a-Day Email Alerts: We're highlighting hot tips, cool tricks, rad resources, and lessons learned for and from evaluators Evaluation Tip-a-Day Emails
In 2007 I actually wrote a concept note for a service like this when I was at an AfrEA workshop in Niamey, Niger. Subsequently I mentioned it at the SAMEA conference as a potential service and everyboy thought it was a good idea. But I just did not have the resources at my disposal to make it happen. During that time I was invovled in running the operations of SAMEA and later AFREA so all my volunteer time and inspiration was swallowed up there. And although I thought that graduate students might be best placed to start compiling the content, I just never could find someone that would be willing to drive such an idea. So I am soooooooo happy that AEA had the same smart idea and were actually able to pull together the resources to make it a reality.
Categorizing Educational Qualifications
I had to look at a survey for a colleague. She was interested in getting information about the education and qualifications of survey respondents and just put in an open-ended question.
Bad idea! Why? Have you ever tried to afterwards classify people's qualifications into coding categories? I'm sure it can be done but its much better to give the people your classification categories, and have them select the appropriate options.
Before we get to the classification question, decide whether you only want to know what a person's highest qualification is, or whether you want to know which combination of qualifications a person holds. Usually we ask for highest only, but this depends on your research interest.
In South Africa we have SAQA (the South African Qualifications Authority) that provides the qualification framework and they are the best people to consult if you want a detailed qualifications framework applicable in South Africa. We seldom want this, because most people don't know what the difference between and NQF level 3 and 4 qualification is. A cool trick is to see what Stats SA uses. This is from an old General Household Survey questionnaire:
Note to users
This question is applicable to all household members. The enumerators are instructed that it is only those qualifications already obtained which must be entered. That means the current level, whereby a person is still busy with is not applicable. It is very important to complete each record even if the person has not attended school. Moreover the enumerators are instructed that diploma and certificates must be of at least six months duration.
Universe
All members of the household in the selected dwelling.
Final code list
0 = NO SCHOOLING
1 = GRADE R/0
2 = SUB A/GRADE 1
3 = SUB B/GRADE 2
4 = GRADE 3/STANDARD 1
5 = GRADE 4/STANDARD 2
6 = GRADE 5/STANDARD 3
7 = GRADE 6/STANDARD 4
8 = GRADE 7/STANDARD 5
9= GRADE 8/STANDARD 6/FORM 1
10 = GRADE 9/STANDARD 7/FORM 2
11 = GRADE 10/STANDARD 8/FORM 3
12 = GRADE 11/STANDARD 9/FORM 4
13 = GRADE 12/STANDARD 10/FORM 5/MATRIC
14 = NTC l
15 = NTC II
16 = NTC III
17 = DIPLOMA/CERTIFICATE WITH LESS THAN GRADE 12/STD 10
18 = DIPLOMA/CERTIFICATE WITH GRADE 12/STD 10
19 = DEGREE
20 = POSTGRADUATE DEGREE OR DIPLOMA
21 = OTHER (specify in column)
22 = DON'T KNOW
99 =UNSPECIFIED
This would unfortunately not work for my colleague because she is doing a survey with respondents who trained all over the world.
The international Equivalent of the SAQA standards is ISCED - The International Standard Classification of Education. It is an UNESCO standard and the last revision was in 1997.
ISCED provides an integrated and consistent statistical framework for the collection and reporting of internationally comparable education statistics. It contains two components:
a statistical framework for the comprehensive statistical description of national education and learning systems along a set of variables that are of key interest to policy makers in international educational comparisons; and
a methodology that translates national educational programmes into an internationally comparable set of categories for (i) the levels of education; and (ii) the fields of education.
Find ISCED here:
http://www.unesco.org/education/information/nfsunesco/doc/isced_1997.htm
Bad idea! Why? Have you ever tried to afterwards classify people's qualifications into coding categories? I'm sure it can be done but its much better to give the people your classification categories, and have them select the appropriate options.
Before we get to the classification question, decide whether you only want to know what a person's highest qualification is, or whether you want to know which combination of qualifications a person holds. Usually we ask for highest only, but this depends on your research interest.
In South Africa we have SAQA (the South African Qualifications Authority) that provides the qualification framework and they are the best people to consult if you want a detailed qualifications framework applicable in South Africa. We seldom want this, because most people don't know what the difference between and NQF level 3 and 4 qualification is. A cool trick is to see what Stats SA uses. This is from an old General Household Survey questionnaire:
Note to users
This question is applicable to all household members. The enumerators are instructed that it is only those qualifications already obtained which must be entered. That means the current level, whereby a person is still busy with is not applicable. It is very important to complete each record even if the person has not attended school. Moreover the enumerators are instructed that diploma and certificates must be of at least six months duration.
Universe
All members of the household in the selected dwelling.
Final code list
0 = NO SCHOOLING
1 = GRADE R/0
2 = SUB A/GRADE 1
3 = SUB B/GRADE 2
4 = GRADE 3/STANDARD 1
5 = GRADE 4/STANDARD 2
6 = GRADE 5/STANDARD 3
7 = GRADE 6/STANDARD 4
8 = GRADE 7/STANDARD 5
9= GRADE 8/STANDARD 6/FORM 1
10 = GRADE 9/STANDARD 7/FORM 2
11 = GRADE 10/STANDARD 8/FORM 3
12 = GRADE 11/STANDARD 9/FORM 4
13 = GRADE 12/STANDARD 10/FORM 5/MATRIC
14 = NTC l
15 = NTC II
16 = NTC III
17 = DIPLOMA/CERTIFICATE WITH LESS THAN GRADE 12/STD 10
18 = DIPLOMA/CERTIFICATE WITH GRADE 12/STD 10
19 = DEGREE
20 = POSTGRADUATE DEGREE OR DIPLOMA
21 = OTHER (specify in column)
22 = DON'T KNOW
99 =UNSPECIFIED
This would unfortunately not work for my colleague because she is doing a survey with respondents who trained all over the world.
The international Equivalent of the SAQA standards is ISCED - The International Standard Classification of Education. It is an UNESCO standard and the last revision was in 1997.
ISCED provides an integrated and consistent statistical framework for the collection and reporting of internationally comparable education statistics. It contains two components:
a statistical framework for the comprehensive statistical description of national education and learning systems along a set of variables that are of key interest to policy makers in international educational comparisons; and
a methodology that translates national educational programmes into an internationally comparable set of categories for (i) the levels of education; and (ii) the fields of education.
Find ISCED here:
http://www.unesco.org/education/information/nfsunesco/doc/isced_1997.htm
Friday, March 25, 2011
Communicating Data
This week, I sat in a two day information quality conference. One of the key points was that data is not necessarily being used to improve service delivery, because (very simplistically)
* data quality is problematic, and
* we don't have people who can tell the story of the data so that others can relate to it.
I've already decided that my new thing for this year will be to think about communicating data. So Im seraching for good examples of visualization methods.
A friend sent this interesting example along:
http://www.guardian.co.uk/world/interactive/2011/mar/22/middle-east-protest-interactive-timeline
And another friend alerted me to this website:
http://www.informationisbeautiful.net/
* data quality is problematic, and
* we don't have people who can tell the story of the data so that others can relate to it.
I've already decided that my new thing for this year will be to think about communicating data. So Im seraching for good examples of visualization methods.
A friend sent this interesting example along:
http://www.guardian.co.uk/world/interactive/2011/mar/22/middle-east-protest-interactive-timeline
And another friend alerted me to this website:
http://www.informationisbeautiful.net/
"Development "
Aid on the edge posted a little satirical cartoon that made me reflect on the value of "development" as we do it these days.
See: http://aidontheedge.info/2011/03/23/there-you-go/
But that is only one side of the coin:
A friend once told me that he worked in development in Africa, and admitted to forcing a Western idea of education into a local village many years ago. The contemporary wisdom is that one should rather embrace indigenous knowledge systems and community structures. When he went back many years later, he wanted to "repent" of his youthful folly.
When he got there, there wasn't much left of the village. Sadly climate change has wreaked havoc and the village's agricultural lifestyle could not longer be sustained. If it wasn't for the "out of place" education initiative that my friend started many years earlier, there would have been no livelihood for the villagers. At least the kids who got educated, got out. They were now earning a living in another way, sending back some money.
I'm not suggesting that forcing education was the right thing to do, but we can also not blame the education and local development for climate change that is happening on a global scale and would have had an effect irrespective of whether development came to that corner of Africa or not.
See: http://aidontheedge.info/2011/03/23/there-you-go/
But that is only one side of the coin:
A friend once told me that he worked in development in Africa, and admitted to forcing a Western idea of education into a local village many years ago. The contemporary wisdom is that one should rather embrace indigenous knowledge systems and community structures. When he went back many years later, he wanted to "repent" of his youthful folly.
When he got there, there wasn't much left of the village. Sadly climate change has wreaked havoc and the village's agricultural lifestyle could not longer be sustained. If it wasn't for the "out of place" education initiative that my friend started many years earlier, there would have been no livelihood for the villagers. At least the kids who got educated, got out. They were now earning a living in another way, sending back some money.
I'm not suggesting that forcing education was the right thing to do, but we can also not blame the education and local development for climate change that is happening on a global scale and would have had an effect irrespective of whether development came to that corner of Africa or not.
Subscribe to:
Posts (Atom)