Every Utilization Focused Evaluator knows: If an evaluation deliverable is submitted late, it can potentially totally negate the reason for doing the evaluation in the first place. If the intended user does not have the information at his/her disposal when the time for the intended use comes, you have a disaster.
Late delivery and a whole range of unpleasant consequences can be prevented, if you have the skill of holding people accountable for broken promises along the way. I find holding my team accountable is relatively easy, but holding a client accountable for broken commitments, is quite a different ballgame- A very unpleasant and daunting one!
Imagine the following totally hypothetical* example: You are working with a client who commits to giving feedback on deliverables by a certain date, and by the time you get to that date, there is no feedback. Or no consequential feedback. Then you try to get sign-off on deliverables, but a second and third round of comments follow, and it takes forever to move along. Hypothetically speaking, it could take you 8 months to get sign off on one deliverable!
It could be really difficult to work effectively with such a (hypotehtical) client, and you may start to doubt your own ability to execute. I cringe if I look at it from the hypothetical client's perspective - and I'm not sure what this must look like to a hypothetical innocent bystander.
Evaluation team leaders need to be able to hold their clients accountable for broken commitments, else the team may be heading for a deep and dark place where everyone just ends up hating working together.
I'm reading a book called "Crucial Confrontations" on exactly this topic. It is packed with so many interesting concepts. I can highly recommend it. See: http://ow.ly/4waEW (Also available on Kindle!)
The book has taught me to be a little bit more discerning about the actual problem that needs to be addressed (amongst other things). Who would've thought that what seems like a simple (hypothetical) problem can actually be so very complex?
1) If the problem is a single instance of not receiving feedback all you need to do is follow up with the client until he / she has met his / her commitment. Getting the feedback once, means the problem is resolved.
2) If the problem is a pattern of broken commitments, just dogging the client until he/ she provides feedback will probably only resolve the problem until the next round of feedback is due. Perhaps the process of feedback and signoff should be changed, to break this pattern?
3) But the problem may be a relationship problem. If the client is in a difficult position in his / her organization he / she may be unable to act quickly / definitively / at all. No amount of process change will solve this quandary. If a pattern of broken commitments have lead to tensions in the relationship between the client and the evaluator, interacting is likely to become more difficult as time goes by, and more commitments will be broken. The solution that is required, is a fix for the relationship!
The book has so many other good suggestions, which would be very handy in planning interactions with a hypothetical problem client. May you never have the need of applying these skills!
**********************************************************************
*My husband says that when people say "theoretically" they mean "not really". When I say hypothetically, I mean exactly that!
Showing posts with label Clients. Show all posts
Showing posts with label Clients. Show all posts
Friday, April 08, 2011
Thursday, November 26, 2009
A lonely brainstorm... Or many minds?
A grantmaking organization (our client) is interested in evaluating the level of their service delivery and relationship management – as perceived by the grantees that they disburse funds to. So here is the question – What are the evaluation standards that we should use?
Grantee perceptions?
The terms of reference indicates that the client expects that the evaluators will interact with the grantees to answer their questions. But if we ask grantees what they think of the grant maker’s processes, approach, involvement, communication etc. we might get senseless data because the wide range of grantees will have very different expectations about what qualifies as good service delivery / relationship management. It will probably be easy to collect data about their perceptions, but that won’t be very useful. And then there is also the issue of possible bias: Those grantees that experienced difficulty in submitting reports etc for monitoring purposes, might actually be slightly more negative than the rest of the grantees that would probably be eager to be complement the people that will dish out their next pay check.
The Grantmakers’ own standards?
It might make sense to determine whether the grantmaker has any implicit or explicit service delivery standards or contracted agreements that could be used as the standard to evaluate their performance against. But if the grantmaker has a standard that says: “All applications must be acknowledged in writing within 6 months from the date of receipt” that would be easy to check, but surely that service standard seems a little odd? Does it really take six months to respond to a submission?
Industry standards and benchmarks?
The alternative would be to look at service delivery standards and benchmarks as set by other industry players. There’s lots of literature about grantmaking internationally, but information about South African grantmakers are limited – There is the CSI handbook, but it doesn’t contain the level of detail that may be required to develop an extensive set of evaluation standards and benchmarks. And grant makers are notoriously secretive about their approach, systems and quality standards, so we will probably not be able to get detailed information from more than a handful of players in the field that we have established past relationships with.
Room for a participatory agreement on what exactly should be measured?
It is possible that a rigorous engagement of grantees and grant makers at the outset of the evaluation could provide the most satisfactory solution to the “which standards should we use” question. And that is probably just what we will do! Background research about all of the above will probably provide a good basis to start the workshop, but it will be interesting to see what the final consensus will dictate!
Grantee perceptions?
The terms of reference indicates that the client expects that the evaluators will interact with the grantees to answer their questions. But if we ask grantees what they think of the grant maker’s processes, approach, involvement, communication etc. we might get senseless data because the wide range of grantees will have very different expectations about what qualifies as good service delivery / relationship management. It will probably be easy to collect data about their perceptions, but that won’t be very useful. And then there is also the issue of possible bias: Those grantees that experienced difficulty in submitting reports etc for monitoring purposes, might actually be slightly more negative than the rest of the grantees that would probably be eager to be complement the people that will dish out their next pay check.
The Grantmakers’ own standards?
It might make sense to determine whether the grantmaker has any implicit or explicit service delivery standards or contracted agreements that could be used as the standard to evaluate their performance against. But if the grantmaker has a standard that says: “All applications must be acknowledged in writing within 6 months from the date of receipt” that would be easy to check, but surely that service standard seems a little odd? Does it really take six months to respond to a submission?
Industry standards and benchmarks?
The alternative would be to look at service delivery standards and benchmarks as set by other industry players. There’s lots of literature about grantmaking internationally, but information about South African grantmakers are limited – There is the CSI handbook, but it doesn’t contain the level of detail that may be required to develop an extensive set of evaluation standards and benchmarks. And grant makers are notoriously secretive about their approach, systems and quality standards, so we will probably not be able to get detailed information from more than a handful of players in the field that we have established past relationships with.
Room for a participatory agreement on what exactly should be measured?
It is possible that a rigorous engagement of grantees and grant makers at the outset of the evaluation could provide the most satisfactory solution to the “which standards should we use” question. And that is probably just what we will do! Background research about all of the above will probably provide a good basis to start the workshop, but it will be interesting to see what the final consensus will dictate!
Wednesday, July 11, 2007
Rant for today*
But hey, I am an evaluator, and evaluative thinking requires me to find out why I was not successful. So I write the “Thanks for the notification, we are disappointed, could you please tell us where our submission was weak… bla-di-blah” email. To which I don’t receive a response. I get my office manager to follow up and get a response of the kind: “We liked your presentation / Sorry we don’t have time to provide feedback / now please just leave us alone”.
What happened to the transparency and fairness thing? Am I unreasonable to think that a two line explanatory email is not much to ask after all of the trouble they put me through?
Ahem… The
http://www.evaluation.org.uk/Pub_library/PRO2907%20-%20UKE.%20A5%20Guideline.pdf
Maybe more people should read that?
*Because this is my blog I get to complain here every now and again. I promise that this will not turn into those rant-upon-rant blogs, but I really need to get the following off my chest.
Monday, September 04, 2006
When it really doesn’t make sense to use euphemisms
The following call for submissions was recently circulated in the M&E community.
“ORGANISATION ABC wants to engage in contracts with representative independent individual contactors in all provinces. We are therefore inviting representative individuals who are well qualified and experienced in Monitoring and Evaluation and related areas to submit their curriculum vitae’s and relevant information for consideration and possible invitation to attend a selection interview and deliver a simulated presentation.”
What is all of this talk about “representative individuals” about? I wonder why they just couldn’t come out and say “Individuals from Historically Disadvantaged Groups”. Strictly speaking, as a white female I am surely representative of some demographic in South Africa, ‘though I don’t think I am exactly what they mean under “representative individual!
And on an entirely non-M&E note, I read the following poem from Antjie Krog in "Verweerskrif" published by Umuzi in 2006 (Also available in English as "Body Bereft". Seems like we won’t get away from representing something ‘till the day death comes knocking.
namens myself
namens niemand hoef ek iets meer te benader nie
namens niemand hoef ek meer verantwoording
te doen of om vergifnis te vra nie.
niemand se gemarginaliseerde perspektief
hoef ek meer op tafel te plaas
of my in ander se vel te verbeel nie
die eerste voorhoedes van die dood
het opgedaag en die liggaam gly soos sand
deur die vingers. apatie neutraliseer die sintuie
oorlewing ontplooi soos ‘n woestaard en sny
jou af van ander sodat jy al meer vertroud
raak met die na-binne-gedraaidheid van die dood
Laai vir laai word jy leeggemaak
Tot net nog die leë binnekant jou raak
“ORGANISATION ABC wants to engage in contracts with representative independent individual contactors in all provinces. We are therefore inviting representative individuals who are well qualified and experienced in Monitoring and Evaluation and related areas to submit their curriculum vitae’s and relevant information for consideration and possible invitation to attend a selection interview and deliver a simulated presentation.”
What is all of this talk about “representative individuals” about? I wonder why they just couldn’t come out and say “Individuals from Historically Disadvantaged Groups”. Strictly speaking, as a white female I am surely representative of some demographic in South Africa, ‘though I don’t think I am exactly what they mean under “representative individual!
And on an entirely non-M&E note, I read the following poem from Antjie Krog in "Verweerskrif" published by Umuzi in 2006 (Also available in English as "Body Bereft". Seems like we won’t get away from representing something ‘till the day death comes knocking.
namens myself
namens niemand hoef ek iets meer te benader nie
namens niemand hoef ek meer verantwoording
te doen of om vergifnis te vra nie.
niemand se gemarginaliseerde perspektief
hoef ek meer op tafel te plaas
of my in ander se vel te verbeel nie
die eerste voorhoedes van die dood
het opgedaag en die liggaam gly soos sand
deur die vingers. apatie neutraliseer die sintuie
oorlewing ontplooi soos ‘n woestaard en sny
jou af van ander sodat jy al meer vertroud
raak met die na-binne-gedraaidheid van die dood
Laai vir laai word jy leeggemaak
Tot net nog die leë binnekant jou raak
Monday, June 12, 2006
Pet Peeves: Kakiebos & Cosmos
I have been doing M&E for about five years or so. In this time, I have come to develop a list of "Pet Peeves" which I will refer to as the "Kakiebos & Cosmos" list for the purposes of this blog. I am sure that I am not the only person that experience these. And I am sure many of these peeves are not unique to the M&E field. In fact, I would venture to say that these things are probably as common as kakiebos(1) or cosmos(2).
Here is my list:
* "Door stop" evaluations. In other words people commision evaluations that lead to reports that are ever only used as door stops, and nothing else.
* "Lottery" evaluations. These are the kind of evaluations where the client gives you (and 25 other service providers) no more than half a page background about the project and expects you to come up with a 25 page proposal that details exactly what needs to be done... Without any indication of what the budget should be... Its like playing the lottery where you have a one in twenty five chance to actually win the assignment.
*"SCiI Evaluations" These are the evaluations where "Scope Creep is Inevitable" and you end up writing the client's evaluation report, annual report, management presentation and also plan next year's evaluation.
*"PR Evaluations" You are engaged to do an evaluation. So you tell the story about the Good, the Bad and the Ugly Fairy Godmother's role in all of it. When you submit the first draft of your report, your client complains that it "Isn't what we envisioned". The euphemism for "What in the world are you thinking? We can't tell people we did not make any impact! Rewrite the report and change all of the findings so that we can impress the boss / shareholders / board / funder!!!"
* "Pandora's Box" evaluations. This is the kind of evaluation your clients let you do while they know there are a myriad of other unrelated issues that will make your job close to impossible. These evaluations tend to happen in the middle of organisational restructuring / just before the boss is suspended for embezzling funds / whilst a forensic audit is happening and everyone is in "hiding" / a year after the online database was started without any training for the users
* "Tell me the pretty story" evaluations. These are the kinds of evaluations where you are expected to produce a pretty report full of pictures with smiling faces and heart-rendering stories, without a single statistic that helps the reader to grasp what the costs or benefits of the project / programme was.
Like kakiebos, these types of evaluations are abundant. And not very useful. Sometimes, these evaluations even resemble cosmos. Still thoroughly useless but at least very pretty to look at for short periods of time. In fact, like kakiebos and cosmos, these evaluations just tap resources that should have been available for doing useful things, like growing sunflowers.
Oh, I don't know. Maybe people that commission / do kakiebos & cosmos evaluations should be sentenced to 100 hours of community service? I think gardening might be a good punishment for them. What do you say?
(1) Kakiebos is the Afrikaans vernacular for the plant Tagetes minuta which is commonly found on disturbed earth e.g. next to roads and is commonly regarded as a weed.
(2) Cosmos is the vernacular for the plant Bidens spp. which is commonly found on disturbed earth e.g. next to roads and is commonly regarded as a weed. In March / April it is, however, quite a spectacular sight to see as these plants carry white, pink and purple flowers.
Here is my list:
* "Door stop" evaluations. In other words people commision evaluations that lead to reports that are ever only used as door stops, and nothing else.
* "Lottery" evaluations. These are the kind of evaluations where the client gives you (and 25 other service providers) no more than half a page background about the project and expects you to come up with a 25 page proposal that details exactly what needs to be done... Without any indication of what the budget should be... Its like playing the lottery where you have a one in twenty five chance to actually win the assignment.
*"SCiI Evaluations" These are the evaluations where "Scope Creep is Inevitable" and you end up writing the client's evaluation report, annual report, management presentation and also plan next year's evaluation.
*"PR Evaluations" You are engaged to do an evaluation. So you tell the story about the Good, the Bad and the Ugly Fairy Godmother's role in all of it. When you submit the first draft of your report, your client complains that it "Isn't what we envisioned". The euphemism for "What in the world are you thinking? We can't tell people we did not make any impact! Rewrite the report and change all of the findings so that we can impress the boss / shareholders / board / funder!!!"
* "Pandora's Box" evaluations. This is the kind of evaluation your clients let you do while they know there are a myriad of other unrelated issues that will make your job close to impossible. These evaluations tend to happen in the middle of organisational restructuring / just before the boss is suspended for embezzling funds / whilst a forensic audit is happening and everyone is in "hiding" / a year after the online database was started without any training for the users
* "Tell me the pretty story" evaluations. These are the kinds of evaluations where you are expected to produce a pretty report full of pictures with smiling faces and heart-rendering stories, without a single statistic that helps the reader to grasp what the costs or benefits of the project / programme was.
Like kakiebos, these types of evaluations are abundant. And not very useful. Sometimes, these evaluations even resemble cosmos. Still thoroughly useless but at least very pretty to look at for short periods of time. In fact, like kakiebos and cosmos, these evaluations just tap resources that should have been available for doing useful things, like growing sunflowers.
Oh, I don't know. Maybe people that commission / do kakiebos & cosmos evaluations should be sentenced to 100 hours of community service? I think gardening might be a good punishment for them. What do you say?
(1) Kakiebos is the Afrikaans vernacular for the plant Tagetes minuta which is commonly found on disturbed earth e.g. next to roads and is commonly regarded as a weed.
(2) Cosmos is the vernacular for the plant Bidens spp. which is commonly found on disturbed earth e.g. next to roads and is commonly regarded as a weed. In March / April it is, however, quite a spectacular sight to see as these plants carry white, pink and purple flowers.
Subscribe to:
Posts (Atom)