The American Evaluation Association has this AEA365 service, which is really cool. I love what they have done with it. I only today managed to look at it in detail and its now my new favourite resource to recommend to new evaluators.
See www.eval.org aea365 Tip-a-Day Email Alerts: We're highlighting hot tips, cool tricks, rad resources, and lessons learned for and from evaluators Evaluation Tip-a-Day Emails
In 2007 I actually wrote a concept note for a service like this when I was at an AfrEA workshop in Niamey, Niger. Subsequently I mentioned it at the SAMEA conference as a potential service and everyboy thought it was a good idea. But I just did not have the resources at my disposal to make it happen. During that time I was invovled in running the operations of SAMEA and later AFREA so all my volunteer time and inspiration was swallowed up there. And although I thought that graduate students might be best placed to start compiling the content, I just never could find someone that would be willing to drive such an idea. So I am soooooooo happy that AEA had the same smart idea and were actually able to pull together the resources to make it a reality.
Thursday, March 31, 2011
Categorizing Educational Qualifications
I had to look at a survey for a colleague. She was interested in getting information about the education and qualifications of survey respondents and just put in an open-ended question.
Bad idea! Why? Have you ever tried to afterwards classify people's qualifications into coding categories? I'm sure it can be done but its much better to give the people your classification categories, and have them select the appropriate options.
Before we get to the classification question, decide whether you only want to know what a person's highest qualification is, or whether you want to know which combination of qualifications a person holds. Usually we ask for highest only, but this depends on your research interest.
In South Africa we have SAQA (the South African Qualifications Authority) that provides the qualification framework and they are the best people to consult if you want a detailed qualifications framework applicable in South Africa. We seldom want this, because most people don't know what the difference between and NQF level 3 and 4 qualification is. A cool trick is to see what Stats SA uses. This is from an old General Household Survey questionnaire:
Note to users
This question is applicable to all household members. The enumerators are instructed that it is only those qualifications already obtained which must be entered. That means the current level, whereby a person is still busy with is not applicable. It is very important to complete each record even if the person has not attended school. Moreover the enumerators are instructed that diploma and certificates must be of at least six months duration.
Universe
All members of the household in the selected dwelling.
Final code list
0 = NO SCHOOLING
1 = GRADE R/0
2 = SUB A/GRADE 1
3 = SUB B/GRADE 2
4 = GRADE 3/STANDARD 1
5 = GRADE 4/STANDARD 2
6 = GRADE 5/STANDARD 3
7 = GRADE 6/STANDARD 4
8 = GRADE 7/STANDARD 5
9= GRADE 8/STANDARD 6/FORM 1
10 = GRADE 9/STANDARD 7/FORM 2
11 = GRADE 10/STANDARD 8/FORM 3
12 = GRADE 11/STANDARD 9/FORM 4
13 = GRADE 12/STANDARD 10/FORM 5/MATRIC
14 = NTC l
15 = NTC II
16 = NTC III
17 = DIPLOMA/CERTIFICATE WITH LESS THAN GRADE 12/STD 10
18 = DIPLOMA/CERTIFICATE WITH GRADE 12/STD 10
19 = DEGREE
20 = POSTGRADUATE DEGREE OR DIPLOMA
21 = OTHER (specify in column)
22 = DON'T KNOW
99 =UNSPECIFIED
This would unfortunately not work for my colleague because she is doing a survey with respondents who trained all over the world.
The international Equivalent of the SAQA standards is ISCED - The International Standard Classification of Education. It is an UNESCO standard and the last revision was in 1997.
ISCED provides an integrated and consistent statistical framework for the collection and reporting of internationally comparable education statistics. It contains two components:
a statistical framework for the comprehensive statistical description of national education and learning systems along a set of variables that are of key interest to policy makers in international educational comparisons; and
a methodology that translates national educational programmes into an internationally comparable set of categories for (i) the levels of education; and (ii) the fields of education.
Find ISCED here:
http://www.unesco.org/education/information/nfsunesco/doc/isced_1997.htm
Bad idea! Why? Have you ever tried to afterwards classify people's qualifications into coding categories? I'm sure it can be done but its much better to give the people your classification categories, and have them select the appropriate options.
Before we get to the classification question, decide whether you only want to know what a person's highest qualification is, or whether you want to know which combination of qualifications a person holds. Usually we ask for highest only, but this depends on your research interest.
In South Africa we have SAQA (the South African Qualifications Authority) that provides the qualification framework and they are the best people to consult if you want a detailed qualifications framework applicable in South Africa. We seldom want this, because most people don't know what the difference between and NQF level 3 and 4 qualification is. A cool trick is to see what Stats SA uses. This is from an old General Household Survey questionnaire:
Note to users
This question is applicable to all household members. The enumerators are instructed that it is only those qualifications already obtained which must be entered. That means the current level, whereby a person is still busy with is not applicable. It is very important to complete each record even if the person has not attended school. Moreover the enumerators are instructed that diploma and certificates must be of at least six months duration.
Universe
All members of the household in the selected dwelling.
Final code list
0 = NO SCHOOLING
1 = GRADE R/0
2 = SUB A/GRADE 1
3 = SUB B/GRADE 2
4 = GRADE 3/STANDARD 1
5 = GRADE 4/STANDARD 2
6 = GRADE 5/STANDARD 3
7 = GRADE 6/STANDARD 4
8 = GRADE 7/STANDARD 5
9= GRADE 8/STANDARD 6/FORM 1
10 = GRADE 9/STANDARD 7/FORM 2
11 = GRADE 10/STANDARD 8/FORM 3
12 = GRADE 11/STANDARD 9/FORM 4
13 = GRADE 12/STANDARD 10/FORM 5/MATRIC
14 = NTC l
15 = NTC II
16 = NTC III
17 = DIPLOMA/CERTIFICATE WITH LESS THAN GRADE 12/STD 10
18 = DIPLOMA/CERTIFICATE WITH GRADE 12/STD 10
19 = DEGREE
20 = POSTGRADUATE DEGREE OR DIPLOMA
21 = OTHER (specify in column)
22 = DON'T KNOW
99 =UNSPECIFIED
This would unfortunately not work for my colleague because she is doing a survey with respondents who trained all over the world.
The international Equivalent of the SAQA standards is ISCED - The International Standard Classification of Education. It is an UNESCO standard and the last revision was in 1997.
ISCED provides an integrated and consistent statistical framework for the collection and reporting of internationally comparable education statistics. It contains two components:
a statistical framework for the comprehensive statistical description of national education and learning systems along a set of variables that are of key interest to policy makers in international educational comparisons; and
a methodology that translates national educational programmes into an internationally comparable set of categories for (i) the levels of education; and (ii) the fields of education.
Find ISCED here:
http://www.unesco.org/education/information/nfsunesco/doc/isced_1997.htm
Friday, March 25, 2011
Communicating Data
This week, I sat in a two day information quality conference. One of the key points was that data is not necessarily being used to improve service delivery, because (very simplistically)
* data quality is problematic, and
* we don't have people who can tell the story of the data so that others can relate to it.
I've already decided that my new thing for this year will be to think about communicating data. So Im seraching for good examples of visualization methods.
A friend sent this interesting example along:
http://www.guardian.co.uk/world/interactive/2011/mar/22/middle-east-protest-interactive-timeline
And another friend alerted me to this website:
http://www.informationisbeautiful.net/
* data quality is problematic, and
* we don't have people who can tell the story of the data so that others can relate to it.
I've already decided that my new thing for this year will be to think about communicating data. So Im seraching for good examples of visualization methods.
A friend sent this interesting example along:
http://www.guardian.co.uk/world/interactive/2011/mar/22/middle-east-protest-interactive-timeline
And another friend alerted me to this website:
http://www.informationisbeautiful.net/
"Development "
Aid on the edge posted a little satirical cartoon that made me reflect on the value of "development" as we do it these days.
See: http://aidontheedge.info/2011/03/23/there-you-go/
But that is only one side of the coin:
A friend once told me that he worked in development in Africa, and admitted to forcing a Western idea of education into a local village many years ago. The contemporary wisdom is that one should rather embrace indigenous knowledge systems and community structures. When he went back many years later, he wanted to "repent" of his youthful folly.
When he got there, there wasn't much left of the village. Sadly climate change has wreaked havoc and the village's agricultural lifestyle could not longer be sustained. If it wasn't for the "out of place" education initiative that my friend started many years earlier, there would have been no livelihood for the villagers. At least the kids who got educated, got out. They were now earning a living in another way, sending back some money.
I'm not suggesting that forcing education was the right thing to do, but we can also not blame the education and local development for climate change that is happening on a global scale and would have had an effect irrespective of whether development came to that corner of Africa or not.
See: http://aidontheedge.info/2011/03/23/there-you-go/
But that is only one side of the coin:
A friend once told me that he worked in development in Africa, and admitted to forcing a Western idea of education into a local village many years ago. The contemporary wisdom is that one should rather embrace indigenous knowledge systems and community structures. When he went back many years later, he wanted to "repent" of his youthful folly.
When he got there, there wasn't much left of the village. Sadly climate change has wreaked havoc and the village's agricultural lifestyle could not longer be sustained. If it wasn't for the "out of place" education initiative that my friend started many years earlier, there would have been no livelihood for the villagers. At least the kids who got educated, got out. They were now earning a living in another way, sending back some money.
I'm not suggesting that forcing education was the right thing to do, but we can also not blame the education and local development for climate change that is happening on a global scale and would have had an effect irrespective of whether development came to that corner of Africa or not.
Monday, March 07, 2011
Specificity and Sensitivity in tests
You are required to identify kids in need of remediation using a scholastic ability test.
If your test is highly specific, a low score will be able to identify everyone that requires remediation. - A lack of specificity indicates that some kids who require remediation are not identified.
If your test is highly sensitive, then a high score will clearly exclude anyone that does not need remediation.
SPIN and SNOUT are commonly used mnemonics which helps to remind us ofthe disticntion: A highly SPecific test, when Positive, rules IN disease (SP-P-IN), and a highly 'SeNsitive' test, when Negative rules OUT disease (SN-N-OUT)
If your test is highly specific, a low score will be able to identify everyone that requires remediation. - A lack of specificity indicates that some kids who require remediation are not identified.
If your test is highly sensitive, then a high score will clearly exclude anyone that does not need remediation.
SPIN and SNOUT are commonly used mnemonics which helps to remind us ofthe disticntion: A highly SPecific test, when Positive, rules IN disease (SP-P-IN), and a highly 'SeNsitive' test, when Negative rules OUT disease (SN-N-OUT)
Friday, March 04, 2011
Questioning the archaic...
We had a debate in our office the other day as to the proper use of spacing after a fullstop. Some of my colleagues insisted that double spacing after a fullstop was the proper way to type whilst others insisted on single spacing. A couple of opinion polls later, I started checking some style manuals and the opinion of the typographers. The jury is not out on this anymore - If you type on a modern computer, single space is what you should use.
Apparently people who type two spaces, were taught by people who were taught by people who learned to type on typewriters that only allowed monospacing - i.e. an "l" and an "m", despite being different in size, was given the same amount of space, because typewriters couldn't work differently. This resulted in lots of white space in the middle of words, hence the need for double spacing between senences. With the introduction of computers, almost all texts are now created in proportional fonts - so the narrower characters take less space, and you don't need a double space after a full stop.
But this got me thinking - How much of what we do as evaluators and researchers do we do just because we were taught by people who had to make use of old archaic technology to get the job done? I mean, think about it - Why do we still insist that the primary output from an evaluation should be a report? Or... hold on to your seat... a PowerPoint presentation?
If use of evaluations (or information) depends on the degree to which the findings are communicated concisely, then we should be building our communications capability and get creative. If you look at the capabilities that simple Mac Software like Keynote offers (and I'm no expert) then really! There is so much more that we should be doing
I share with you three examples of what I would like to see more of:
Hans Roslin inspires with visualization of statistics (This guy rocks!):
http://ow.ly/47WDV
This "Story of Stuff" clip combines presentation, story telling and animation in really interesting ways
http://ow.ly/47WIh
And Here is how a normal presentation with some voice can help to get the message across, a lot better than just a PowerPoint presentation
http://ow.ly/47WOA
Apparently people who type two spaces, were taught by people who were taught by people who learned to type on typewriters that only allowed monospacing - i.e. an "l" and an "m", despite being different in size, was given the same amount of space, because typewriters couldn't work differently. This resulted in lots of white space in the middle of words, hence the need for double spacing between senences. With the introduction of computers, almost all texts are now created in proportional fonts - so the narrower characters take less space, and you don't need a double space after a full stop.
But this got me thinking - How much of what we do as evaluators and researchers do we do just because we were taught by people who had to make use of old archaic technology to get the job done? I mean, think about it - Why do we still insist that the primary output from an evaluation should be a report? Or... hold on to your seat... a PowerPoint presentation?
If use of evaluations (or information) depends on the degree to which the findings are communicated concisely, then we should be building our communications capability and get creative. If you look at the capabilities that simple Mac Software like Keynote offers (and I'm no expert) then really! There is so much more that we should be doing
I share with you three examples of what I would like to see more of:
Hans Roslin inspires with visualization of statistics (This guy rocks!):
http://ow.ly/47WDV
This "Story of Stuff" clip combines presentation, story telling and animation in really interesting ways
http://ow.ly/47WIh
And Here is how a normal presentation with some voice can help to get the message across, a lot better than just a PowerPoint presentation
http://ow.ly/47WOA
Subscribe to:
Posts (Atom)