Evaluation based on critical systems heuristics

Evaluation based on critical systems heuristics

1. When to use CSH

   • purposeful
     • ny area or situation of concern that might be associated with human purpose
     • evaluating
       • ought not to be confused with evaluating purposes
       • sphere of intended effects
         • possibly later impact back on the system of interest
         • unethical in the wider scheme of human activity
       • planning
         • Goal planning
         • Objective planning
         • Ideal planning
   • not used
     • goal-oriented (or purposive) evaluation
       • emphasis
   • typically
     • is used
       • evaluating plans
       • planning processes
       • in-situ formative evaluation
     • Churchma
       • stress the importance of locating the planning process
     • Ulrich

2. Why use CSH?

   • There are three good reasons
     • Boundary judgements encapture key dimensions of any purposeful system of interest
       • draws in a range
         • factors which other evaluation
       • Mainstream evaluation
         • ‘knowledge’
         • 'power'
         • measures of success’
     • Value judgements are made transparent
       • are asked in an 'is/ought' mode
       • situational change
       • ‘evaluation’
       • evaluator transparent
     • Securing improvement provides the driving principle for evaluation
       • Purposeful systems
         • boundary
         • value judgements

3. The Technique: doing a CSH evaluation

   • 12 questions
     • different 'systems'
   • Skill
     • from practical use
     • unfolding of CSH questions
   • varies between
     • practitioners with different interests
     • prior experiences of using CSH or similar techniques
     • between different contexts of use
   • comes from my own experience
   • 1
     • Identify the system of interest (SoI) which you are evaluating
   • 2
     • Reflect and make a note on your own role as evaluator in the system of interest being evaluated
   • 3
     • For the SoI identified
       • For the SoI identified
       • objective
       • ideal planning
     • SoI operates
       • operational/ administrative
       • management
       • policy design level of planning
   • 4
     • Focusing on the SoI
       • its underlying purpose
       • dentify associated stakeholders representing beneficiaries
       • decision makers
       • experts
       • witnesses
   • 5
     • Monological
       • Build up a picture of the SoI
       • making notes
         • reflections
   • 6
     • Decision environment
     • Dialogical
       • questionnaire
         • stakeholder
   • 7
     • Expert
     • The ‘final’ evaluation
       • your own role
       • reference to a normative
       • Evaluation using CSH
   • 8
     • Expertise
   • 9
     • Guarantee
   • 10
     • Witness
   • 11
     • • Emancipation
   • 12
     • Worldview

4. Case Study: natural resource management

   • Identifying the system of interest
   • Role of evaluator
   • Level of planning
     • Participatory Rural Appraisal (PRA) Pilot Project
     • Natural Resource Management Project (NRMP)
     • Botswana Range Inventory & Monitoring Project (BRIMP).
     • three progressively wider domains of planning
       • PRA Pilot
         • administration (‘goal planning’)
       • NRMP
         • management (‘objective planning’)
       • BRIMP
         • policy design (‘ideal planning’).
   • Stakeholder groups
     • generic social roles of beneficiaries
       • decisionmakers
       • experts
       • witnesses associated
     • Primary role in NRUA projects
       • Beneficiary
         • Government department
       • Decision-maker
         • Donor agency
       • Expert (professional)
         • Consultancy
       • Witness
         • NGO
           • Institutional type
   • Monological appraisal
     • Motivation
       • Beneficiary
         • purpose
           • measure of improvemetn
     • control
       • Decision-maker
         • Resources
           • decision environment
     • Expertise
       • Expert
         • Expertise
           • guarantee
     • Legitimation
       • Witness
         • Emancipation
           • worldview
     • system of selforganisation
       • appraisal amongst conscientious natural resource
   • Dialogical appraisal
     • stakeholder mapping
     • normative mapping
     • different stakeholder interviewees
       • the perceived stakeholder role
       • the particular level of planning
     • • Primary stakeholder role in NRUA projects
         • Beneficiary:
           • Government department
           • How to reconcile tradition of centralised roles for government
         • Decision-maker
           • Donor agency
           • How to transfer ‘ownership’
         • Expert (professional):
           • Consultancy
           • How to ensure impartial production of knowledge whilst changing validity criteria for appraisal output?
         • Witness
           • NGO
           • How to avoid conflict of interests given that NGOs are generally answerable primarily to government & donors?
   • Reporting
     • The three projects were evaluated over a relatively long period of time
     • Each report
       • what I perceived
         • main issues of the evaluation, couched in terms appreciated by the stakeholders
       • role and purpose with respect to the evaluation exercise
       • disclaimer regarding any pretence to having made a ‘scientific’ evaluation
       • Boxes
         • 1
           • Participatory Rural Appraisal (PRA) Pilot Project
             • Motivation
             • Control
             • Expertise
             • Legitimation
         • 2
           • Motivation
           • Control
           • Expertise
             • Natural Resource Management Project (NRMP)
           • Legitimation
         • 3
           • Botswana Range Inventory & Monitoring Project (BRIMP)
             • Motivation
             • Control
             • Expertise
             • Legitimation

5. Role of evaluator

   • ‘expert’
   • ‘witness’