Evaluation

Evaluation

1. Why

   • Main Purpose
     • 1. To determine the overall quality or value of something. 2. To find areas for improvemenet 3. Both of the above
       • Changes the methodology
     • Independently mode
       • Independent evaluation. Nobody on the team should have a significant vested interes in whether the results are good or bad
     • Participatory mode
       • Organizational learning capacity Learn from success and failures
       • Degree of stakeholder participation
   • Big picture questions
     • Absolute merit or worth. How effective/valuable/meritorius is/was this?
     • Relative merit or worth. How does it compare with the other options or candidates?
   • Questions about absolute or relative quality value
     • Absolue quaility or grade
       • i. Absolute merit or worth (e.g., How effective/valuable/ meritorious is/was this? Is/Was it worth the resources [e.g., time, money] put into it?) ii.
       • • Is this intervention, product, or individual good enough to implement, buy, or hire (i.e., up to minimum requirements)? • How should this level of performance be characterized (on a particular dimension of merit or overall)? Is it excellent? Good? Satisfactory? Poor?
     • Relative quality
       • Relative merit or worth how does it compare with the other options or candidate?. It is always evaluated in comparison with one or more other evaluands
       • Ranking. Even it does not result in a strict numercial rank for each evaluand
         • • Which of these three pilot interventions is the most cost-effective and should be implemented throughout our organization?

2. Key concepts (Based on Scriven checklist)

   • Evaluand
     • Whatever is being evaluated
   • Evaluation
     • The process of determining (or the expression of a conclusion about the goodness and/or badness, wrongness and/or rightness of something.More specifically, about the merit, worth, or significance
   • Dimensions of merit
     • The characteristics of the evaluand (X) that definitionally bear on its merit, worth or significance
   • Indicators of merit
     • Refers to factors that are empirically but not definitionally linked to the evaluand’s merit, worth or significance
   • Professional Evaluation
     • Simply evaluation requiring specialized tools or skills that are not in the everyday repertoire; it is usually systematic (and inferential)
   • Merit
     • Is the intrisic value of something, the term is used interchangeably with quality
   • Worth
     • Worth is the value of something to an individual an organization, an institution, or a collective. The term is used interchangeably with value

3. Types

   • Summative Evaluation (Scriven, 1981)
     • Support decisions about the program to determine the overall quality of value of an evaluand
     • Evaluation that are done primarily for reporting and decision-making purposes other than improvement of the evaluand itself
       • Useful for
         • Accountability to funders or managers. Useful even to demonstrate return on investment ROI
     • Grading question: Is this national healt intervention worth what it cost ( in terms of time, money and other resources)?. Ranking questions: Which of these three executive development interventions we tried was the most cost-effective?
   • Formative evaluation
     • to identify needed improvements to the evaluand
     • Grading questions: How well does the content of this trainging program ( coverage, breadth and depth) match the real needs of our minority trainees?. Ranking questions. How do the initial improvements in anufacturing efficiency compare with those achieved elsewhere iin the industry?
   • Ascriptive evaluation
     • To enlarge or refine our body of evaluative knowledge
   • Compressive evaluation
     • rating personnel for a job or task

4. Parts of the evaluation

   • Foundations
     • Background and context
     • Descriptions and Definitions
     • Resources
     • Values
       • Explain on what basis will you determine whether the evaluand is of high quality, valuable, and so forth. Where will you get the criteria and how will you determine "how good is good"?
     • Stakeholders
       • Upstream stakeholders
         • People who worked on the design, implementation, and or managment
       • DownStream
         • Immediate recipients, other downstream impactees
           • Consumers
             • Who migth be affected by the evaluand The person or entity who buys or usesa product or service. Enrolls in or is the recipient of a program. Is directly affected by a program policy, etc. who potentially could have or should have done so
             • Downstream stakeholders
           • Immediate recipients are the people who actually bought a product, signed up for a program, or received services directly from the evaluand, whereas downstream impactees are those who were not direct recipients but who were affected nevertheless. Downstream impactees need not be individuals; instead, they could be the unit or organization where direct recipients work, the local community, or society in general.
   • Preliminaries
     • Executive summary
     • Preface
     • Methodology
   • SubEvaluations
     • Process Evaluation
       • Evaluation of the content ( design) and implementation ( delivery) of an evaluand. How good, valuable, or efficient is?
     • OutcomeEvaluation
       • How good or valuable are the impacs on immediate recipients oand other impactees?
     • Compartive cost-effectiveness
     • Exportability
       • What elements of the evaluand might make it potentially valuable or significant contribution or advance in another setting?
     • Overall significance
   • Conclusions
     • Recommendations and explantations (OPC)
     • Responsabilities (OPC)
     • Reporting and follow-up
     • Meta-evaluation

5. Overarching framework. Guiding framework for the evaluation team members

   • Make sure that all important ingredients to allow valid evaluative conclusions are included
   • Checklist of necessary ingredients and framework to help guide evaluation planning and reporting

6. Basic ideas

   • Based on Jane Davidson Book
   • Causation
   • Que debería crear
     • Un modelo flexible que se adapte a un entorno social con características particulares como e que propone streven
   • Things that we might evaluate sistematically
     • Projects, programs, or organizations. Personal or performance, policies or startegies, products or sevices, processes or systems, proposals contract bids, job applications

7. Evaluative criteria

   • Needs assessment
   • If we can understand what the true needs of consumers or impactees are, this gives us a solid basis for finding out how well a program is doing by seeing how well it is helping to meet those needs. In other words, needs that criteria