Evaluation Research

Evaluation Research

1. Features

   • •applied social research – rigorous and systematic research approaches •diversity of research skills •social context of evaluation •policy making relevance and significance •dissemination and utilisation of evaluation findings

2. Qs

   • •What is the nature and scope of the problem requiring new, expanded or modified interventions or services ? •What feasible interventions could be applied ? •Who are the appropriate target populations for the intervention ? •Is the intervention/service reaching its target population ? •Is the intervention being implemented in the ways envisioned ? •Is it effective ? •How much does it cost relative to its benefits and effectiveness ?  

3. Types

   • •Process evaluation –assess the extent and process of intervention implementation •Impact evaluation –assess the immediate effects of the intervention •Outcome evaluation –assesses the long-term effects of the intervention
   • Monitoring
     • systematic and continuous surveillance, concerned with procedure and process; performance indicators
   • Evaluation
     • Summative
       • assessment of the outcomes of an intervention has the programme/intervention reached its goals ? 'does it work'?
     • Formative
       • –assessment of the process of a programme »how the programme/intervention produces its effects 'how it works'??
   • Process evaluation
     • •Is the intervention being fully implemented ? –programme reach –programme implementation –quality of programme materials –participant reaction/satisfaction •Documentation for programme replication •Improve and develop the intervention •Interpretation of outcomes
     • Programme implementation
       • Approach or strategy
         • –intervention delivery - materials and programme components –intervention fidelity - delivered as intended –how variations in delivery affect intended outcomes
       • System or context
         • –organisational structure –management and staff practices –local contextual factors – ethos, culture, history
       • Conceptual Framework for Evaluating Implementation
         • •Clear articulation of programme theory –causal mechanisms underlying the change process     ‘action theory’ and ‘change theory’ •Theory informs the focus of the evaluation and clarifies the connections between an intervention’s operations and its effects –the theory of the change processes linking programme implementation with intermediate and early outcomes
         • Articulation of programme theory
           • Causative theories - mechanisms underpinning the intervention
           • Prescriptive theories - dynamic process of change
   • Impact and outcome evaluation
     • •Impact evaluation –immediate effects - have the programme objectives been achieved ? •Outcome evaluation –longer-term effects - have the programme goals been achieved ?
     • Identify goals & objectives
       • •Goal - abstract idealised statement of desired outcomes •Objective – SMART-operationalisation of the desired outcomes in measurable criteria of success •Formulation of agreed goals and objectives - collaborative and appropriate to the problem and target group in question
     • Criteria of success
       • –effectiveness - extent to which objectives are achieved –efficiency- the ratio of benefits to costs –equity- equal treatment or provision for equal need; reduce inequities –appropriateness - relevant to the needs of the target group –acceptability- target group, cultural sensitivity accessibility- time, location of services

4. Model

   • •Input –Resources - staff, skills, financial, materials, –organisational structure for activities •Process –intervention activities and their delivery •Outputs –implementation and impact of activities •Outcomes –outcomes or effects of the intervention

5. Role

   • •Evaluation research has a critical role to play in the development of practice and policy •Necessary for the ongoing survival and viability of services and programmes •Research as a practical activity which feeds into the debate about accountability, quality assurance and best practice •Policy-makers »accountability, added-value •Practitioners »confidence in the programme; practicality of implementation •Population »relevance to perceived needs; process •Researchers »quality of the evidence; knowledge base

6. issues

   • •Evaluation from whose perspective ? stakeholder groups - use of multiple criteria •Specify goals and objectives •Define target populations •Define the intervention and its delivery systems •Reach agreement on measurable and testable intervention activities & objectives •Select an evaluation approach  »take account of methodological and contextual factors