Please enable JavaScript.
Coggle requires JavaScript to display documents.
Program Evaluation (MHPE – PRE) - Coggle Diagram
Program Evaluation (MHPE – PRE)
Basics of Evaluation
Definition: systematic collection & analysis of info on a program's design, implementation & outcomes to improve it (AMEE 67)
Evaluation = value judgment about merit / worth
"Program" = any organised action: session, course, curriculum, service, policy
Evaluation vs Assessment vs Appraisal
Assessment: measuring learners' achievement
Appraisal: two-way dialogue on personal & professional needs
Evaluation: judging a program, not individuals
Evaluation vs Research
Evaluation: stakeholder-driven, local, actionable, cyclical
Research: curiosity-driven, adds to generalizable knowledge
Both methods-neutral: same designs & methods
Programs are about change
Intended vs unintended change
Positive vs negative outcomes
Planning–evaluation cycle
Evaluability: clear, measurable goals & objectives
Needs → indicators → effectiveness → recommendations
Theories behind models
Reductionism
Cause-effect, linear, predictable
Whole understood through its parts
→ Experimental / quasi-experimental, Kirkpatrick
General System Theory (Bertalanffy)
Whole greater than sum of parts
Open vs closed systems
Equilibrium vs steady state
Equifinality: same outcome via different paths
Isomorphism: shared principles across fields
→ Logic model (AMEE)
Complexity Theory
Nonlinear: small change → large effect
Emergence, uncertainty & ambiguity expected
Context & relationships central
→ CIPP; Logic model (Ahmady)
History of approaches
Utilitarian: objectives-oriented, management-oriented
Intuitionist-pluralist: naturalistic, participant-oriented
Older: linear, product-based
Newer: multidimensional, process-based
Types of Evaluation
By timing
Formative: during program, for improvement
Summative: end of program, for judgment
Needs assessment: before program
By approach
Decision-oriented
Outcomes-oriented
Expertise-oriented (e.g. accreditation)
Goal-based: predetermined objectives as standard
Theory-driven: based on theory of change
Models
Experimental / quasi-experimental
Randomized controlled trial
Intact-group design
Time-series design (pre/post)
Ex post facto design
Internal vs external validity
Shows outcomes, not why
Kirkpatrick's 4 levels
Level 1: Reaction / satisfaction
Level 2: Learning (knowledge, skills, attitudes)
Level 3: Behaviour change
Level 4: Results / impact
New World Kirkpatrick: chain of evidence, not causal chain
Criticism: ignores context, process, stakeholders
Logic Model
Situation / problem
Assumptions
Inputs: resources
Activities
Outputs: products of activities
Outcomes: short, medium, long-term
Impact: wider system change
External factors / context
Theory of change
Term coined by Wholey (1979)
Pitfall: linearity; add feedback loops
CIPP
Context: What needs to be done?
Input: How should it be done?
Process: Is it being done?
Product: Is it successful?
"Not to prove but to improve"
C, I, P formative; Product summative
Sustainability & transportability
Utilization-focused
Intended use by intended users
Participatory / collaborative / empowerment
Stakeholders co-design & co-conduct
Fetterman's 5 tools: culture of evidence, critical friend, reflection–action cycle, community of learners, reflective practitioners
Before–During–After model
Planning & Conducting
Stakeholders & key evaluation questions
Indicators & standard of acceptability
Evaluation proposal format
Design
Mixed methods
Triangulation
Cross-sectional
Pre–post comparison
Retrospective record review
Quantitative: causality & magnitude
Qualitative: experience, "how" & "what"
Causality
Attribution vs contribution
Contribution analysis (Mayne): 6 steps, contribution story
Data collection
Questionnaires: validated vs adapted, Likert, open & closed items
Interviews / key informant interviews
Focus groups (4–8 people)
Snowball review
Nominal group technique
Document review & observation
Tracer survey
Bibliometric search
Sampling
Census (total population)
Purposive sampling
Selection bias
Analysis
Descriptive statistics
Paired t-test / Wilcoxon signed-rank
Regression
Thematic & content analysis (Framework method)
Quality of measures
Reliability: consistency
Validity: accuracy
Pilot testing
Role of the Evaluator
Internal vs external evaluator
Engage stakeholders & intended users
Give specific, actionable recommendations
Facilitate use of findings
Communicate well & sustain engagement
Ethics
Informed consent & voluntary participation
Confidentiality, de-identification, aggregate reporting
Ethics review: needed if intent to publish / generalize
Conflict of interest disclosure
Stakeholder pressures: preset findings, altering or suppressing results (Morrison)
Worthen's 7 ethical standards
Service orientation
Formal agreements
Rights of human subjects
Complete & fair assessment
Disclosure of findings
Conflicts of interest
Fiscal responsibility
AEA guiding principles
Systematic inquiry
Competence
Integrity / honesty
Respect for people
Responsibility for general & public welfare
Program Evaluation Standards (JCSEE)
Utility
Feasibility
Propriety
Accuracy
Evaluation accountability
Meta-evaluation: evaluating the evaluation
Purpose of Evaluation
Improve implementation & effectiveness
Accountability to funders & stakeholders
Justify or secure funding; manage resources
Accreditation & academic standards
Social accountability
Evidence-based decisions & policy
Feedback to faculty; appraisal & promotion
Document intended & unintended outcomes
Internal vs external reasons
Report & Use of Findings
Report organised by domain, judged against standards
Actionable recommendations
Limitations
Limited generalizability
Social desirability bias
Recall bias
Incomplete records
Challenges
Timeliness
Measuring only what is easy
Survey fatigue & low response rates
Balancing positive & negative
Dissemination
Technical report
Summary, slides, memo, video
Debrief / feedback session
Communication plan
Match users' information-processing style
Types of use
Instrumental: drives decisions
Conceptual: changes understanding
Enlightenment: adds to the field
Process: change from taking part
Symbolic: done for show
Factors increasing use
Stakeholder involvement
Implementation quality & communication
Decision-maker characteristics
Evaluation capacity building (CLIPs, communities of practice)