III.5 Program Evaluation — Policy and Programme Evaluation

🧠

Key points — 30-second recap

• Programme / policy evaluation = the systematic assessment of a programme's design, process and results — did it achieve its objectives efficiently, fairly and sustainably?
• Approaches: process · outcome · impact (with counterfactual) · formative vs summative (Scriven 1967) · efficiency / economic (CBA, CEA) · participatory · theory-based.
• Four distinguishing bases (2023 Q2): Timing · Purpose · Focus · Method.
• OECD-DAC criteria: relevance · coherence · effectiveness · efficiency · impact · sustainability.
• Results chain: inputs → activities → outputs → outcomes → impact.
• Hook: Monitoring asks "Are we doing it?"; evaluation asks "Did it work, and was it worth it?"

📜

Syllabus item III.5: "Program Evaluation" (renumbered from old III.4).

1. Meaning, Purpose and Scope

• Definition: policy evaluation is a systematic, evidence-based assessment of a policy or programme's design, implementation and results.
• Carol Weiss (paraphrase): assess programme operations and outcomes against standards to support improvement.
• Central question: what worked, for whom, why, at what cost, and under which conditions?
• Scope: examine intended results, unintended effects, distribution and whether benefits can last.
• Policy versus programme: a policy sets the wider direction; a programme is an organised intervention used to implement it. Programme evaluation may assess design, delivery, outcomes and costs together.
• Simple example: for a girls' schooling stipend, checking payments is insufficient; evaluate attendance, inclusion, learning and value for money.

Timing and historical accuracy

• Evaluation commonly examines ongoing and completed interventions; its planning should begin during policy design.
• Front-end work assesses needs, logic and readiness before delivery. Forecasting alone does not establish achieved impact.
• Avoid the claim that evaluation simply “originated in Canada and the USA in the 1970s.” Canada's federal evaluation policy dates to 1977, but official Canadian records describe evaluation activity in the 1960s.
• Sources: OECD recommendation on public policy evaluation; Canada's evaluation-policy history; Auditor General of Canada, 1983.

2. Why Evaluate?

• Accountability — to legislature, auditors, taxpayers, donors.
• Learning — what works and why.
• Better resource use — shift money to what works.
• Decision-making — continue, expand, modify or terminate.
• Feedback — closes the policy cycle.

Two complementary perspectives

• Connection: reliable evidence supports accountability; public scrutiny makes evidence harder to suppress.
• Qualification: evaluation informs judgments about trust and legitimacy; it does not automatically establish either.

Three core evaluation questions

• Goals: did the policy achieve its intended objectives?
• Other effects: what unintended benefits or harms occurred, and for whom?
• Value for money: could comparable results have been achieved at lower cost?

Memory cue: Goals → Effects → Costs. Also check equity, sustainability and credible attribution; these three questions are a starting point, not the whole evaluation.

Who can conduct evaluation?

• Evaluations may be internal, external or jointly conducted.
• Independence is supported by transparent methods, conflict management, quality assurance and publication; an external label alone does not guarantee objectivity.
• Reference: HM Treasury Magenta Book, managing an evaluation.

3. Monitoring vs Evaluation

4. Approaches to Evaluation (2023 Q2)

• Process (implementation) evaluation
  • Was the programme delivered as designed?
  • Example: were subsidised-flour shops opened on time and in the right places?
• Outcome evaluation
  • What changed for the target group?
  • Example: did malnutrition fall?
• Impact evaluation
  • How much change was caused by the programme, compared with what would have happened anyway (the counterfactual)?
  • Methods: randomised control trials, difference-in-differences, matching.
• Formative vs summative (Michael Scriven, 1967): improvement versus overall judgment. See section 5 for timing, examples and how these overlap with other approaches.
• Efficiency / economic evaluation
  • CBA: compare social costs and benefits in money over an appropriate time horizon, discounting future values. Net present value = present value of benefits minus present value of costs. Consider distribution, unmonetised effects and uncertainty as well as the headline result.
  • Cost-effectiveness: compare cost per unit of a specified result, such as each additional child attending school. Choose comparable outcomes; cheap delivery is not necessarily effective delivery.
  • These methods can support appraisal before adoption and evaluation using observed costs and results. HM Treasury Green Book.
• Participatory evaluation — beneficiaries help judge the programme.
• Theory-based evaluation — tests the programme's theory of change link by link.
• Pseudo / political vs genuine evaluation (Dunn) — PR exercises vs objective measurement.

Choosing the right tool

MCA and economic-impact analysis are explained in III.3 Policy Analysis. Use them where the question warrants them; do not present forecasting or “if–then” reasoning as proof of achieved impact.

5. Distinguishing the Approaches — Four Bases

Start here: four questions, one evaluation

Think of evaluating a school-support scheme from four angles:

Running example — hypothetical: a provincial government gives stipends to low-income families to help girls attend school. All figures below are invented for learning, not findings about an actual programme.

Pakistan application — use a real programme with the four questions

Real anchor: BISP commissioned evaluation work on Waseela-e-Taleem. The table below shows how to apply the four bases to an education cash-transfer programme; it does not claim that the actual report used every approach listed. BISP evaluation record.

One sentence to remember: A study of Waseela-e-Taleem could be summative in purpose, impact-focused, conducted after a funding phase and use mixed methods—the four labels answer different questions.

A documented correction example: in September 2020, Ehsaas introduced a portal for eligible people facing payment problems, including biometric verification difficulties. This illustrates feedback used to improve delivery; the announcement alone does not establish that a formal formative evaluation caused the change. Official record.

Use the numerical examples below only as invented teaching calculations. For the historical BISP/WeT enrolment estimates and their methods, see the Pakistan examples section at the end of this page.

A. Purpose — Why evaluate?

1. Formative evaluation: improve the programme while it can be adjusted

• In simple words: find problems early and fix them.
• Practical example: during a pilot, mothers report that payment centres are too far away. The department introduces nearer collection points.
• Use: improve design, access or delivery during development, piloting or implementation.
• Exam sentence: “Formative evaluation provides feedback for improvement; for example, beneficiary interviews may reveal payment barriers and guide changes before wider rollout.”

2. Summative evaluation: judge the programme's overall merit

• In simple words: assess whether the programme has been worthwhile.
• Practical example: after a funding phase, assess attendance gains, costs and exclusion before deciding whether to renew or expand the stipend.
• Use: support continuation, expansion, redesign or termination.
• Exam sentence: “Summative evaluation assesses overall performance at a decision point and informs whether an intervention should continue, change or end.”

Remember: formative = improve; summative = judge. Summative evaluation does not require permanent programme closure, and formative evaluation is not limited to the period before launch.

Where does programme evaluation fit? It is the umbrella term for assessing a programme. A single programme evaluation can contain formative, summative, process, impact and economic components.

B. Focus — What are we examining?

1. Process evaluation: was it delivered properly?

• In simple words: check how the work was carried out.
• Practical example: did eligible families receive the correct stipend on time? Were remote households excluded? Why were payments delayed?
• Evidence: payment records, staff interviews and beneficiary complaints.
• Exam sentence: “Process evaluation examines implementation quality and explains delivery gaps, such as delayed payments or exclusion of eligible beneficiaries.”
• Limit: successful payment delivery alone does not prove better school attendance.

2. Outcome evaluation: what changed?

• In simple words: check whether the intended result occurred.
• Practical example: regular attendance rises from 60% to 75% among participating girls.
• Evidence: reliable attendance records before and during the programme.
• Exam sentence: “Outcome evaluation measures changes in target results, such as improved attendance, without necessarily establishing that the programme caused them.”
• Limit: better transport or a new school could also explain the improvement.

3. Impact evaluation: did the programme cause the change?

• In simple words: separate the programme's contribution from changes that would have happened anyway.
• Practical example: attendance rises by 15 percentage points in programme schools and by 5 in comparison schools. The difference is 10 percentage points.
• Condition: interpreting that difference as causal requires a credible design and assumptions, including comparable trends without the programme and no different external shock.
• Exam sentence: “Impact evaluation estimates the intervention's causal contribution by comparing observed results with a credible counterfactual—what would have happened without it.”
• Limit: a simple before–after comparison cannot establish causation.

4. Efficiency / economic evaluation: were the results worth the resources?

• In simple words: relate results to their cost.
• Practical example: Scheme A costs Rs 1 million and produces 100 additional regular attendees; Scheme B costs Rs 1.2 million and produces 150. Costs per additional attendee are Rs 10,000 and Rs 8,000 respectively.
• Interpretation: B is more cost-effective on this measure if outcomes, periods, cost coverage and estimates are comparable. Check fairness and quality too.
• Exam sentence: “Economic evaluation examines value for money through cost-effectiveness or cost-benefit analysis, rather than treating low expenditure alone as success.”
• Distinction: cost-effectiveness measures cost per specified result; cost-benefit analysis compares social costs and benefits in monetary terms.

Memory cue: Process = delivered → Outcome = changed → Impact = caused → Economic = worthwhile.

C. Timing — When do we assess?

1. Front-end analysis / ex-ante appraisal: before delivery

• In simple words: check the problem and proposed solution before committing resources.
• Practical example: investigate whether girls miss school because of poverty, unsafe transport or the absence of a nearby school.
• Exam use: “Front-end analysis tests needs and design assumptions before implementation; projected benefits are not evidence of achieved impact.”

2. Mid-term evaluation: during delivery

• In simple words: take stock while the programme is running.
• Practical example: halfway through the funding phase, investigate low attendance despite timely payments.
• Exam use: “Mid-term evaluation assesses progress and supports timely adjustment.”
• Distinction: mid-term describes when; formative describes why. A mid-term study may also inform a major funding judgment.

3. Ex-post evaluation: after a programme or phase

• In simple words: look back at results once there has been time for effects to emerge.
• Practical example: examine whether attendance gains continued after a stipend phase ended.
• Exam use: “Ex-post evaluation assesses realised results and their persistence after an intervention or phase.”

Related readiness check — evaluability assessment

• In simple words: can we evaluate this programme credibly and usefully?
• Practical example: check whether goals are clear, baseline records exist, comparison options are ethical and the evaluation has adequate funding.
• Action: if attendance records are unreliable, improve recording before promising a precise impact estimate.
• Exam use: “Evaluability assessment establishes whether the questions, data, design and resources permit a useful evaluation.”
• Important: undertake it early where possible and revisit it as needed. It is not a stage restricted to implementation.

Timing caution: impact evaluation can examine an ongoing programme after sufficient exposure; it need not wait for permanent closure.

D. Method — How do we obtain evidence?

1. Quantitative: use numbers

• Practical example: analyse attendance rates, payment delays and cost per additional attendee.
• Best for: estimating how much, how many and how results vary.
• Limit: numbers alone may not explain why families face barriers; causal claims require an appropriate design.
• Exam sentence: “Quantitative methods measure the scale and distribution of results using numerical evidence.”

2. Qualitative: investigate experiences and explanations

• Practical example: interview mothers, girls and teachers to understand why attendance remains low.
• Best for: understanding how delivery works, why barriers occur and how people experience the policy.
• Limit: a few interviews do not establish how common a problem is across the whole population.
• Exam sentence: “Qualitative methods explain experiences, mechanisms and contextual barriers through interviews, observation and case studies.”

3. Mixed methods: combine numbers and explanations

• Practical example: records show lower attendance in remote villages; interviews identify transport insecurity as a possible explanation.
• Best for: connecting the scale of a problem with an explanation of how it arises.
• Limit: findings must be integrated and disagreements investigated, not merely placed side by side.
• Exam sentence: “Mixed-method evaluation combines numerical patterns with contextual explanations to produce a fuller assessment.”

E. Put the four bases together

Example: at the end of a stipend phase, evaluators assess whether it deserves renewal, estimate attendance effects using a credible comparison, and interview families.

• Purpose: summative — judge whether to renew.
• Focus: impact — estimate the programme's causal contribution.
• Timing: ex-post for that phase.
• Method: mixed — combine numerical analysis and interviews.

Lesson: these labels describe different features of the same evaluation. They are not mutually exclusive types.

F. Compact CSS answer

“Approaches to policy evaluation differ along four bases: purpose, focus, timing and method. By purpose, formative evaluation supports improvement, whereas summative evaluation judges overall merit. By focus, process evaluation examines delivery, outcome evaluation measures changes, impact evaluation investigates causation, and economic evaluation assesses value for money. By timing, assessment may occur before, during or after an intervention; evaluability assessment checks whether a credible study is feasible. By method, quantitative approaches measure patterns, qualitative approaches explain experiences and mechanisms, and mixed methods combine them. For example, an evaluation of a girls' stipend may be summative in purpose, focused on impact, conducted after a funding phase and use mixed methods. The choice should follow the evaluation question, available evidence and intended decision.”

For a longer answer: expand each distinction with one example and limitation. Recall Why → What → When → How.

Reference for evaluation design and methods: HM Treasury Magenta Book.

6. Criteria: What Counts as Success?

A. OECD-DAC framework

• Relevance: fit with needs — does the stipend address the actual attendance barrier?
• Coherence: fit with other interventions — does transport provision support the stipend?
• Effectiveness: achievement of objectives — did attendance reach its target?
• Efficiency: results relative to resources — what is the cost per additional regular attendee?
• Impact: significant wider effects — does the intervention affect progression, wellbeing or inequality?
• Sustainability: durability of benefits — can gains continue with reliable funding and institutions?

Source: OECD evaluation criteria. These are criteria for judgment; they do not prescribe a single evaluation method.

B. Six policy-analysis criteria

Use this complementary framework when the question asks about effectiveness, efficiency, adequacy, equity, responsiveness and appropriateness.

Key distinction: meeting a modest target may show effectiveness without adequacy. A low average cost may show efficiency while concealing inequity.

Memory cue: Results → Resources → Sufficiency → Fairness → Needs → Suitability. Keep this framework separate from the six OECD-DAC labels.

7. Steps in Programme Evaluation — With a Running Example

Use a hypothetical girls' schooling stipend throughout an answer.

1. Define questions and intended use.
   • Ask whether attendance improved, for whom and at what cost.
   • Identify the decision: continue, redesign, expand or stop.
2. Map the theory of change and results chain.
   • Funds → verified payments → households receive stipends → attendance improves → longer-term educational gains.
   • Check assumptions: accessible schools, safe transport and teaching capacity.
3. Set indicators, baseline and evaluation readiness.
   • Record initial attendance, eligible population, payment timeliness and costs.
   • Disaggregate by location, poverty and disability; verify data access and ethics.
4. Choose design and methods.
   • Combine administrative records and interviews; use a credible comparison design for causal questions.
   • Explain assumptions rather than treating a comparison group as automatically valid.
5. Collect, validate and analyse evidence.
   • Cross-check registers; investigate missing records and exclusions.
   • Compare results and costs, and examine unintended effects.
6. Report findings and limitations.
   • State what changed, strength of attribution and who benefited.
   • Separate verified findings from uncertainty.
7. Act and follow up.
   • Recommend continuation, redesign, expansion or termination with reasons.
   • Assign responsibility and deadlines; assess whether agreed changes occur.

Recall: Question → Logic → Indicators → Design → Evidence → Report → Action.

8. Worked Example — Conditional Cash Transfer for Girls' Schooling

• Process: were payments made on time to the right households?
• Outcome: did enrolment and attendance rise?
• Impact — hypothetical illustration:
  • attendance rises 12 percentage points in programme schools and 5 in comparison schools: estimated difference-in-differences = 7 percentage points.
  • A causal interpretation requires credible assumptions, including parallel trends without the programme and no confounding differential shock; similarity alone is insufficient.
• Efficiency: cost per additional girl enrolled.
• Formative: mid-term interviews reveal transport problems → adjust.
• Summative: final evaluation decides scale-up.

9. Challenges and Ways Forward

Three safeguards: design evaluation early → assess evidence independently → connect findings to decisions.

10. 🇵🇰 Evaluation in Pakistan

• Existing machinery — PC-III (progress), PC-IV (completion) and PC-V (operation) reports; Planning Commission's monitoring wing; provincial M&E directorates; Auditor-General's performance audits; donor evaluations (World Bank, ADB, DFID / FCDO).
• Good example — BISP and its Waseela-e-Taleem conditional cash transfer were studied with impact evaluations.
• Weaknesses — focus on financial progress (money spent), not outcomes; few independent evaluations; reports not published; little feedback into new PC-Is.
• Reforms — national evaluation policy; independent evaluation unit; results-based budgeting; publish evaluations; train officers.

11. Critical Evaluation

• Evaluation turns policy from guesswork into learning.
• Exam verdict: A programme that is never evaluated can never be improved — evaluation closes the policy cycle and feeds the next plan.

12. 📌 Past Paper Questions — CSS 2015–2026

Subjective questions

• 2023 · Q2 · 20 marks — Approaches to policy and programme evaluation and how they differ. → 2023 · Q2 · Approaches to policy and programme evaluation (20 marks)
  • Use: sections 4–5 + example.
• 2025 · Q4(a) · part of 20 marks — Define policy evaluation (with policy analysis). → 2025 · Q4 · Policy analysis, evaluation and implementation reform (20 marks)
  • Use: sections 1–3.

MCQ practice

• Formative and summative evaluation were distinguished by: (A) Scriven (B) Weiss (C) Dunn (D) None of these → ✅ A.
• Which is NOT an OECD-DAC criterion? (A) Relevance (B) Coherence (C) Popularity (D) Sustainability → ✅ C.
• Comparing results with what would have happened without the programme means using a: (A) baseline (B) counterfactual (C) logframe (D) None of these → ✅ B.
• In Pakistan, the project completion report is: (A) PC-II (B) PC-III (C) PC-IV (D) None of these → ✅ C.

13. 🎯 CSS Exam Perspective

13.1 How often it comes

• One full question (2023) and part of 2025 Q4.

13.2 Likely 2027 angles

• Monitoring vs evaluation.
• Types / approaches of evaluation with examples.
• Why evaluation is weak in Pakistan.

13.3 Outline for a 20-mark question

• Q: Discuss the various approaches to programme evaluation and how they differ.
  • I. Introduction — definition (OECD, Weiss)
  • II. Purpose of evaluation
  • III. Monitoring vs evaluation (table)
  • IV. Approaches
    • A. Process
    • B. Outcome and impact
    • C. Formative and summative
    • D. Economic
    • E. Participatory and theory-based
  • V. Four bases of difference
  • VI. OECD-DAC criteria
  • VII. Pakistan — PC-III to PC-V; weaknesses; reforms
  • VIII. Conclusion

13.4 How to score

• Draw the results chain.
• Use one programme throughout as the example.
• Separate monitoring from evaluation.

13.5 Common mistakes

• Treating before–after change as proof of impact.
• Mixing purpose, focus, timing and methods into one exclusive list.
• Confusing effectiveness with adequacy, or efficiency with equity.
• Restricting evaluability assessment to implementation.
• Assuming summative evaluation always requires permanent programme closure.
• Presenting a hypothetical EPA, metro or drinking-water example as a verified evaluation finding.

13.5A Exam-ready paragraph

Policy evaluation systematically assesses the design, delivery and results of public interventions to support accountability and learning. Its approaches differ by purpose, focus, timing and method. Formative evaluation improves delivery, while summative evaluation judges overall merit. Process evaluation examines implementation; outcome evaluation identifies changes; impact evaluation investigates whether the intervention caused those changes. Economic evaluation considers value for money. For example, a hypothetical girls' schooling stipend can be evaluated through payment records, attendance data, interviews and a credible comparison design. A rise in attendance alone does not prove impact, and an average improvement may conceal exclusion of remote households. Therefore, evaluation should combine appropriate criteria, reliable evidence and transparent limitations, with findings linked to decisions on continuation, redesign or termination.

Adapt to the question: expand each named approach into definition → question → example → limitation. Use the outline above for a full-length answer.

13.6 Active recall

• Five approaches with examples.
• Six OECD-DAC criteria.
• Four bases of difference.

🇵🇰 Pakistan examples — understand, apply and cite

How to use: choose the case that proves your point. “Exam sentence” is original model wording, not a quotation from an official. Facts are tied to the linked source and reporting period; practical applications are analytical explanations.

1. BISP evaluation — a real programme with an evaluation history

• Documented fact: BISP commissioned Oxford Policy Management to evaluate impacts, with follow-up surveys compared against a 2011 baseline; WeT also had baseline and follow-up work. Source.
• Plain meaning / application: This is a factual Pakistan example of building evaluation into programme management. Merely having a baseline is not sufficient to establish causality.
• Exam sentence: BISP's commissioned evaluations illustrate the use of systematic evidence to assess social-protection performance.
• Qualification: Do not assume every BISP study used the same design.

2. Waseela-e-Taleem — an attributable result to cite carefully

• Documented fact: The World Bank's 2017 technical assessment (p. 46) reports a 10-percentage-point primary-enrolment effect for combined BISP income support and WeT, and a 9-point marginal WeT effect. Source.
• Plain meaning / application: This is a historical education result. It concerns enrolment, not automatically learning quality, and a percentage-point change is different from a percent increase.
• Exam sentence: The World Bank's 2017 assessment reports a 10-percentage-point primary-enrolment effect for combined BISP transfers and WeT, illustrating the value of impact evidence.
• Qualification: The combined estimate used regression discontinuity; the marginal estimate used propensity-score matching. Keep populations, methods and period attached to each estimate.

3. Punjab land records — output versus service outcome

• Documented fact: A World Bank account in September 2017 reported 144 new land-record offices and much faster access to registered records. Source.
• Plain meaning / application: Offices and digitised files are outputs; users' waiting time and access are service outcomes. A causal corruption claim would need additional evidence.
• Exam sentence: Punjab's land-record reform illustrates why evaluation should move from counting offices to examining the citizen's experience.
• Qualification: A programme report supports reported performance, not every possible causal claim about reform.

4. Peshawar BRT — inclusion and mixed evidence

• Documented fact: ITDP's December 2024 case study highlights women's access and participation by operators and community groups. Source.
• Plain meaning / application: Illustrative evaluation: combine usage records with passenger interviews about safety, affordability and reaching stations. This is a proposed method, not a claim about a particular evaluation design.
• Exam sentence: Peshawar BRT offers a practical case for combining transport-performance indicators with users' accounts of accessibility.
• Qualification: Usage figures alone do not establish causal effects on employment or overall social welfare.