Testing and Evaluation

FPSC Part-III (Pedagogy) — Topic 3 of 4. Covers the concepts, types, and qualities of educational testing, measurement, assessment, and evaluation.

Key Definitions (in increasing order of scope)

Exam angle: Ordering from narrowest to broadest — Test → Measurement → Assessment → Evaluation — is a recurring MCQ.

Formative vs Summative Evaluation

• Formative Evaluation: ongoing, during instruction; purpose is to improve learning/teaching (e.g., quizzes, class questioning, drafts). Low/no stakes.
• Summative Evaluation: at the end of instruction; purpose is to judge/certify achievement (e.g., final exams, board exams). High stakes.
• Analogy often quoted: "When the cook tastes the soup, that's formative; when the guest tastes the soup, that's summative." — (Robert Stake)
• Example (Formative): Teacher checks students' rough drafts and gives feedback before the final essay is due.
• Example (Summative): The final board exam at the end of Grade 10 that decides promotion.

Norm-Referenced vs Criterion-Referenced Tests

• Norm-referenced: compares a student's performance to that of other students (a norm group); reports rank/percentile (e.g., most IQ tests, many entrance tests). Standard item difficulty is usually set around 50% so scores spread out and rank well. Example: "You scored in the 90th percentile" — meaning better than 90% of test-takers.
• Criterion-referenced: compares performance to a fixed standard/criterion regardless of others' scores (e.g., a driving test, mastery tests). Example: "You correctly solved 8/10 problems, which meets the 80% mastery standard" — everyone who reaches 80% passes, regardless of others.

Classification of Tests

Two main frameworks: by Method (how it's built) and by Purpose (why it's given), plus a few special-purpose test labels.

1. Classification by Method

A. Objective Type — one correct answer, high reliability (no grader bias)

• Selection Type (student picks the answer)
  • MCQs — most popular objective item
    • Parts: Stem (question — must be positive form, avoid "NOT"), Answer (correct choice), Distractors (wrong choices)
    • Rules: avoid qualifiers (often, sometimes, generally) & absolutes (always, never) — they hint the answer; keep all choices similar length
  • True/False — binary choice (True/False, Yes/No, Agree/Disagree)
    • Pros: objective, no grader bias
    • Cons: 50% guessing chance
  • Matching — links two related columns
    • Column A = Premises; Column B = Responses
• Supply Type (student produces the answer)
  • Short Answer — brief answer (word/phrase/number). Good for simple recall; not for problem-solving/creativity
  • Completion (Fill in the Blanks) — insert missing word. No options given — guessing eliminated

B. Essay (Subjective) Type — for complex outcomes: critical thinking, creativity, synthesis

• Restricted Response — limited scope, targets one specific SLO. e.g., "Describe two causes of global warming."
• Extended Response — free-form, integrates/evaluates ideas; easy to set the question, but scoring is subjective & less reliable

2. Classification by Purpose

• Teacher-Made Test — made by the class teacher; non-standardized; tracks day-to-day progress
• Standardized Test — made by experts, built from a Table of Specifications; high validity & reliability. e.g., IELTS

🧪

**Piloting:** a small-scale trial run of a standardized test on a sample group — checks feasibility, gathers feedback, fixes flaws — done **before** final rollout.

• Norm-Referenced (NRT) — compares to peers; gives rank/percentile; item difficulty usually ~50%. e.g., final semester exam — full comparison above ⬆
• Criterion-Referenced (CRT) — compares to a fixed standard; others' scores don't matter. e.g., driving test — full comparison above ⬆

3. Other Special Test Categories

• Maximum Performance Test — candidate performs to their best ability; measures aptitude/achievement/potential. (Contrast — Typical Performance Test: measures usual behavior, e.g., a personality/interest inventory; no right/wrong answer.)
• Power Test — hard items, generous/unlimited time; measures depth of mastery. (Contrast — Speed Test: easy items, strict time limit; measures speed, not depth.)

Other Purpose-Based Labels (also commonly tested)

• Achievement — what's been learned. e.g., end-of-chapter test
• Aptitude — potential to learn. e.g., engineering-aptitude test
• Diagnostic — pinpoints exact weakness. e.g., which fraction skill is missing
• Prognostic — predicts future success. e.g., pre-admission test

Characteristics of a Good Test

• Validity: the test measures what it is intended to measure (the most important quality). Types: Content validity (covers the intended content/curriculum — e.g., a Grade 8 Math test only asks Grade 8 syllabus questions), Construct validity (measures the underlying trait, e.g., "intelligence"), Criterion-related validity (correlates with an external, established measure — concurrent or predictive — e.g., a new entry test's scores match students' later first-year results).
• Reliability: consistency of scores across time/forms/raters. Methods: Test–retest (same test, two occasions — e.g., the same IQ test given a month apart gives similar scores), Split-half (correlating two halves of one test), Equivalent/parallel forms (two different but equivalent versions).
• Objectivity: different examiners scoring the same response arrive at the same score.
• Practicality/Usability: reasonable cost, time, ease of administration and scoring.

Exam angle: A test can be reliable without being valid, but it cannot be valid without being reliable — validity is the more important and inclusive quality, but reliability is a necessary (not sufficient) condition for validity.

Continuous Assessment

• Ongoing evaluation throughout the term/year (assignments, quizzes, class participation) combined with terminal exams — aims to reduce over-reliance on one final exam and better reflect overall learning.

Classroom Example (Teacher–Student)

🗣️

Scenario: After a weekly quiz on fractions, Hamza scores 4/10.

Formative use of the result: Teacher: "Hamza, I can see you're mixing up the steps when denominators are different. Let's redo two questions together on the board before the next quiz." The score is used to adjust teaching, not to grade him harshly — this is formative evaluation in action.

Summative use of the result: At the end of the term, the same type of quiz score (now averaged across the term) is entered into Hamza's report card as part of his final Mathematics grade — this is summative evaluation.

Validity vs reliability check: The teacher notices the quiz used long, wordy story problems, and several students with weak English (not weak Math) scored low. The test may be reliable (consistent scores across two similar quizzes) but not fully valid for measuring Math ability, since it is partly measuring English reading skill instead.

How to Implement in Your Classroom

To make classroom tests more valid and reliable:

1. Write test items directly from your stated learning objectives (content validity) — don't test what you didn't teach.
2. Mix item difficulty and Bloom's levels — don't rely only on recall-type (Knowledge) questions.
3. Use clear, simple language so you're testing the subject, not reading ability (unless that's the point).
4. Pilot a new test on a small group first, or compare with a previous similar test, to check consistency (reliability).
5. Use a marking scheme/rubric for subjective items so different markers (or the same marker on different days) would give the same score (objectivity).

To build continuous assessment into your term:

1. Combine small, frequent formative checks (weekly quiz, class questioning, homework) with fewer high-stakes summative tests (mid-term, final).
2. Give feedback on formative work quickly enough that students can act on it before the summative test.
3. Keep a simple running record per student so patterns (like Hamza's fractions gap) are visible before the final exam, not after.

---

MCQ Traps

• Order from narrowest to broadest: Test → Measurement → Assessment → Evaluation — reversed order is a common wrong option.
• Validity ≠ Reliability: a bathroom scale that always reads 5 kg too heavy is reliable (consistent) but not valid (not accurate) — a classic exam example.
• Formative = during/for improvement; Summative = end/for judgment — often swapped in question stems.
• Norm-referenced compares students to each other; criterion-referenced compares to a fixed standard — don't reverse.
• Split-half and test–retest are both reliability methods, not validity methods — a frequent confusion.
• Supply Type = student writes the answer (Short Answer, Completion); Selection Type = student picks the answer (True/False, Matching, MCQs) — both are still Objective, not Essay.
• Extended Response = no length/structure limit; Restricted Response = limited to a set point/length — both are Essay type, don't confuse with Objective type.
• MCQs are Selection + Objective, never classify them as Essay type — a common careless trap.
• An MCQ's Stem must be positive, not negative ("NOT") — negative stems are considered poor item-writing and are a common trap.
• In Matching, Column A = Premises, Column B = Responses — the two labels are frequently swapped as a distractor in MCQs about test construction.
• Table of Specifications = the blueprint used to build a Standardized test — don't confuse it with a marking rubric (used for scoring essays).
• Piloting happens before final rollout of a standardized test — not after.
• Power Test = hard items + generous time; Speed Test = easy items + strict time limit — a frequently reversed pair.
• Maximum Performance Test (best-effort, right/wrong answers exist) vs Typical Performance Test (usual behavior, e.g. personality tests, no right/wrong answer) — don't mix these up.

Frequently Asked MCQs (Past Papers)

• Q: Which quality of a test refers to consistency of results? — A: Reliability.
• Q: Which quality of a test refers to measuring what it claims to measure? — A: Validity.
• Q: A test given at the end of a course to assign final grades is called ___ evaluation. — A: Summative.
• Q: A test given during instruction to provide feedback and improve teaching is called ___ evaluation. — A: Formative.
• Q: Comparing a student's score to a fixed passing standard (not to other students) is an example of a ___ test. — A: Criterion-referenced.
• Q: Correlating two halves of the same test to check consistency is called the ___ method. — A: Split-half (reliability) method.
• Q: Matching, True/False, and MCQs fall under which type of test item? — A: Objective — Selection Type.
• Q: Short Answer and Completion items fall under which type? — A: Objective — Supply Type.
• Q: An essay answer with no fixed length or structure is called ___ response. — A: Extended response.
• Q: An essay answer confined to a specific point or length is called ___ response. — A: Restricted response.
• Q: What are the three structural parts of an MCQ? — A: Stem, Answer, and Distractors.
• Q: In a Matching item, what is Column A called, and what is Column B called? — A: Column A = Premises; Column B = Responses.
• Q: A standardized test is built from which blueprint document? — A: A Table of Specifications.
• Q: What is "Piloting" in standardized test development? — A: A small-scale trial run to check feasibility and fix flaws before final use.
• Q: Which test type uses hard items with generous/unlimited time? — A: Power Test.
• Q: Which test type uses easy items with a strict time limit? — A: Speed Test.
• Q: A personality inventory with no right/wrong answer is an example of which category? — A: Typical Performance Test.
• Q: Arrange in order of increasing scope: Test, Evaluation, Measurement, Assessment. — A: Test → Measurement → Assessment → Evaluation.

Self-Test Questions

1. Differentiate between test, measurement, assessment, and evaluation.
2. Explain formative and summative evaluation with one classroom example each.
3. What is the difference between norm-referenced and criterion-referenced testing?
4. List and briefly define the three types of validity.
5. Why can a test be reliable but not valid? Give an example.
6. Sort these into Supply/Selection/Essay: MCQ, Short Answer, Extended Response, Matching, True/False, Completion, Restricted Response.
7. Name the three parts of an MCQ and state the rule for wording the stem.
8. Differentiate a Power Test from a Speed Test.
9. Differentiate a Maximum Performance Test from a Typical Performance Test.
10. What is "Piloting," and at what stage of test development does it happen?