Assessment Reliability and Validity: Key Concepts

Understanding Reliability

Reliability refers to the degree to which an assessment tool produces stable and consistent results. It focuses on the consistency of the assessment.

• Interrater Reliability: Assesses whether different observers are consistent in their judgments.
• Test-Retest Reliability: Evaluates the consistency of a test across different administrations over time.
• Parallel Forms Reliability: Compares two different versions of a test that measure the same construct with different, but equivalent, questions.
• Internal Consistency Reliability: Measures the consistency of results across items within a single test. This can be assessed through:
  • Average Inter-Item Correlation: Calculates the average of all correlation coefficients between individual items on a test.
  • Split-Half Reliability: Divides a test into two halves and administers them to the same group. Reliability is indicated if students score similarly on both halves.
• Alternate Form Reliability: Involves administering two different versions of the same test at different times.
• Equivalence Form of Difficulty: Ensures that all tests have the same level of difficulty and establishes a passing score based on that difficulty level.

Reliability Coefficients:

• 0.60: Acceptable for administrative purposes and group scores.
• 0.80: Acceptable for screening purposes.
• 0.90: Required for important educational decisions regarding an individual.

Understanding Validity

Validity refers to how well a test measures what it is intended to measure. A test must be valid to be reliable.

• Face Validity: A subjective measure of how appropriate a test appears to be.
• Construct Validity: Experts evaluate each question to determine if it accurately assesses the intended construct.
• Criterion-Related Validity: Used to predict future or current performance by correlating test results with another criterion of interest.
• Formative Validity: Assesses how well a measure can provide information to improve the program under study.
• Sampling Validity: Ensures that the measure covers a broad and representative range of the content being assessed.

Validity Coefficients:

• 0.7 and up: Strong
• 0.50-0.69: Moderate-Strong
• 0.30-0.49: Moderate
• 0.10-0.29: Small
• Less than 0.09: Very Small

Statistical Measures in Assessment

Standard Deviation (SD) and Related Measures:

• Approximately 68% of scores fall within one standard deviation (-1 to +1) of the mean.
• Scores within one SD of the mean:
  • Percentage of scores: 34.13% (-1 to 0) and 34.13% (0 to +1)
  • Standard Score (SS): 85-115
  • Percentile Rank: 16-84 (16-25 is considered at risk)
  • Normal Curve Equivalent (NCE): 30-70
  • Z-score: -1 to +1 (same as SD)
  • T-score: 40-60
  • Stanine: 3.5-7.5

Z-Scores and T-Scores

Z-scores and T-scores indicate how many standard deviations a score is from the mean. T-scores are used when the sample size is small or the population standard deviation is unknown.

IQ Deviation

IQ deviation measures how an individual's IQ score deviates from the average score of 100, indicating their intelligence relative to their age group.

Criterion-Referenced vs. Norm-Referenced Tests

• Criterion-Referenced Tests: Compare a student's score to a predetermined standard or learning goal. The performance of other students does not affect an individual's score.
• Norm-Referenced Tests: Compare a student's performance to the performance of their peers at the same age or grade level.