Classification Accuracy and Classification Consistency

Using Expected Classification Accuracy and Classification Consistency to Guide the Test Development Process for an Adult Education Assessment with Multiple Cut Scores

Abstract – Measurement precision is an important component in the process of evaluating the validity of an assessment. One way to examine measurement precision is through the reliability coefficients of classical test theory. Another is through examination of consistency of scores across replications of a testing procedure. The Standards for Educational and Psychological Testing, states that the latter approach employs different methods to examine this consistency in terms of standard errors, reliability coefficients per se, generalizability coefficients, error/tolerance ratios, item response theory (IRT) information functions, or various estimates of classification consistency (AERA, APA, NCME, 2014). For criterion-referenced tests that classify examinees into performance categories, a primary focus of measurement precision should be the degree of classification accuracy (CA) or classification consistency (CC). The CASAS GOALS test series are examples of criterion-referenced tests developed on an underlying IRT scale with a principal purpose of correctly and consistently classifying examinees into performance categories for adult education students. By estimating and examining the CA and CC during the development process, CASAS ensures that the final test forms meet the desired CA and CC, while adhering to the test blueprint. This paper describes and summarizes a now popular approach for using an item response theory-based method to estimate CA and CC and examines how this information informs test construction considerations such as item selection and test length.