In this post we cover four potential check for evaluating whether a standard is set correctly.
When possible, external checks of the correctness of the standard should be implemented. Since there is not a definitive correct answer, these checks can only be considered as rough guides to help decide whether the standard is set correctly. “A consistent pattern of results supporting the appropriateness of the proposed passing score can provide convincing evidence for the standard, and a pattern suggesting that the proposed interpretation is inappropriate can provide convincing evidence against the standard” (Kane, 1994, p. 449). Kane suggested four potential checks to determine if a standard is set correctly.
Since most exams requiring a standard are developed to determine if a person has enough knowledge and skills to effectively engage in certain activities, Kane suggested allowing the examines to take the test and engage in the activity. Evidence of the validity of the test would be established if those scoring high on the test were skilled in the activities and vice-versa. Unfortunately, depending on the nature of the test content, this method of checking may have significant drawbacks. For example, if the test is designed to certify medical doctors, allowing examinees that failed the exam to practice medicine may have serious consequences.
Conduct a new or parallel standard-setting study with a different set of judges, and using a different methodology. Since standard-setting methodologies often arrive at different standards, this check can only give a general guideline. A very large discrepancy between methods would likely indicate a problem with one or both of the studies.
Compare the pass rates resulting from the standard to pass rates from other exams. For example, suppose the population of nurses has not changed, the pass rates from the previous year licensure test should be similar to the new pass rates.
Check the overall pass rates to verify they are consistent with the population. For example “we would probably expect the pass rate for a group of experienced, successful professionals in a profession to be very high if they were administered the licensing examination in their profession. On the other hand, we might expect a group of examinees with little education or experience related to the profession to have a low pass rate” (Kane, 1994, p. 453).
REFERENCES
Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64(3), 425-461.
Verifying the Location of the Standard
Major Standard Setting Methods