The Checklist is a the oldest artifact of the RTD Project, its earliest forms even predating the term “rigorous test development.” It was our original attempt to improve large scale assessments through improve the processes that contribute to the development of test items.
What’s with the Name?
In the assessment industry, there are two major type of review committees for items. They are generally termed Content Review and Fairness Review. From the beginning, we have been concern with the validity of items. That is, their ability to support inferences about the test taker proficiency with what they were purported to assess. This even before we formalized the term item validity as their ability to elicit evidence of the targeted cognition for the range of typical test takers. It was already clear that so-called content committees had to do more than simply verify the accuracy of items and their components; they had to review items through the perspectives of test takers, and do so in a rigorous fashion. This meant much more than simply reviewing the content of the items. Hence, we referred to them as content validity reviews and committees.
While RTD is built on their and ideas, it aims to practical and usable. Whenever possible, we try to clarify ideas and procedures to single-page cheat sheets, often in the form checklists — even when they included in packets that explain the thinking and theory behind them. Checklist are strong scaffolding for novices to learn a process, to redirect experience practitioner into new practice and even to help support like-minded professionals as they engage in the work.
At some point along the way, we started to refer to the Content Validity Checklist simply as The Checklist. It was an inevitable shorthand for a document that we referred to frequently.
Who Should Use The Checklist?
The RTD Content Validity Checklist was first intended for panelists on content (validity) review committees. In short order, it expanded to useful for content development professionals (CDPs), as well. The packet downloadable from the sidebar to he left inlaces a version for each group.
Not Really a Checklist
While The Checklist is structured as a checklist, it is actually directs a structured and scaffolded review of an item that culminates in a question that is much more important than anything else in it, What are the different full thought processes that test takers might go through to get from the Stimulus and Stem to the Key? Yes, the checklist does walk walk the reviewer through checking different parts of the item for issues, and those steps can be viewed as a kind of structure for finding issues in each part. But the real purpose of each of those steps is to ensure that the reviewers has paid attention closely to the entire item, both holistically and though close examination of each part of the item.
Thus, then reviewers are asked that most important question, they have already given thought to the different pieces and will not not jump to conclusions based on their own quick reading of the item.
The Checklist for CDPs changed quite a bit once we developed Item Alignment Examination (IAE). We embedded IAE into this procedure, as it is a more rigorous process for understanding an item.
Are the Other Questions Just Window Dressing?
Once items get to committee review, the other issues should have already been caught. But if committee members do spot issues when addressing other steps in The Checklist, they certainly should raise them. Mistakes do happen, and it is important to catch them before field testing.
CDPs should always keep their eyes open for various kinds of issues, but they do not all need to be fixed immediately. There may be more pressing issues or other goals for their particular review of the item. Certainly, some kinds of problems are downstream from others, such that fixing those other problems inevitably result in cascading changes. Items to do not have to be full refined and polished until the end.