Process Packets
Downloads
Targeted Cognition and the Intended Task
Stems
Distractors
Refinement
Expertise in item development is not simply a matter of memorizing procedures and terminology. It certainly does not stop with putting out questions that merely look like standardized test items. Instead, it is about producing items that actually elicit evidence of the targeted cognition for the range of typical test takers. Expert professional practice — complete with expert and refined professional judgement – requires understanding deeply the how to think about the different stages in developing an item.
Unpacking the standards or assessment targets to which an item is written.
Conceiving a tasked that depends appropriately upon the targeted cognition (i.e, sometimes just a particular facet of a standard).
Writing a directions and/or a stem that will prompt test takers to engage in that task.
Developing distractors by which to record lack of proficiency (i.e, negative evidence).
The iterative process of revising and refining an item until it is acceptable.
While we present these five thinking processes in a linear order, in practice they can overlap and content development professionals (CDPs) can find themselves moving back and forth and around through them in an iterative fashion. When CDPs work building on the item drafts of others (e.g., item writers), the importance of this stage of this thinking process depends on the strengths and weaknesses of the item draft. Certainly, the last stage often calls on revisting earlier stages.
Each of the process packets downloadable from the sidebar to left lay out and explain the principles and thinking processes that go into stage of of this work.
Unpacking Standards
The first stage of thinking in developing an item is making sure that one understands the knowledge, skills and/or abilities (KSAs) that a standard gets at. This requires unpacking the concise language of a standard — in light of the distinct other standards in a domain model — to find the cognition that is worth building an item towards. This work must be done by item writers and by content development professionals (CDPs). It can be aided by recording the results of this process in a task model, but not all projects use task models and not all task model formats include details on the targeted cognition. (Download the Unpacking Standards white paper from the sidebar to the left for more information on this thinking process .)
Targeted Cognition and the Intended Task
The second stage of thinking in developing an item is designing some intended task that depends appropriately on the targeted cognition. This means that the task cannot be successful completed with appropriate proficiency with the targeted cognition and that there are no other KSAs will act as barriers that prevent test takers with the appropriate proficiencies from successfully completing the task. Only by avoiding those barriers and alternative paths to a successful response can a CDP avoid the item producing false positive and/or false negative evidence for a significant number of test takers. (Download the Targeted Cognition and the Intended Task white paper from the sidebar to the left for more information on this thinking process.)
Stems: Prompting the Intended Task
The third stage of thinking in developing an item is about communicating the intended task to a test taking audience. That is, the CDP must develop directions and/or a stem that makes clear what the item expects of the test taker. This is particularly challenging with items for large scale assessments because CDPs cannot depend on well worn understanding as exists between those who have experience communicating with each other (e.g., classroom teachers and students). So much communication between people depends upon a speaker recognizing when someone does not understand them and offering clarifications or restatements until they do understand. When that is not available, communication is far more difficult and clarity that much more important.
Distractors: Gathering Negative Evidence
The fourth stage of thinking in developing an item is crafting the specific alternative answer options to test takers that multiple choice (and many other technology enhanced items) offer. Item quality almost invariably relies on the quality of these distractors, despite that fact that few understand how difficult it is to write a set of high quality distractors. They must have facial plausibility so that test takers cannot simply jump to the correct answer without going through the cognitive work that the item intends for them. They also must each be deeply plausible by offering by landing spots for the most common misunderstandings and/or misapplications of the targeted cognition. They must do all of this without any of them standing out as obviously the correct or incorrect answer option. Crafting a high quality set of distractors builds on understanding the targeted cognition, the cognitive path(s) the stem can prompt and the kinds of misunderstandings that test takers might bring to the item to meet all of these requirements.
Refinement: The Interative Process of Revising an Item
Item refinement is the last stage of thinking when developing and item, and it usually requires CDPs to step in an our of previous stages. Item refinement requires CDPs to simultaneously hold in their minds the entire item and the specific aspect or component they are alternating. It requires thinking about the various paths that test takers may take through an item to a response and considering how potential alterations might impact those different paths. It requires paired holistic understanding of how the item functionings with understanding of how various details of the item contribute to that functioning. It is unlike most any other form of editing, as items can be as delicate as the fine machinery of mechanical watch and as subtle and full of meaning as a tightly crafted poem. This interactive process of refinement can begin with gross reworking of the architecture of an item and move through finer and finer refinement until the CDP believes that the item can elicit evidence of the targeted cognition or the range of typical test takers. Not only is this process interative, but is also highly interactive, with the CPDs understanding of the content standards/assessment targets, understanding of the perspectives of test takers, understanding of how items work, and their own thinking and problem-solving approach all interacting through the entire process.