Content & Cognition Traits — Rigorous Test Development

Content & Cognition Traits of an Item

Download

Content & Cognition Traits Packet

Content and Cognition Traits of High Quality Valid Items (NERA 2023 paper)

Valid items elicit evidence of the targeted cognition for the range of typical test takers. This list of traits of an item get at the appropriateness of the content in an item and that an item requires of test taker, and gets at the nature of of the cognitive paths that an item prompts in test takers.

These are the most substantive traits of any item.

The traits below — both the general traits and the multiple-choice-specific traits — can each be expanded for a brief explanation. Download the Content & Cognition Traits Packet (available from the sidebar to the left) for fuller explanations of each trait.

General Content & Cognition Traits

Items should not support alternative cognitive paths to a successful response that do not depend appropriately on the targeted cognition.
Items should not have construct (i.e., the aligned standard) irrelevant barriers to producing a successful response.
Items should be free of content errors.
Items not meant to test some SPECIFIC prior knowledge – particularly declarative prior knowledge – should not support bypassing the targeted cognition by relying on that prior knowledge.
Items targeting the application of specific reasoning skills to a scenario or text should not support paths to a successful response that do not rely on the scenario or text.
Items should align to the specific grade-level version of a standard. Items aligned earlier grade levels are not aligned to the standard. (Diagnostic tests may require items aligned to a range of grade levels, and even there each item must align to its targeted standard and grade level.)
Items should not support cognitively less complex paths to a successful response than envisioned by the standard – unless that cognitive simplicity is strictly a function of high proficiency with the targeted standard.
Items meant to assess high proficiency (i.e., high automaticity) with the KSAs in a standard should not support more complex paths to a successful response than envisioned by the standard – unless that cognitive simplicity is strictly a function of low proficiency with the targeted standard.
While a single item need not necessarily assess the entire scope of a standard, it should focus on the meaningful core of the standard – rather than some tangential easier to assess portion of the standard.
Items should resemble the work students do in class and/or the contexts in which the targeted cognition is actually used elsewhere.
Polytomously scored items should allocate points based on proficiency(ies) with the targeted cognition, and not based on progress towards a solution.
Items written to standards that do not suggest rote or memorized skills should be sufficiently novel that test takers cannot be prepared with drilled solution paths.
Items should not be so novel that test takers must engage in notable learning in order to successfully respond to it.
Items should present all the KSAs that are necessary to respond successfully that are not a part of the targeted cognition and cannot be safely assumed to be known by the range of typical test takers.
Items should clearly state every element of the test taker’s work product and/or process that are required for a fully successful response.
Items should be sufficiently engaging to test takers that they authentically undertake the task prompted by the item.
Items should be in the item type and modality that provides the best opportunity for test takers to make use of the targeted cognition.
Items should make the nature of the cognitive task clear with a direct question or a direct instruction, rather requiring test takers to muddle through to figure out what they are charged with doing.

Multiple Choice Item Content & Cognition Traits

Items should not subtly or obviously signal which answer options are correct and which are incorrect without test takers applying the targeted cognition.
The key must be the definitively correct answer option, not merely arguably the correct answer option.
Distractors should be the results of the most common mistakes, misunderstandings and/or misapplications of the targeted cognition – and not merely be similar or close to the key. (They exist to provide an inviting option for test takers with this specific misunderstandings or who make those specific mistakes.)