Item Alignment Examination (IAE) is a structured procedure to rigorously examine items for alignment with their assessment targets/targeted standards. It requires knowledge of the content domain, knowledge of the structure of the construct definition (e.g., the set of standards) and knowledge of the thinking of the range of typical test takers. A full explanation of Item Alignment Examination can be found in the draft chapter Item Alignment Examination downloadable from the sidebar to the left. A short explanation can be found in the annotated packet. Other useful guides and documents also available in the sidebar.
Where does IAE come from?
Item alignment examination is based on the best practices of highly skilled and capable content development professionals (CDPs). These practices were brought together and organized into a framework that makes clear and explicit the various steps of checking an item for alignment. It does not exactly match what any expert CDP actually does, because their expertise allows them to work without the deliberately separated steps that IAE presents.
When should IAE be used?
IAE can be used any time someone reviews a draft of an item, at any stage. It can be used by item writers when reviewing their own work, by CDPs when first reviewing items, by committee members when reviewing items, and even by anyone reviewing an item after changes have been applied. IAE is certainly useful for selected response items, and it is also useful for constructed response items.
Must the entire IAE procedure be used every time?
IAE offers a framework for rigorously working through the process of examining an item against its purportedly aligned standards. It is the most rigorous way to ensure that items are properly aligned. However, practicing professionals may internalize IAE in different ways, perhaps conflating steps or taking steps out of order. The more expert and proficient the CDP, the more they may customize their own approach to this task, and the fewer external signs they may present that they are following it. Less experienced and/or proficient CDPs are the most likely to benefit from following IAE more diligently.
Review committee members may be offered an abbreviated version of IAE that better fits the skills and knowledge they bring to their work reviewing items.
What are the 8 steps of Item Alignment Examination?
Step 0: Do Not Prejudge. Put out of your mind your expectations about the item and what it is supposed to assess.
Step I: Do The Item Yourself. Simply do the item yourself, getting through it as your current self would do the item.
Step II: Metacognition/Re-creation. Think back on the path your just took and make yourself more conscious of every step and decision you make along the way to your final response.
Step III: Examine Your Atypicalness. Review the path you took through the item to understand why you made the decisions you did and why you took the steps you did to recognize any that might not be typical of test takers.
Step IV: Radical Empathy. Work through the item to a response through the perspective of one sort of typical test taker. Do this again and again for different sorts of successful and unsuccessful test takers. Note the steps and decisions that each makes along the way.
Step V: Compare Tasks. Review the different successful and unsuccessful paths through the item to identify the Key KSAs that differentiate successful paths from unsuccessful paths.
Step VI: Double Check the Item. Go back to the item to consider whether it suggests other successful or unsuccessful paths that you omitted in Step IV. If needed, revisit Step V.
Step VII. Consider Alignments. Consider the Key KSAs you identified in the context of the entire structure of the content definition to determine which element(s) (e.g., standard(s)), if any, those Key KSAs are a part of.
Step VIII. Confirm Alignment Accuracy. Check whether the actual alignment(s) identified in Step VII matches the purported alignment of the item.
How many paths should Step IV uncover?
More than you think. More than you have time for. Ideally, all the paths. New Core Principle #5 recognizes that the demands of ensuring alignment and item validity are often in tension with available resources (e.g., time). So, barring time to explore all of the cognitive paths, instead aim for all of the most common paths through an item.
What if IAE reveals that an item is not aligned?
IAE can reveal that an item is aligned to the wrong standard, that the item is actually aligned to a standard in a different level or grade, that an item is aligned to multiple standards or even that an item is not aligned to any element in the formal structure of the construct definition.
These are generally all bad things. And they usually require pulling the item or reworking it to some degree. (Items that are aligned to multiple standards fail to provide high quality positive and/or negative evidence of a test takers’ proficiency with targeted cognition because their success or failure with the item cannot be linked to proficiency (or lack thereof) with a particular standard. In practice, this may be less of a problem when both standards are in the same reporting category, but it still poses challenges to ensuring compliance with test blueprint design.)
IAE does not determine how to fix an item?
No, it does not. It should uncover problems with an item that impact its alignment and thereby should help to focus attention on what might be fixed. But the dedicated and interconnected nature of test items mean that no simple procedure can provide all the answers. Rather, item refinement decisions depend upon the professional judgment born of experience and reflection.