A term taken from ECD, this refers to the knowledge, skills and abilities that a test takers makes use of when reading, making sense of and responding to an item other than the specific cognition targeted by the item.

Cognition, cognitive processes and cognitive steps that a test taker engages in instead of the intended task, upon reading an item. An alternative task might or might not rely on the targeted tognition, and might not rely on it appropriately. (An RTD term.)

The quality of fluid and less conscious application of knowledge, skills and/ability. Higher proficiency often leader to greater automaticity and lower proficiency often leads to more a more deliberate approach, instead.

What an item asks a test taker to do to do to produce a work product – be it selection of a response that answers a question or the construction of a response. The more common use of the meaning of the term task. (An RTD term.)

Short for Content Validity Checklist, an old RTD procedure that scaffolds a review of an item to the goal of considering the cognitive path that test takers might take upon reading the item. This procedure is appropriate for review panels, whereas serious CDP refinement of items relies on deeper and more sustained item review. (An RTD term.)

A quality of a standard and/or an item that is district from item difficulty or the content of the standard. There are many different typologies of — or at least approaches to understanding — cognitive complexity. Norman Webb’s Depth of Knowledge is the most commonly used typology in the field of assessment.

The series of cognitive steps that a test taker takes upon reading an item as they work towards their final responses. They may be more or be less less conscious of various steps. (An RTD term.)

The first phase of five phases in the life of a test. This phase including initial recognition of a test, business plans, high level and the highest leverage decisions about the nature of the assessment. It also includes the hiring of some key personnel. While tests might return to the commissioning phase, this is very unlikely. (An RTD term.)

The rigorous practice of maintaining awareness of the degree of one’s expertise and the nature and limited degree of expertise of others. Confidence is critical to collaboration, to supporting the learning of others and one’s organization and to producing high quality work products. Confidence can be seen in those who know when to speak and to answer others’ questions. Half of one of the Pillar Practices of Rigorous Test Development work. (An RTD term.)

The final phase of five phases in the life of a test. This phase includes various things that are outside the control of test developers because they all occur as downstream consequences of an assessment. This includes the uses and purposes to which a test is put, and the following consequences, as well. (An RTD term.)

An explanation of what a test or test item is intend to asses. It may vary in scope (e.g., the entire domain or a single standard). It may also vary in explicitness and detail. Individual standards are a type of construct definition, and even they demonstrate how construct definitions can vary in score and in detail.

Those responsible for the contents of assessment items. CDPs own the process and work of improving items’ alignment with their standards and their item validity. CDPs must have knowledge of the content area, of assessment and item development, and of the range of takers of the assessment in question. Also known as an item specialist, a content specialist and other organization-specific terms. (An RTD term.)

When an item includes or is based on a misunderstanding or misapplication of some element of the content domain. The mistake need not been with the intended standard or the targeted cognition. It could take the form of a miskeyed item, or could result in deeper problems with the item.

The second phase of five phases in the life of a test. This includes domain and construct definition, blueprint design, score report design, task model development and most of the issues addressed in Evidence Centered Design.. It stops short of actual item development. (An RTD term.)

The rigorous practice of maintaining awareness of the limits of one’s expertise, the nature and degree of expertise of others, and a healthy uncertainty about conclusions. Humility is critical to collaboration, to learning from others and to producing high quality work products. Humility can be seen in those who know when to listen and to ask questions. Half of one of the Pillar Practices of Rigorous Test Development work. (An RTD term.)

The method by which content development professionals intend or anticipate test takers will respond to an item. The intended task incorporates how test takers are expected to understand the item and the broad outline of what they expected to do. (An RTD term.)

The building block of an assessment. It is the smallest complete unit that a test taker responds to. For example, a multiple choice item include the question, the directions and the answer options. Though in the most common usage, CDPs do not include any shared stimulus when referring to an item, strictly speaking an item includes any stimulus — even if it is shared.

A full framework for the involved process of reviewing an item for the multiple cognitive paths that the range of test takers might take when responding to an item. This process in a rigorous approach to determining whether an item is properly aligned( and for whom. (An RTD term.)

Traits, qualities of elements of an item that may undermine alignment of item validity that can be flagged without considering test takers’ cognitive paths and/or without deep knowledge of the content area.

The basic design of an item, the reasoning explaining how the features, elements and requirements of that design elicit evidence of the targeted cognition. (An RTD term.)

The rigorous practice of editing items to improve their item validity. Item refinement usually follows item review, though sometimes it done simultaneously with item review. Item refinement often benefits from reflecting on and reconciling what is learned in item review, especially when item review is done by multiple individuals. One of the Pillar Practices of Rigorous Test Development work. (An RTD term.)

The rigorous practice of examining an item to determine how test takers might respond to it and which — if any — elements of the item might lead them astray from the item’s intend task. The goals of item review are generally to determine the degree of alignment between the item and its purported standard and/or what elements of the item are undermining that alignment. One of the Pillar Practices of Rigorous Test Development work. (An RTD term.)

Valid items elicit evidence of the targeted cognition for the range of typical test takers. This is similar to the broader idea of item alignment, but it adds consideration of variation among test takers (i.e., fairness) as an equally important consideration as content concerns. (An RTD term.)

The knowledge, skills and/or abilities that successful test takers use when responding to an item and unsuccessful test takers do not use. (An RTD term.)

The complex interplay of the various elements of an item – down the level of individual words – that create a delicate context and creates meaning for test takers and communicates to them what is expected of them. A metaphor based on the intricacies of a mechanical watch. (An RTD term.)

The act of thinking about thinking. Metacognition includes planning later work and reviewing the thinking behind earlier work. It also incudes monitoring one’s thinking and work for mistakes.

The fourth of five phases in the life of a test. This phase follows all the work of developing a test and includes delivery, administration, scoring and reporting on test results.

The third of the five phases in the life of a test. This phase includes on the development of items, field testing, and form construction, range finding and perhaps standard setting. It is called production because it is focused on the development of a high number of highly refined and developed good — like automobile production. (An RTD term.)

The process of examining the incorrect answer options of a multiple choice item to identify the cognitive paths that might lead to their selection, and thereby the misunderstanding and/or misapplication of the targeted cognition or other cognition. QDR is vital to identifying what inferences might be appropriately made based upon unsuccessful responses from test takers.

The rigorous practice of thinking through a cognitive path in response to an item through the perspective of someone other than oneself, repeatedly. One of the Pillar Practices of Rigorous Test Development work. (An RTD term.)

Revised Depth of Knowledge. This is a typology for cognitive complexity that is strongly grounded in Norman Webb’s original Depth of Knowledge construct. The primary difference between rDOK and wDOK is rDOK’s explicit recognition that the cognitive complexity of a task can vary from test taker to test taker. (An RTD term.)

A rigorous protocol for recording and/or communicating content and/or cognition issues with an item. This protocol includes identifying i) who (i.e., the subgroup of test takers that might have the problem), ii) where (i.e., the point or element that prompts the problem), iii) how (i.e., how the test test taker’s cognitive path will be disrupted) and iv) which (i.e., which knowledge, skills and/or abilities are implicated). (An RTD term.)

One element of a domain model or definition. The unit to which items are generally meant to align — though standards often include a broader range of KSAs (knowledge, skills and/or abilities) or applications than a single item can asses. Standards are often written in a short/pithy form that requires unpacking to identify the included KSAs and applications.

The KSA (knowledge, skills and/or abilities) that an item is intended to elicit evidence of. This is often a subset or facet of a standard, though it may be an entire standard or even cross multiple standards. (An RTD term.)

In RTD, the cognitive work or path that a test taker engages in when responding to an item. Differs from the charge. (An RTD term.) Elsewhere in assessment, the charge and the task are both referred to as the task.

A fantastical — though commonly accepted idea. RTD acknowledges that there is no one typical test taker or typical type of test taker. Rather, any given assessment will typically be taken by a range of test takers — who will vary by identity, backgrounds, experiences, abilities, preparation and current state (e.g., sleepiness, hunger, mood, stress level). The fact that a test taker may me closer to the median in any number of these dimensions does not make them any more typical than kinds of test takers who are further from those medians but nonetheless still take the assessment regularly.

The rigorous practice of thinking about a standard or assessment target to identify the KSAs (knowledge, skills and/or abilities) and applications that a standard may include. This may be recorded in a Task Model or elsewhere, and it may be done on the fly by CDPs. One of the Pillar Practices of Rigorous Test Development work. (An RTD term.)