One of the earliest ideas in RTD is the tension between construct isolation and authenticity in assessments.

What is construct isolation?

Construct isolation is the practice of testing just one thing at a time. One skill. One piece of knowledge. The idea is that if an item tests more than one thing at a time, we do not know why the test taker got it wrong. If a test contains ten items, each of which tests two things, a test taker could get them all wrong, despite knowing half of the targeted content. More importantly, we simply cannot know whether a test taker who got the item wrong lacked all of the targeted skills or just one of the targeted skill.

Alternatively, if an item tests multiple KSAs (i.e., knowledge, skills and or abilities) by allowing each alone to provide a path to a successful response, the items fails to tell us which KSA — or even KSAs — the test taker has demonstrated proficiency with.

That is, without construct isolation, we invite countless false positive and false negative inferences about test takers’ proficiencies. Thus, the argument that items should produce high quality evidence of test taker proficiency with the targeted cognition quickly suggests the importance of construct isolation.

What is authenticity?

KSAs are rarely applied in isolation. They are part of some toolbox or store of knowledge, skills and abilities that people use in a variety of contexts — both practiced and novel — as needed. Truly skillful proficiency with a KSA usually entails recognizing when it is needed, combining it with other KSAs and actually using it.

Authentic use of KSAs is in combination with with other KSAs. And authenticity increases test taker engagements. Perhaps more importantly, if authentic use of KSAs is in combination with other KSAs, isolated use of KSAs does not match the true targeted cognition. Instead, it distorts and simplifies it. It renders it more predictable, therefore drillable. This, in turn, hurts both item and test validity — unless the goal is to measure how well drilled test takers are with isolated skills.

Furthermore, this creates challenges to facial validity, which further undermines item and test validity. (See Facial Validity.)

A Tension to be Managed

Unfortunately, there is no great resolution to this inevitable tension between construct isolation and authenticity. RTD’s fifth New Core Principle is Test development requires being mindful of and managing competing tensions – at times even issues that do not at first appear to be in tension. This is not the only tension that CDPs and other test developers must manage, but it was the original tension that challenged us, as content developers.

There are ways to address this tension. High quality distracts can collect information about how/why a test taker came to an unsuccessful response — though those are limited in number. Larger constructed response items can reveal more about test takers’ cognitive paths — though they are expensive to score and do not provide as much information once the cognitive path goes astray. Items can be grouped together so that collectively they build a more authentic experience — though the very order of their presentation is just one way in which they make the experience less than authentic. Of course, there are many others, as well.

Not Unique to Assessment

To be fair, this authenticity vs. construct isolation challenge exists throughout schools. When students ask, “But when will I ever use this?” they are asking about authenticity. When teachers are trying to balance the place of drills and worksheets with larger projects, they are working with construct isolation vs. authenticity. Scaffolded learning paths often begin with construct isolation or deviate into periods of construct isolation. That is often good teaching. And then teachers need to figure out how much authentic practice they should — or even can — bring to their classrooms. As with assessments, there are time constraints and coverage objectives, to say nothing of other constraints.

Hence, both educators and assessment professionals face the challenge of finding the right balance of construct isolation and authenticity.