Review panels are committees of external subject matter experts whom test sponsors/owners and test developers bring in to review items that go on tests. They generally either focus on the alignment/content validity of an item or fairness issues in the items. There are advantages to keeping these reviews separate and there are advantages to bringing these reviews together — as item validity highlights the intertwined nature of alignment and fairness.
Who serves on review panels?
The experts selected to serve on item review panels bring an array of expertise to the the table — even when the meetings occur virtually. This may include deep content expertise, knowledge of instructional approaches for the material, knowledge of the testing population (or perhaps specialized knowledge of some subgroup of the testing population).
With K-12 assessments, these subject matter experts are primarily classroom teachers. They generally possess a variety of expertise. However, even with K-12 assessment, there is room for others on review panels, as well. For other sorts of assessment (e.g., professional licensure exams), review panels are often dominated by experienced professionals in the field — a group who often also serves as item writers earlier in the process.
Fairness review panels may be drawn from the same sort of groups as content validity panels. They may also include professionals with particular knowledge of a a group and/or have special claims to representing them and/or their perspectives.
Not only do these sorts of panelists bring their useful expertise and experiences to this committee work, but they also serve an important signaling function to the public of the seriousness with which test developers take both content validity and fairness issues.
What should review panels focus on?
It really is not for review panels to fix items. They simply lack the assessment and item development expertise to balance all the concerns and delicacies that make item development work so challenging. Rather, they should focus on identifying problems and explaining exactly what their concerns are (see Rigorous Item Feedback). This is the best use of their time, the best use of the special knowledge, experience and expertise that they bring to this work. Furthermore, they should not worry about consensus or unanimity on an issue, if none is immediately available. Review panels are not democratic bodies for making decisions. Rather, they are a group of experts there to share a breadth of insight to inform professionals who engage in their own specialized work.
Moreover, by focusing on spotting problems and explaining why and how they undermine an item’s ability to elicit evidence of the targeted cognition for the range of typical test takers, they can help CDPs to better understand the content or — and more importantly — the perspective of some group among the range of typical test takers. Hence, this helps their feedback to inform future item development efforts, as well.
What do CDPs do on review panels?
Review panels are usually led/facilitated by full time content development professionals. They should not be trying to fix items during these meetings any more than panelists. Rather, they should focus their efforts on understanding panelists’ feedback, on facilitating them to think even more deeply about the perspectives of test takers as they might work through these items and on facilitating them to share their insight as clearly and completely as possible (i.e., as through supporting them to supply Rigorous Item Feedback).
They are generally called facilitators because their focus should be facilitating all this.
What happens after review panels are done?
CDPs must then review the feedback and suggestions they have received from the panels to consider what revisions and/or refinements are needed in each item. Simply due to scheduling, the responsible CDP may not always be the facilitator of all the panels who reviewed an item. Furthermore, panels at times offer contradictory or mutually incompatible advice. CDPs have to use their best professional knowledge, skills and judgment to make sense of, evaluate and appropriately act on the feedback offered by the panel(s).