What is Rigorous Test Development?

What is Rigorous Test Development?

RTD is an item-centric and validity-focused approach to standardized test development and therefore a principled approach to test development. It begins with the idea that tests are built of items (e.g.,, questions and opportunities to give answers) and that if those questions are not high quality, then nothing that is based upon them can be, either. You cannot make chicken salad out of chicken scat. As items are the foundation and building blocks of all assessments, ensuring that they provide strong information is the most important aspect of test development and a necessary precursor to any significant use of a test.

While the RTD project offers views and explanations for many parts of assessment development and the use of tests, it is primarily concerned with the production phase, where items are created and refined and tests are built. It picks up where Evidence Centered Design leaves off and respects much about psychometrics. More than anything else, it is about content development and the work of CDPs (content development professionals).

The RTD project is an attempt to build a professionalized content development practice that focuses on individual item quality, particularly by leaning into the importance of validity throughout the content development process. It assumes that content development professionals develop professional judgment that can be raised, honed and calibrated by providing frameworks and clarifying explanations in ways that account for the constraints and demands of typical practice within test development, today.

What is a Content Development Professional?

CDPs are the full-time assessment industry professionals who develop and refine test items, with focus on the contents of items and of tests. CDPs are not item writers—who contribute early drafts to the long and complex process of item development.

Historically, CDPs are among the lowest status workers in the assessment industry. While many come to this work with advanced degrees and relevant professional experience (e.g., teaching experience), there are no degree programs in content development and there are no professional certifications for this work. This stands in contrast to psychometrics and what is often called "measurement,” which generally picks up after items are complete and often just assumes that that items measure what they are supposed to measure.

Therefore, CDPs have few opportunities to learn the skills and knowledge they need to do their work at a high level. They either work with local versions of received wisdom and/or have to reinvent the wheel themselves. It is difficult for them to advance their knowledge or for a professionalized knowledge base of theory and practice to develop. As the workhorses of the industry, they often face enormous workloads with tight deadlines—all without time to reflect on their work or grow their understanding of best principles to guide their work.

What is the Rigorous Test Development Project?

The RTD project is an attempt to address this gap and these problems. It aims to provide a cohesive approach to thinking about content development work that is grounded in theory and offers an array of tools, practices and approaches for content development professionals. It is an effort to develop a professionalized practice that raises content development work to at least the status of psychometric work and thereby give CDPs the seat at the table that they deserve—as they are the ones who think about content and think about test takers in the course of their work. Therefore, they are the ones who are carry the deepest responsibility for validity and for fairness.

It is an ongoing effort, grounded in the real work of standardized assessment development—particularly content development work. While some of it has an academic flavor, it is meant for the practicing professionals who do this work first and foremost. It will continue to offer more tools, packets, explanations and guides to help dedicated CDPs to do their work at a higher level and to develop items that better elicit evidence of the targeted cognition for the range of typical test takers.