Standard Setting with SetScore!

Standard Setting Made Simple

SetScore! is a new, highly-interactive cut-score study tool designed to guide organizations through the standard setting process. More than a tool to calculate SME ratings, SetScore! provides an assisted flow through standard setting studies, acknowledging that collecting ratings is only part of the process. SetScore! addresses all ancillary work in conducting a study, benefitting test sponsors, facilitators / psychometricians, and SMEs.

Strasz Assessment Systems used a holistic approach to design SetScore! to readily coordinate SME recruitment, standard setting training, rating sessions, and post-session reporting. The system provides an intuitive framework for the entire process, including a kick-off call, guided practice rating session, rating sessions with real-time rating feedback to facilitators, rating-based recommendations for setting the cut-score, and the generation of supporting documentation. Use SetScore! to facilitate online or in-person meetings with SMEs and replace paper exams, rating forms, and surveys with an intuitive user experience.

An invaluable tool for standard setting teams, SetScore! can coordinate concurrent studies. Facilitators view past and current studies through intuitive dashboards.

Assistance from an App

The test sponsor and facilitators will appreciate the user-friendly process of importing participating SMEs’ data using APIs from the test sponsor’s system or spreadsheets in CSV or Excel format. The test sponsor’s staff or the study facilitator is promoted by SetScore! to add the exam form for rating and any accompanying reference documentation such as training materials.

Import items directly from Strasz Assessment Systems’ Pro! item development and banking application or another item store of your choosing – SetScore! consumes CSV or XML files. Item presentation is central to the rating process and SetScore! presents items based on the format of the imported items, including coupling items with their exhibits and matching font requirements. Item statistics can be imported and optionally viewed by raters, depending on the study type the facilitator has chosen.

Facilitators benefit from the virtual assistant features that help plan kick-off meetings and sessions in one central hub. Determine session types (Angoff and Modified Angoff at product initiation), add documentation for raters, and set the pace of rating sessions using timer functionality.

SMEs have training documentation and helpful definitions, such as the minimally competent candidate definition, on screen while performing the rating exercise. After SMEs affirm they understand the training modules, they proceed to the scheduled rating sessions. SetScore! displays the progress of SMEs through their session with a visual progress bar and a heads-up timer to help with pacing. SMEs can add comments about the items as they rate them. Upon completion of the sessions, the SME is presented with a survey / exit poll.

Efficient, In-person or Remote

Notably, no built-in downtimes are required to facilitate calculations outside of sessions. Statistics for the ratings are compiled and calculated in real-time as raters work! Facilitators are notified if ratings on items have a skew and need additional discussion sessions or direct SME attention using in-system communication features.

Documentation generated by the system supports accreditation or re-accreditation applications, including but not limited to SME participant lists and NDAs, SME training adequacy reports, SME comment forms, minimally competent candidate definitions, exam form metadata such as blueprint classification, current statistics from candidate administrations, and more. All reports are accessible to the moderator and the test owner for download in multiple formats.

Jumpstart your standard setting with SetScore!

and engage your teams