Standard setting is the process by which expert raters align test performances and scores with proficiency levels; it is a fundamental part of creating a new test. While much attention is given to the framing question(s) used for a standard setting study, an equally important question is how rater judgments will be collected. This poster presents two tools that were successfully used to collect judgments in two online standard-setting meetings for the BEST Literacy 2.0 reading and writing test and the BEST Plus 3.0 oral proficiency test for adult English learners. The researchers used a “proficiency ruler” tool to help judges visualize test-takers’ proficiency levels and assign corresponding ratings that could be easily used for psychometric analysis. This proficiency ruler aligned with mutually agreed upon proficiency level descriptors established in a pre-study meeting. The researchers also used Microsoft SharePoint to create a shared online workspace for participants. They created web-hosted Excel spreadsheets for judges to enter their judgments and linked these spreadsheets to a compilation sheet that was accessible to staff only.

This poster will show the advantages of these approaches for online standard-setting, including the convenience of data collection, the ability to restrict the visibility of judges’ responses, and the ways in which it simplified the process for the judges. The poster will also present considerations for future materials development, such as improved formatting of the “proficiency ruler” tool and the possibility of a digital version of the ruler. The materials presented will be useful to other researchers who are developing a new test or aligning a test with a new proficiency framework, and who wish to plan a remote standard setting study.