Poster: Using the CAL Validation Framework for Domain Analysis and Domain Modeling of an English Language Proficiency Test

-

Please consult the event program for date, time and location.

Presented at: LARC 2018

In building a validity argument for an assessment, the claims made about the construct being assessed must be backed by evidence to support them. For an ongoing operational assessment administered to students in grades Kindergarten(K)-12, as content standards and grade-level expectations change, these claims must be evaluated periodically. This poster addresses two critical aspects of test research and development: (1) the usefulness of a validation framework as an organizing road map in establishing claims and evidence to support those claims; and (2) the need in an operational assessment program to periodically gather up-to-date evidence about the target language use domain in order to maintain a validity argument regarding the claims being made about the assessment. In particular, this work draws on the CAL Validation Framework, which joins Evidence-Centered Design (ECD; Mislevy, Almond, and Lukas 2004; Mislevy, Steinberg, and Almond 2003) and the Assessment Use Argument (AUA; Bachman and Palmer 2010). Also influenced by Kane (2002), the CAL Validation Framework embeds the development of the validity argument within the test development process and the documentation of that process.

This poster focuses on domain analysis and domain modeling for the writing portion of the large-scale summative ACCESS for ELLs 2.0 assessment, which assesses the academic English language proficiency of K-12 students who have been identified as English learners. As part of continual refinement and refreshment of the assessment for the domain of writing, survey and focus group data were collected from educators from across states that use the test to gather evidence of target language use in grades 1-12 writing tasks. Survey data were collected on the prominence of a range of genres in current standards-based classroom instruction and assessment from 1st through 12th grade. Focus group data were gathered to provide up-to-date information on task topics and characteristics that are common in each grade from 1-12. These data were used to evaluate claims made about the assessment and to inform the development of new task types that address genres and topics which are currently prevalent in content standards which are taught in grade-level classrooms. Using the CAL Validation Framework as a roadmap, findings from the research will be shared, along with examples of their impact on the assessment.