Written by Angie Stansell and Marcie Kamb, September 29, 2021
How does a team that functions by conducting in-person workshops continue to develop high-quality assessments to meet client needs when you cannot host in-person meetings? It’s simple: you go back to the drawing board, establish a plan on conducting virtual workshops, and strive to meet the needs of the client while navigating uncharted waters.
The Center on Education and Training for Employment’s (CETE) Assessment program, working with the Ohio Department of Education (ODE) and the Ohio Department of Higher Education (ODHE), works together with subject-matter experts (SME) to develop course tests for technical-education programs throughout the state. The team relies heavily on the willing participation of teachers, post-secondary instructors, and even industry leaders to attend in-person workshops to create these test item banks, under the facilitation of CETE’s team members. In March 2020, the global pandemic and resulting state shutdown caused this process to come to a very sudden halt right in the middle of an item-writing year.
Determined to continue working to meet ODE’s needs, the Assessment program developed a process on conducting virtual workshops – what was needed for test security, how to make sure all items were completed with the high-quality standards that were already being achieved and meeting the delivery needs of ODE. Over the next few months, a plan was developed and then implemented. The first set of virtual workshops were conducted with supplemental reviews in May of 2020. This allowed the team to fix any issues and move on to finishing up the 2019-2020 Construction item-writing in the Fall of 2020.
The team continued to refine this procedure, making it more efficient and enjoyable. Construction was completed in December 2020, conducting 20 virtual workshops to complete the remaining 10 courses affected by the shutdown. Much of the success of this completion lies in the quick recruitment of SMEs, a task that was coordinated by Ruth Ann Falconer, the newest member of the Assessment team.
This chart illustrates the responses from subject matter experts in a workshop evaluation survey, showcasing that nearly 90% of SMEs preferred the virtual format of workshops over the in-person format. The survey represents data captured 2020-2021.
2021 would prove to be an even greater challenge for the team due to having to conduct the entire item-writing year for Health Science in a virtual format. This resulted in 3 workshops per course, with 32 courses in the career field, resulting in 96 virtual workshops for Health Science. There were other workshops needed as well to finalize Construction, provide supplemental reviews for Engineering & Manufacturing, and begin the creation of a performance assessment for Baking & Pastry Arts. In total, over 100 workshops were conducted to complete the Health Science item-writing; all these workshops were conducted between February and August of 2021. A total of 12 supplemental workshops were held from May 2020 through July 2021. For the test development process to be completed for 2020-2021, the CETE Assessment Services team held over 120 virtual workshops.
The virtual workshop format proved to be highly beneficial for the subject-matter experts. Nearly 90% of SMEs who completed our workshop evaluation survey stated that they preferred the virtual workshops, with most of them saying it was a more efficient and more effective use of their time.
Many SMEs stated that they enjoyed the fact that they did not have to travel and get a substitute for their classroom, which would often put them behind their desired curriculum pace. The benefits of the virtual workshops were easy to promote and increased the participation of the subject-matter experts. “Not only were the SMEs excited to write the new test items for their career field, but they were also willing to learn and experience a new workshop format. In my opinion, the “perks” of virtual workshops (not having to leave their homes, etc.) increased recruitment.” – Angie Stansell, Assessment Services Coordinator.
Due to this response, and the slow return to normality, the Assessment program decided to continue with virtual workshops moving forward. The team will hosts large in-person workshops at the beginning and end of 2021-2022. The item-writing workshops themselves will all be virtual and take place at the end of the school day, allowing the SMEs to participate without disrupting their classroom planning.
The Assessment program hopes to continue this workshops style moving forward, as it is their goal to not only meet the client’s needs but to meet the needs of subject-matter experts.