Online Proctoring Needs Assessment

Summary

OIT conducted a comprehensive needs assessment to evaluate the implications of decommissioning the campuswide online proctoring solution, Proctorio. CU Boulder made a quick decision to adopt Proctorio because of the pandemic in 2020. Prior to the start of the 2023-24 academic year, campus leadership made the decision not to renew the Proctorio license, set to expire in August 2024, prompting the need for the assessment. Data analysis showed that online exam proctoring usage has declined since the height of the pandemic and that the usage is concentrated mainly in the Leeds School of Business and some Mathematics and Economics courses in Arts and Sciences:

• 80% of faculty respondents to the Academic Technology survey indicated that online proctoring is not a necessary tool for their courses.
• In fall 2023, 21% of students used Proctorio, down from 34% peak usage in spring 2021.
• 6% of instructors and 5% of courses used Proctorio in AY 23-24; 45% of the courses using Proctorio were online courses. 
• 73% of the courses that used Proctorio were in the Leeds School of Business, followed by 16% of courses in two subject areas in Arts and Sciences (ECON and MATH).

Additionally, online exam proctoring is not without controversy. After CU Boulder adopted Proctorio in 2020, students petitioned to stop its use. Some students expressed concerns about privacy and reported increased anxiety levels during test-taking. Centering the student experience is crucial when implementing educational technologies such as online exam proctoring. Balancing academic integrity with student well-being helps ensure a positive learning environment.

Reduced and limited usage across campus, increasing and unsustainable licensing costs, and adverse effects on students’ learning experience informed OIT’s decision not to provide a campuswide proctoring solution at this time. IT Governance endorsed the decision. However, because the Leeds School of Business has decided to continue using Proctorio for AY 24-25, Proctorio will remain OIT’s recommended proctoring technology for instructors or departments wishing to pursue their own licensing arrangements. 

Project Background

Questions concerning the equity, accessibility, and transparency of exam proctoring software have complicated its use. Following the campuswide adoption of Proctorio in 2020, the CEAS Remote Exam Working Group issued recommendations and best practices for remote exams. The group recommended against using Proctorio because of privacy concerns and the invasive nature of the technology.

Additionally in fall 2021, the Provost charged the Boulder Faculty Assembly (BFA) committee with exploring the continued necessity for a campuswide Proctorio license. The BFA convened a subcommittee to gather and analyze Proctorio usage and cost data. They also met with faculty to identify the service’s strengths and weaknesses. Based on their research, the Subcommittee concluded that there was “not enough evidence to make a strong recommendation regarding the campus-wide subscription to Proctorio.”  The Subcommittee suggested further study on the usage of Proctorio and its intersections with ethical, labor, and pedagogical concerns.

OIT initiated an RFP in 2019 to provide a centrally-supported, online proctoring solution that would streamline the campus proctoring experience. The technology that initially was chosen had issues with deployment at scale, requiring a quick pivot to Proctorio, partly because of Proctorio’s early adoption and wide use within the Leeds School of Business. Since spring 2020, CU Boulder has maintained a campuswide license for Proctorio. The service received funding through the CARES Act and then from the Office of the Provost.

Proctorio’s high cost and low campuswide usage (5% in AY 22-23) prompted academic leadership not to renew the Proctorio license when it expires in August 2024. To understand how this decision will impact teaching and learning and in order to identify performance gaps with the removal of the proctoring service, OIT undertook a comprehensive assessment of online proctoring needs for the campus.

Approach

OIT solicited input from current and past (2020-present) Proctorio users to understand what proctoring features instructors use or need and to determine how online proctoring contributes to teaching and learning experiences at CU Boulder. For data gathering, OIT

• Distributed a Qualtrics survey to over 900 current and past Proctorio faculty
• Gathered feedback through 1:1 interviews with faculty
• Facilitated group listening sessions with power users in Leeds and Mathematics

OIT also sought different perspectives and solutions through  

• Discussions with campus partners in Disability Services, the Digital Accessibility Office, and the Center for Teaching and Learning
• Reviews of the academic literature on online exam proctoring
• Comparative analyses of online exam proctoring at CU’s peer institutions in the Pac-12 and AAU
• Evaluation of other viable technologies

OIT’s Technical and Business Operating Principles guided the work for the needs assessment. OIT also presented its findings to advisory groups and IT Governance:

• ATAG – Academic Technology Advisory Group
• BFA ATSC –  Boulder Faculty Assembly Academic Technology and Services Committee
• SATC – Student & Academic Technology Committee

Analysis of the data and consultation with these groups allowed OIT to finalize its recommendation for the campus.     

Findings and Conclusions

Data analysis revealed a limited need in online exam proctoring at CU. 6% of instructors and 5% of courses used Proctorio in the 2023-24 academic year. 

In addition, 80% of faculty respondents to the Academic Technology Survey indicated that online exam proctoring is not necessary for their courses. The majority of courses that use Proctorio are within the Leeds School of Business (73%) and two subject areas in Arts and Sciences (16% across ECON and MATH). It follows that Proctorio usage among students is concentrated mainly within these colleges and departments: 

Even though the data showed a steady decline in the use of exam proctoring on campus, OIT wanted to explore other viable technologies that would help service the limited proctoring needs of the instructors and courses that still rely on Proctorio. Thus OIT collaborated with Leeds to evaluate other proctoring solutions. The evaluations considered factors including:

• Cost and scalability: what are the vendor’s pricing model and fees for limited and enterprise licensing
• Platforms: what operating systems and devices are compatible with the technology
• Integrations: does the technology integrate with CU’s learning management system and does it allow for integration with third-party exams
• Browser lockdown: does the technology lock down browser applications, and can the feature be enabled and disabled in exam settings
• Accessibility: does the technology meet accessibility requirements
• Support: does the vendor provide live support and 24-hour assistance

One of the solutions featured innovations not seen in the other technologies: mobile-device detection and exam-content scrubbing from internet sites. However, the pricing model was not sustainable at scale. OIT determined that a different solution that was evaluated would provide the best balance of cost and flexibility for the wider campus but that adopting the technology would not meet the needs of some groups that had high Proctorio usage. This product’s main limitation was the inability to disable its browser lockdown feature in exam settings. Partly for this reason, Leeds decided to continue with Proctorio for another academic year.

OIT recognized that pivoting to a new technology or recommending a second solution would require some students to navigate two distinct proctoring technologies. Leeds shared that its business minor program includes approximately 2000 students. Having more than one proctoring solution on campus, such as Proctorio for Leeds and a different technology for other departments on campus, would require these students and a small number of others to use different proctoring platforms for their courses. To provide a consistent proctoring experience for students, OIT decided that Proctorio will remain the recommended provider for online exam proctoring.  

OIT presented these findings, along with survey and interview feedback, to advisory groups and IT Governance. A majority of ATAG members, representing a diversity of colleges and departments, including leading Proctorio users from Economics and Leeds, advised against a campuswide proctoring solution. The BFA ATSC committee expressed no major concerns about this approach. SATC endorsed the decision not to provide a campuswide proctoring solution.  

Recommendation

To prioritize the user experience, minimize redundancy and allocate financial resources responsibly, OIT does not currently recommend supporting a centrally-funded proctoring solution. Instructors who wish to use a proctoring solution are advised to consult with their departments for individual or departmental arrangements. If in the future the university sees a shift in its enrollment model toward more online course offerings or other relevant changes in university strategy or faculty practices, OIT will reevaluate online exam proctoring at CU. Given the potential for rapid advances in AI technology, more cost-effective options could become available in the future.

Project Team