AI Writing Detection Assessment

Project Background

The emergence of large language models, starting with OpenAI's ChatGPT in November 2022, has introduced new challenges to academic integrity, while also amplifying existing issues. In response, faculty have had to reevaluate their teaching practices to address the following concerns:

• Undisclosed or unallowed use of AI for assignment completion
• Erosion of authorship, original thought, and student voice
• Difficulty in detection of AI-generated text
• Vulnerability of traditional assessments and assessment methods
• Ethical and policy gaps

Some educational technology vendors have expanded their product offerings in response to these concerns.

Prior Turnitin AI Tool Review

Turnitin released an AI checker in April 2023, providing a beta version as a free integration in Canvas. In December 2023, the free trial ended, and continuing the tool would have required additional funding. OIT surveyed faculty users and met with advisory groups to gather feedback and recommendations on the tool's usage and efficacy. The Academic Technology Advisory Group (ATAG) unanimously recommended not to support Turnitin's AI detection feature, and OIT did not upgrade the Turnitin license to include the tool. This decision was in alignment with many of CU's peer institutions who also decided not to adopt AI detection tools.

Among the reasons informing the decision not to adopt the Turnitin integration were concerns about its accuracy and bias. Turnitin has refuted an often-cited study from Stanford University researchers that GPT detectors are biased against non-native English writers. Additionally, one recent study supports Turnitin's claim that its AI writing detection yields a < 1% false positive rate. Given these developments, the rapid evolution of AI tools, and faculty interest, Academic Technology is re-examining Turnitin's AI Writing Indicator tool.

Process

The study will include the following:

Peer Institution Review

A review of AI detection practices at peer institutions to determine if any have adopted Turnitin's tool or a similar solution. For schools that have not adopted or endorsed AI content checkers, we want to know their reasons and any pedagogical practices that they are advocating to preserve academic integrity and help mitigate unethical AI use among their students.

Literature Review

A review of recent literature to determine if Turnitin's accuracy claims can be verified. The literature review will include research into how Turnitin and other AI detection tools have addressed issues of transparency, bias, and student privacy and security.

Faculty and Student Feedback

An analysis of current faculty practices and sentiment regarding the use of AI detection tools. We will also review recent survey results assessing student use and perceptions of AI tools.

Alternative Solutions

A light review of other AI detection tools, as well as process-oriented tools designed to enhance transparency throughout the writing process.

Consultation with the CTL

The Center for Teaching and Learning is developing guidance and resources on Teaching and Learning in the Age of AI. They also have a moderated Teaching, Learning, and AI Resource Repository for sharing implementable classroom resources. We will be consulting with them to gather their thoughts on AI detection tools and pedagogical guidance.

Results

At the conclusion of the study, we will provide findings and recommendations to faculty and student advisory and governance groups.

OIT Technical & Business Operating Principles

• User experience matters: Examine whether Turnitin's AI Writing Indicator supports accurate assessment practices and equitable learning outcomes.
• Security is foundational: Explore whether Turnitin's data handling practices align with institutional security measures and regulatory requirements.
• Strategically use governance: Present findings to leadership and advisory groups to gather feedback and inform decision making.
• Innovate where it matters: Explore how the tool and its alternatives have contributed to innovative assessment practices, AI-informed pedagogy, and AI literacy.
• Understand higher ed landscape: Evaluate whether the tool aligns with emerging norms in AI use in higher education.

Project Participants and Roles