Researchers warn AI detection tools unfit for student assessment

A study by a James Cook University researcher has found that using artificial intelligence detection software in student assessment is fundamentally unreliable and risks undermining, rather than protecting, academic integrity.

Wayne Bradshaw, Research Services Officer at James Cook University, co-authored the study as part of a team led by Associate Professor Mark Bassett at Charles Sturt University. He said the rapid rise of generative AI had prompted many institutions to rely on detection tools to identify AI-generated assignments.

"The problem is that, unlike plagiarism software, which compares text against existing sources, AI detectors estimate the likelihood that text was generated by AI using linguistic markers that often differ between human and AI-generated writing," Dr Bradshaw said.

"The result is only a probabilistic estimate that cannot be independently verified."

He said this meant the technology failed to meet the evidentiary threshold required for academic misconduct investigations.

The research team also questioned attempts to strengthen detection results through linguistic analysis, the use of multiple detection programs or comparisons with a student's previous work.

"These methods introduce confirmation bias rather than providing independent evidence," Dr Bradshaw said.

"There is also the problem that written work cannot be neatly classified as either human- or AI-generated. Modern writing increasingly exists on a spectrum, with some students using AI as a support tool.

"The presence of linguistic markers commonly associated with AI-generated text does not indicate that the text was written by AI any more than a student paper containing linguistic patterns similar to Shakespeare's indicates that the student is Shakespeare."

The researchers believe instead of investing in increasingly sophisticated detection systems, universities should redesign assessment to reflect AI's growing impact on education.

"Unsupervised assessments can no longer be fully secured, and institutions should focus on ways of evaluating learning that recognise AI's potential use," Dr Bradshaw said.

The researchers concluded the current reliance on AI detection software does not strengthen academic integrity but instead weakens confidence in the fairness and purpose of higher education assessment.

JCU Deputy Vice-Chancellor Education Professor Mitch Parsell said the University approaches the assurance of learning through assessment design, not detection.

“We have chosen not to invest in AI detection tools. They are unreliable, and they cannot keep pace with new AI releases,” Professor Parsell said.

He said JCU’s focus is on what students do in clinical and industry placements, laboratory and fieldwork, capstone projects and portfolios and problem-solving while embedded in communities and workplaces.

Link to paper here.