Key Points

• This article argues that while there is great promise in the everyday automation of higher education to create benefits for students, efficiencies for instructors, and cost savings for institutions, it is important to critically consider how AI-based assessment will transform the role of teachers and the relationship between teachers and students.
• The focus of the work is on exploring whether and to what extent the requirements set forth in Article 22 of the General Data Protection Regulation (GDPR) apply within the context of AI-based automatic assessment systems, in particular the legal obligation to ensure that a teacher remains in the loop, for example being capable of overseeing and overriding decisions when necessary.
• Educational judgments involving automatic assessments frequently occur in a complicated decision-making environment that is framed by institutional processes which are multi-step, hierarchical, and bureaucratic. This complexity makes it challenging to determine whether the output of an AI-based automatic assessment system represents an ‘individual decision’ about a data subject within the meaning of Article 22.
• It is also unclear whether AI-based assessments involve decisions based ‘solely’ on automatic processing or whether teachers provide decisional support, excluding the application of Article 22. According to recent enforcement decisions, human oversight is entangled with institutional procedures and safeguards as well as system design.