Stakeholder-Driven Contextual Evaluation of Language Models in Education

With the increasing reliance of AIED on opaque, black-box scaffolds such as large language models to support student learning, there is a growing concern about their limitations when used in diverse pedagogical contexts. To address these challenges, we developed AIBAT (AI Behavior Analysis Tool), a workflow and system designed to support stakeholders in auditing and critically evaluating the potential benefits and harms of AI systems within their specific pedagogical contexts (e.g., subject matter, grade level, English proficiency). In this half-day tutorial, participants will use AIBAT to identify and make sense of AI-related risks and use evidence to calibrate their trust in AI scaffolds. At the end of the tutorial, we will deliberate on AI auditing processes and discuss broader implications for promoting responsible and effective stakeholder participation in the evaluation and deployment of AI systems in educational settings.