Five Areas to Evaluate AI Risk
Data Privacy is often
the first concern, and rightly so. When a school adopts an AI tool, that tool
will touch student data. The question is not whether, but how much and where.
Questions for your vendor:
Then ask: is the vendor compliant with FERPA, COPPA, and your state's student privacy laws?
This matters because the attack surface grows with every tool added to your system. When PowerSchool suffered a breach in late 2024, more than 62 million student records were exposed. Each vendor you work with becomes a potential vulnerability.
Questions for your vendor:
- What student data does this tool collect?
- Does it need names and ages, or just performance on assignments?
- Does the tool store data on its own servers, or
does it process it locally on school devices?
Then ask: is the vendor compliant with FERPA, COPPA, and your state's student privacy laws?
This matters because the attack surface grows with every tool added to your system. When PowerSchool suffered a breach in late 2024, more than 62 million student records were exposed. Each vendor you work with becomes a potential vulnerability.
If a vendor says "we're FERPA compliant" but cannot tell you exactly what data they collect, where it is stored, and when it is deleted, that compliance claim is not worth much. Ask for the data inventory in writing. If they do not have one, they have not done the work.
Accuracy and Reliability is the second area, and it is where many educators first run into friction with AI tools.
Many AI systems, especially generative models, can produce content that sounds entirely plausible but is completely wrong. This behavior is called a "hallucination," and it is not a bug that will be patched in the next version, though the systems do get more accurate over time.
Questions for your vendor:
• Does this tool generate content, and if so, what is your process for catching errors?
• If a student gets an incorrect explanation from the system, who catches it?
• If the tool scores assignments, what is the accuracy rate on student work samples, and what happens when it makes a mistake?
A grading tool that is even slightly miscalibrated can affect transcripts and college applications.
Every AI tool will get something wrong. The question is whether anyone will notice. If the vendor cannot describe a specific process for catching errors, or if the answer is "the teacher reviews everything," ask yourself whether that is realistic given actual class sizes and workloads.
Equity and Access requires asking whether the tool works the same way for all students.
Some AI systems perform well on majority-group data. Research from Stanford found that AI detection tools produce false-positive rates above 61% for essays written by non-native English speakers. Students with disabilities may also find the interface inaccessible.
Questions for your vendor:
• Has the vendor tested this tool with non-English learners, students with documented disabilities, and students from different socioeconomic backgrounds?
• What were the results?
• Does the tool require specific devices, high-speed internet, or paid subscriptions?
The UK's experience with automated grading in 2020 is instructive. The algorithm systematically disadvantaged students from lower socioeconomic backgrounds, a problem that was not obvious until results were published and challenged.
If a vendor says "it works for all students" but cannot show you test results broken out by English learners, students with disabilities, and students from different socioeconomic backgrounds, they are making a promise they have not verified. A tool that has only been tested on one population has only been tested for one population.
Transparency means you can understand what the tool does.
If your school is using AI to flag students for intervention, make decisions about academic placement, or generate content that students see, you should be able to explain that system to parents and staff.
Questions for your vendor:
• Can you explain how your AI makes decisions or generates outputs in language a teacher could understand?
If the vendor's answer is "it's a black box" or "even we're not sure," that is a red flag. You are accountable for tools in your school, which means you need to understand them. Ultimately, ask yourself - will you feel comfortable explaining to parents how the tool does what it does?
Security is the final area, and it covers the basic safeguards around your data.
Questions for your vendor:
• How does the vendor protect data?
• Is it encrypted? Who has access to it?
• What is their incident response plan if a breach occurs?
These questions may seem technical, but they are not. They are about whether the vendor takes the problem seriously. A vendor who has thought about these questions and can answer them clearly is more trustworthy than one who has not.
According to the Center for Internet Security, 82% of K-12 schools experienced at least one cyber incident over a recent 18-month period. With such a high rate of attacks, leaders especially need to be mindful of the AI tools being allowed within the school.
How to Use This Framework
7140 Heritage Village Plaza,
Gainesville, Virginia 20155 USA
Gainesville, Virginia 20155 USA
Copyright © 2026