AI Is Already in Schools—So the Real Question Is: How Do We Use It Without Hurting Kids?
Artificial intelligence (AI) is no longer a “future trend” in K-12 education. It’s already showing up in lesson planning tools, writing support, tutoring platforms, online proctoring, content filters, student monitoring systems, and district-level analytics. That fast adoption is exactly why the Pennsylvania Advisory Committee to the U.S. Commission on Civil Rights released its December 2024 report on the civil-rights impact of AI in K-12 education.
The report’s core message is simple: AI may help, but it can also scale harm. And when the harm shows up, it often lands hardest on students in federally protected classes—students of color, students with disabilities, LGBTQ+ students, English learners, and students from low-income communities.
For schools, the goal isn’t “AI everywhere.” The goal is thoughtful, measured use—especially when technology touches discipline, surveillance, grading, placement, or access to opportunity.
Two Big Buckets: Teaching AI vs. Using AI to Teach (and Judge) Students
One of the most helpful ideas in the report is the recommendation to separate two conversations:
- Teaching students about AI (how it works, how to evaluate it, how to use it responsibly)
- Using AI to teach, evaluate, monitor, or make decisions about students
The report suggests we should accelerate the first (AI literacy) because students will encounter AI in the world no matter what. But we should slow down the second because the risks—bias, privacy violations, and civil-rights impacts—can be serious and long-lasting.
Where AI Can Help (When Adults Stay in Charge)
The Committee heard testimony that AI can be useful when it supports educators rather than replacing them. Examples include:
- Teacher support: drafting emails, creating rubrics, generating lesson plan ideas, or producing differentiated materials
- Student learning support: summarizing text, brainstorming, outlining, translation support, and faster feedback loops
- Accessibility support: text-to-speech, speech-to-text, captioning, note support, and other assistive functions (with caution about accuracy)
- District operations: spotting broad patterns in attendance or performance to guide resource allocation (again, with caution)
But the report repeatedly emphasizes a key condition: humans must remain accountable decision-makers. AI output should be treated like a draft, a suggestion, or a signal—not a verdict.
The Civil-Rights Risks: Bias at Scale, With Less Visibility
AI can amplify bias faster than older technologies because it can generate content and make predictions at speed. The report describes AI as “opinions embedded in code,” shaped by training data, developer assumptions, and real-world deployment choices.
In education, that matters because biased systems can influence:
- Who gets placed into advanced courses vs. remedial tracks
- Who gets flagged as “high risk”
- Who gets accused of cheating
- What content students can access
- Who is surveilled, disciplined, or referred to law enforcement
Example: E-Proctoring and “Cheating” Flags
The report highlights repeated issues with AI-based proctoring tools, including:
- Facial recognition failures for students with darker skin tones and for trans/non-binary students
- False flags for normal behaviors (eye movement, reading aloud)
- Disproportionate impact on students with disabilities and neurodivergent students whose movement, gaze, or speech patterns differ from what the system expects
When these systems label a student as suspicious, the educational and emotional consequences can be significant—especially if there is no clear, fast human appeal process.
Example: Early Warning Systems and Predictive Analytics
Early warning systems can be used to identify students who may need support. But the report warns that these tools can also:
- Falsely flag Black and Hispanic students at higher rates
- Use demographic and socioeconomic data in ways that embed historic inequities
- Create “permanent” labels that follow students across systems
Even when the intent is supportive, a predictive label can change how adults treat a child—especially if teachers are given risk lists without training on what the system means and how to respond fairly.
Example: Content Filtering That Becomes a “Digital Book Ban”
The report cites concerns that filtering tools may disproportionately block content related to LGBTQ+ identities and communities of color. If this happens without transparency, monitoring, and correction, it can quietly restrict access to identity-affirming or culturally relevant information—while schools may not even realize it’s occurring.
Privacy and Surveillance: When “Safety Tech” Becomes Student Control
Because AI systems thrive on data, schools face growing pressure to collect more of it: academic performance, behavior, location, browsing history, biometrics, and more. The report raises major concerns about student monitoring tools and data-sharing practices, including the possibility of sensitive school data being used for policing.
One especially troubling theme: students who rely more on school-issued devices—often low-income students—may be monitored more heavily, increasing the risk of discipline and law enforcement contact compared to peers using personal devices.
From a civil-rights perspective, the report suggests districts should treat surveillance-related AI as a high-risk category requiring strict limits, transparency, and accountability.
The Digital Divide: AI Can Widen Gaps Unless Access and Training Are Real
Even if AI tools are “available,” that doesn’t mean access is equal. The report emphasizes that AI effectiveness depends on basics like devices, broadband, and adult support. Students in under-resourced districts may face:
- Less reliable internet access
- Less staff capacity to vet tools and monitor outputs
- Less high-quality data (which can worsen AI performance)
- Less at-home support to navigate tools safely and effectively
At the same time, the report acknowledges a hopeful point: AI literacy and early exposure to technology skills can open doors for students who have historically been excluded from high-opportunity pathways. That’s why the “teach AI, slow AI decision-making” split is so important.
Procurement Is Power: How Districts Can Force Better Vendor Behavior
A major insight from the report is that schools have leverage through purchasing. District procurement is enormous, and the Committee heard testimony that districts can use that leverage to require vendors to meet clear standards before they ever enter classrooms.
Practical guardrails districts can adopt include:
- AI-specific procurement benchmarks for safety, privacy, data security, and equity
- Third-party audits or certifications to verify vendor claims
- Contracts with real remedies if vendors fail to meet standards over time
- Shared repositories of vendor performance reviews so under-resourced districts aren’t left to “test” risky tools alone
- Regional purchasing consortia to improve bargaining power and standardize protections
Don’t Forget the Human Work: Relationships, Social-Emotional Learning, and Trust
The report repeatedly returns to something educators already know: school success is built on relationships. Especially for younger students, learning is deeply social—play, movement, peer interaction, and trust with adults.
Over-reliance on technology can reduce meaningful student-teacher connection and may contribute to isolation. The report suggests districts should be particularly cautious about heavy AI use in early grades and should evaluate whether a tool supports or disrupts the human core of education.
What This Means for Schools Using Online Services (Including Teletherapy)
At TinyEYE, we work with schools in a space that is both high-impact and high-responsibility: student support services. The report is a reminder that any digital tool touching student wellbeing should be designed and deployed with:
- Strong privacy protections and clear limits on data use
- Transparency so families and educators know what tools are used and why
- Human alternatives and fast escalation paths when technology fails or flags something incorrectly
- Equity checks so supports don’t unintentionally underserve students with disabilities, English learners, or marginalized groups
In other words: innovation is welcome, but student rights are non-negotiable.
A Simple “Next Steps” Checklist for District Leaders
- Decide what category each AI tool falls into: AI literacy, teacher support, student support, assessment/placement, or surveillance/discipline.
- Require vendors to explain data practices in plain language and sign strong privacy agreements.
- Train staff to spot bias, errors, and over-reliance—and to document and report issues.
- Create a feedback loop for families and students to raise concerns and get human review.
- Avoid high-risk uses (especially surveillance and automated discipline) unless there is compelling evidence, transparency, and oversight.
For more information, please follow this link.
