Design Brief 5: Protecting Privacy, Security & Safety

ByNext Path Design

In Season 1, Episode 1 of the AI for Educators Design Lab podcast, we looked at how AI is exposing vulnerabilities in our assignments. In Episode 2, we stepped back and examined whether our learning goals still hold up when AI can handle so many of the tasks those goals were originally built around. In Episode 3, we dug into AI literacy, and I argued that students need more than just tool proficiency. They need the judgment to evaluate AI output critically, and we need to deliberately design for that.

And in Episode 4, we focused on the human interactions that make learning possible, arguing that in an AI-integrated environment, learning is shaped not just by how students interact with technology, but by the people who design, guide, and participate in those interactions.

We now turn to a topic with both legal and ethical implications. The question we consider is where responsibility for data privacy, security, and safety lives in AI‑integrated learning.

🎙️ Podcast Focus

In Episode 5, the argument is that data privacy, security, and safety in AI-integrated learning aren’t just IT or compliance concerns, or problems tech vendors should monitor and control. They’re a core design responsibility for educators, learning experience designers, and educational leaders that shapes everyday tool choices, workflows, and prompt decisions.

This episode was recorded amid reports of a major Canvas/Instructure security incident that may have exposed data for up to 275 million students, faculty, and staff. It’s a stark reminder of what’s at stake when private information escapes the systems we trust to protect it.

But while large-scale breaches grab headlines, the more common everyday risk is far quieter. It could be a well-intentioned teacher pasting student names, grades, IEP content, or full assignments into tools like ChatGPT, Claude, or Gemini, often without clear guidance about where that data goes, how long it’s retained, or whether it’s used to train the model.

Unfortunately, we’re dealing with a troubling convergence of three forces:

  1. AI adoption is expanding rapidly,
  2. Institutional policy is lagging behind, and
  3. Educator training, where it exists, often doesn’t address privacy, data governance, or compliance specifically.

Educators sense the risk as seen in recent surveys that show the majority are concerned about AI privacy and security. However, students, parents, and teachers report feeling largely “in the dark” about how AI tools and monitoring technologies work.

🤔 Learning Experience Design Challenge

This episode reframes data privacy not as a policy problem to solve later, but as a design decision already being made. Privacy decisions in AI-integrated learning occur at the moment of tool selection, assignment design, and prompt construction, well before any institutional policy applies. Educators are the most frequent data handlers, which means much of the design responsibility lives with us, not only with IT or with vendors. The challenge itself is evolving.

As AI capabilities expand and tools change, our practices need to keep evolving with them. Awareness and protections need to be built into daily teaching practice and into the design of learning experiences. The five design considerations in this episode walk through that shift along an arc that begins with you and works outward to your tools, to how you use AI day to day, to how you teach students about it, and finally to how you communicate with the families and communities in which your students belong.

Building off these design considerations, the AI Data Privacy & Security Design Audit below is offered as a diagnostic tool to help you work through those design questions in a real-world learning experience of your own.

A note before you begin: This episode and Design Brief aren’t intended to provide legal advice on data privacy and safety requirements. The Future of Privacy Forum at https://fpf.org/ is a helpful starting point for practical resources and regulatory guidance, but you’ll need to do your own research into the specific laws and policies that apply in your context.

📋 Your Design Decisions

This episode posed five design considerations in the form of questions to help you examine how data privacy, security, and safety show up in your AI-integrated learning experience. This isn’t an exhaustive list of design considerations, but it will get you started thinking about several core issues related to this challenge as they relate to your learners, your discipline, and your context.

🤔 Design Considerations

  1. Educator Grounding: To what extent do you have a privacy-aware workflow when using AI to support learning, and where might there be gaps between how you currently work and a workflow that better protects your students’ data and your own? Privacy and safety decisions aren’t only made in district contracts or institutional policies. They’re also made at the moment of tool choice and prompt design, which means they live in your daily practice. A privacy-aware workflow includes habits like enabling multi-factor authentication on your AI accounts, using pseudonyms or placeholders like “Student A” instead of real names when prompting about student work, signing up for tools with an institutional email or alias, enabling no-training or temporary chat modes when available, and treating new AI features (like an AI button that suddenly appears in your LMS) as new tools that warrant a privacy check before use. And while much of the privacy conversation focuses on student data, your own data, your account history, your professional communications, and your notes about students is also at stake.
  2. AI Tool Selection: How are you deciding which AI tools cross the threshold into your classroom, and what trade-offs are you accepting when you choose between free, paid personal, enterprise, and institution-approved versions? Data protections aren’t binary. They exist on a spectrum, and where a tool falls on that spectrum often depends on the specific version or plan you’re using. Free consumer accounts typically offer the least protection, with broader data collection, longer retention, and inputs that may be used for model training. Enterprise and education-specific licenses are more likely to include Data Processing Agreements that limit training on student data and establish rules around retention and deletion. Two additional wrinkles worth knowing about: institutional approval is a trade-off, not a free win. It protects students from vendor misuse, but it also creates institutional visibility into your practice. And watch out for hidden third parties. This might include aggregator tools that route prompts to multiple underlying models, and impostor apps or browser extensions that mimic well-known platforms but have no official connection to them.
  3. Data Minimization During Use: At the moment you actually use AI, what does data minimization and privacy-aware prompting look like in your context, especially for the students who would be harmed most if something went wrong? Educator AI use spans a spectrum from light brainstorming at one end to uploading a student’s full assignment at the other, and the disclosure risk grows along that spectrum. Beyond identification risk, student work is also intellectual property. Uploading it raises questions about authorship and dignity, not just privacy. And the students with the most to lose if something goes wrong are often the ones with the least power to consent or opt out. Privacy in this context isn’t only about keeping data secret. It’s about power, consent, vulnerability, and the right to learn without being unnecessarily tracked, profiled, or judged by opaque systems.
  4. Teaching Privacy Literacy: How are you building privacy literacy into the learning experience as a skill students actively practice, not just a rule they follow? Privacy is most often communicated to students as a one-course rule (“don’t share personal information with AI”) that may not protect them when they’re using AI on their own. What’s more durable is treating privacy as a practiced skill students carry forward into the rest of their lives. Major AI literacy frameworks, including UNESCO’s recent AI competency guidance for students, position privacy and data protection as core parts of responsible AI use. Practical curriculum moves include teaching prompt hygiene (removing personal, school, family, health, or location details before submitting to AI), running data trail mapping activities, comparing two AI tools’ privacy policies in age-appropriate language, practicing with synthetic or fictional data, and modeling your own privacy decisions when you use AI in front of students.
  5. Transparency and Consent: If AI use is required for your course, or if you use it without students’ knowledge, what does meaningful transparency, equitable consent, and a privacy-safer alternative look like, including for minors and the families advocating for them? AI use that isn’t transparent to students, such as surveillance technology, behind-the-scenes grading, or analytics they don’t know about, eliminates their ability to even know what they’re consenting to. AI use that’s required eliminates the consumer-privacy mechanism of “just don’t use it.” Students who can’t refuse without an academic penalty have no real choice, and in those cases, transparency and equitable alternatives become the only real protection. For minors, the family is also a stakeholder; communicating data practices to parents and guardians is part of meaningful consent under FERPA and COPPA, not a courtesy. The artifact this design consideration produces can be like a boundary statement: clear syllabus language, a real opt-out alternative where possible, or a parent-facing disclosure where appropriate.

📋 AI Data Privacy & Security Design Audit

In this episode, we examined how data privacy, security, and safety in AI-integrated learning are not only IT and compliance concerns but design responsibilities that shape everyday tool choices, workflows, and prompt decisions. The AI Data Privacy & Security Design Audit worksheet helps you audit your current AI workflow against these five design considerations, identify gaps, and begin drafting a course-level boundary statement you can actually use with your learners and their families.

  • 👉 Access the AI Data Privacy & Security Design Audit Worksheet
  • 🖥️ Save a copy of this document to complete on your own.
    • If working in Google Docs: Go to File → Make a copy
    • If working in another program: Go to File → Download and choose your preferred format (e.g., Word or PDF)
  • 🧑‍💻 Select a learning experience where AI is currently in use, or one where you’re planning to introduce it, and work through each of the design considerations in the context of that experience.
  • ⏰ Plan for about 15–20 minutes to complete the audit.

🎙️ Looking Ahead

In Episode 6 of the AI for Educators Design Lab Podcast, we stay inside the equity conversation but look at a different dimension. We turn to what happens when AI integration assumes access that not all of your students have. Uneven access to devices, connectivity, or paid AI tools can quietly widen inequities, and we’ll think about how to design around these gaps. What assumptions about access are baked into your current AI use, and who might they leave behind?

Thank you for being here. I look forward to engaging with you in this conversation!

Jennifer Maddrell, PhD
Next Path Design

See other Design Briefs

Podcast Sources

    AI insights for educators.
    Delivered to your inbox.

    Stay connected with Next Path Design by subscribing to
    our free newsletter & email updates.

      Unsubscribe anytime. See our Privacy Policy and Terms of Use.​

      Built with Kit