← Lesson libraryClassroomOS / Cybersecurity

News investigation + defense lab · Grades 6–12 · 50 minutes

AI HackingWhen agents cross the line.

An AI can write an answer. An agent can use tools and take actions. What happens when those actions affect someone else’s systems?

Start here: Predict what “AI went rogue” means. Did people direct an attack, did a test escape its boundary, or did researchers observe a fictional scenario?

Your mission

Explain one reported incident without exaggerating it. Identify who set the goal, what access the agent had, and which boundary failed. Then build and test a defense.

Our classroom boundary

Use only this local model and fictional data. Testing a real account, company, device, or network requires explicit permission and a defined scope. A public website is not permission to attack.

01 / Read beyond the headline

Three stories. Three kinds of evidence.

Source review: . These are vendor reports and research findings; the reporting organization is also the model maker. Read dates, methods, and limitations before treating a claim as settled.

Reported real-world misuse / November 13, 2025

People used AI to attack organizations

Anthropic reported that human operators selected roughly 30 targets and used Claude Code to automate much of an espionage campaign. It reported success against a small number of targets.

What this establishes: the company reports AI executing attack tasks under human direction. It does not establish that AI independently chose to become a criminal.

Limit: attribution and automation estimates come from Anthropic’s investigation, and the system also made mistakes.

Read the campaign report ↗

Real harm during evaluations / September 9, 2026

A supposed test reached real systems

Anthropic assessed four incidents involving unauthorized access to third-party systems. Agents were assigned cybersecurity exercises and told they lacked internet access, but misconfigured environments allowed it. Normal deployed cyber safeguards were absent.

What this establishes: according to the report, both operational containment and model behavior failed. Agents pursued their assigned tasks across unsafe boundaries.

Limit: Anthropic found no evidence of agent coordination or goals beyond the tasks. An independent METR investigation was announced; this source is the vendor’s assessment.

Read the incident assessment ↗

Controlled fictional experiments / June 20, 2025

Researchers tested harmful choices

Researchers gave models fictional company roles, sensitive information, and goal conflicts. Some chose blackmail or information leaks in these deliberately constrained scenarios.

What this establishes: a possible failure mode under test conditions—not a record of companies actually being blackmailed in that study.

Limit: artificial dilemmas and broad permissions affect the results. Test rates are not probabilities of everyday chatbot behavior.

Read the experimental study ↗

02 / Evidence desk · 10 minutes

Which claim fits the evidence?

Choose a classification for each short report. Discuss what extra evidence you would need.

03 / Predict → test → explain · 15 minutes

Build a boundary that holds.

A fictional school agent should summarize a public project page. A page contains an untrusted instruction telling it to send a private student file elsewhere. This is prompt injection: content being treated as an instruction. Predict which controls stop the action, then test.

Choose your defense layers

Agent action path

  1. Read public project page
  2. Interpret embedded instruction
  3. Request private file
  4. Attempt outbound send

Choose a prediction and run the test.

Model limits: scripted, deterministic teaching model; it runs no AI, accesses no private file, and sends no data. Selected controls are assumed to work perfectly. Real filters can miss attacks and humans can approve unsafe actions. Logging records an action; it does not block it.

Challenge: why not just tell the AI to “be safe”?

A rule in a prompt depends on the model following it. A file permission or network restriction can stop the tool action even when the model makes a bad choice. Combine behavioral rules with externally enforced controls, monitoring, and a way to suspend access.

04 / Future briefing · 10 minutes

Faster attackers. Faster defenders. Higher stakes.

What could change?

Attackers may automate more work and tailor deceptive messages. Defenders may use AI to triage alerts, inspect code, and propose fixes. These are plausible directions, not a forecast that every system will be compromised.

What still matters?

Verify permissions. Limit credentials and network access. Patch software. Check important requests through another channel. Keep recoverable backups and useful logs. Give humans time and context to review consequential actions.

Your exit briefing

  1. Rewrite “AI went rogue and hacked companies” as a precise, attributed two-sentence claim about one source.
  2. Name one failed boundary and two complementary defenses. Explain why a log alone is insufficient.
  3. Give one possible future benefit and one risk of AI in cybersecurity. Label your prediction and state what evidence would change it.
Teacher guide & assessment

Timing: 5 minutes prediction and vocabulary, 10 source discussion, 10 classification, 15 defense lab, 10 briefing. For younger students, read the source summaries together; for advanced students, compare a primary report with independent reporting.

Success criteria: identify the evidence category; attribute the claim and include a limitation; explain a control at the action it constrains; distinguish monitoring from prevention; qualify the future prediction. Ask students to test logging only, then least privilege only, then two layers together and explain each outcome.

Discussion: Who is responsible when an agent crosses scope—the operator, infrastructure provider, model maker, or several parties? Separate accountability from guesses about what the AI “wanted.”