Safe by Design: What Building AI for Children Actually Requires

## The bolt-on problem

Most "kid-safe AI" is an adult chatbot with a swear filter and a cheerful mascot glued on top. Same model, same training data, same tendency to hallucinate or mishandle something serious — just with a word-blocklist out front, checking IDs by squinting.

That fails exactly where it matters. A blocklist catches "kill yourself." It does not catch a child describing, in their own words and without a single trigger phrase, that someone at home is hurting them. It catches profanity but waves through a checkout that lets an eight-year-old spend £40 on gems because nobody asked whether they were allowed to. The filter is busy policing vocabulary while the actual risks — a disclosure, a purchase, a stranger — walk straight past it.

We build the other way round. Safety is the first commit, not the last, and it decides what the product is even allowed to become. So you know where we're speaking from, the honest status in one line: Pharoah Kids World is live, with payments switched off until the parental gate is built and verified; Aegis, our age-assurance and guardrail engine, and our Kid-Safe Companion both run as internal prototypes, not certified compliance products. That's it — said once. Everything below is the architecture we hold ourselves to, some of it shipped and some of it still being wired, and the thinking matters more than our changelog.

## Start with the law, not the feature list

The UK's Age-Appropriate Design Code (AADC), enforced by the ICO under the Data Protection Act 2018, sets out 15 standards for any online service likely to be accessed by children. It is not advisory. It requires data protection impact assessments, high-privacy defaults, and — the standard most products quietly skip — designing for the age of your user instead of assuming adult defaults hold until someone complains.

COPPA does something parallel in the US for under-13s: verifiable parental consent before you collect a child's personal data. There are narrow exceptions, but consent is the spine of it.

The teeth are in that phrase "likely to be accessed by children." It is deliberately broad, and it does not care what you intended to build. A general-purpose app with no cartoon anywhere near it can still fall inside the AADC because children turned up. "We're not a children's product" is not a defence; it's a finding of fact waiting to be made about you. Which is why treating this as a design constraint from the start, rather than a category you can opt out of, is the only position that survives contact with a regulator.

Treat both as your product spec, not your legal team's homework. If a feature cannot satisfy AADC and COPPA at the same time, it does not ship in that shape for that market. Applied at the start, that constraint changes what you build. Applied at the end, it is a recall.

## Age assurance is infrastructure, not a gate

Age assurance cannot be a single yes/no at signup. It has to persist for the whole session. A 9-year-old and a 16-year-old should not share chat permissions, spending limits, or companion behaviour, and that distinction cannot evaporate the moment onboarding ends. A gate you pass once is a gate you have already defeated; assurance that travels with the session is a property of the system, not a doormat at the entrance.

That is the idea behind Aegis: a policy console, not a one-off checkpoint — somewhere the 15 AADC standards live as enforced rules you can set and audit, rather than legal text a developer re-translates into tickets every time the regulation is revisited. The unglamorous claim underneath it: compliance you cannot query is compliance you are guessing at.

## Privacy by default, not privacy by opt-out

Our rule for anything a child touches is deliberately more expensive to build than the alternative: **no analytics, no third-party trackers, self-hosted assets.** No ad-network SDK quietly fingerprinting a seven-year-old's device. No "anonymised" behavioural exhaust feeding a model somewhere else.

This is data minimisation taken literally — collect what the product needs to function, and nothing a marketing team might find useful later. It is slower; growth loops are harder to build when you refuse the instrumentation everyone else takes for free. That is the trade, and for this audience it is the right one.

Purchases are the sharpest edge. In Pharoah Kids World, payments stay off until one rule is enforced in the flow, not promised in a footer: any purchase must sit behind a real parental action, never a checkbox that says "I am over 18." Loot boxes and impulse spend are a child-safety problem before they are a monetisation-ethics debate, and the gate is a hard design requirement precisely because the commercial pull to weaken it is strong, constant, and always dressed up as "reducing friction."

## The guardrail has to know when to escalate — and when to stop talking

This is the part filters cannot do, and it is the whole game.

A kid-safe companion will eventually be told something serious. A child will mention self-harm, or describe abuse, in language that matches no keyword list — because children don't talk like compliance documents. They don't say "I am experiencing domestic abuse." They say their tummy hurts on the days they have to go to a particular house. The signal is oblique, rare, and enormously consequential — the exact inverse of what a blocklist is good at. A blocklist is a classifier tuned for precision on known-bad strings; a disclosure is a high-stakes, low-frequency event with no signature. You cannot blocklist your way to catching it.

So the design problem was never "block the rude word." It is two problems most products never name. First: recognise a signal that never uses the obvious phrase. Second — and this is the one people miss — know that the correct response is not to answer.

The most dangerous thing a model can do with a child's disclosure of abuse is handle it well enough to feel like help. A fluent, empathetic, generated reply to "someone at home hurts me" is worse than silence, because it keeps the moment inside a chatbot instead of routing it to a person who can act on it. Our companion prototype is built around escalation, not deflection: a signal consistent with self-harm or abuse triggers a defined path that signposts the child to Childline and does not try to resolve the disclosure with more text. The model's job in that moment is to hand off, fast, and get out of the way.

There is a trap here that a naive design walks straight into: alerting the parent. The default assumption in most parental-controls thinking is that the grown-up on the account is the safe adult. For a disclosure that implicates the home, that assumption can be the whole danger — routing "someone at home hurts me" to a notification on that same someone's phone is not a safeguard, it's a tip-off. So the parent alert cannot be the universal answer. A signal that points outward — bullying, a stranger, something at school — can reasonably surface to a guardian; a signal that points at the home has to route to a child-facing route out, not back to the person it might be about. Getting that distinction wrong is worse than having no escalation at all, and it is exactly the kind of judgement a keyword filter cannot make.

Detecting the oblique cases reliably is hard, and we treat it as an open problem rather than a solved one — the honest state of the art here is intent and behaviour classification, not keyword matching, and it is imperfect. So the design leans on one deliberate asymmetry: in this single domain, a false alarm is cheap and a missed disclosure is catastrophic, and the two errors are not remotely equal. That tilts every threshold toward over-escalating — toward handing an ambiguous case to a human rather than letting the model talk itself into resolving it. Most of the product is tuned to reduce false positives; this one path is tuned to tolerate them. But the architecture assumes the moment arrives and routes it to a human process built to help. A filter can suppress a word. It cannot run an escalation protocol, because nobody wrote one — and "the model declined to respond" is not a safety feature. It is an absence dressed as one.

## What building this way actually costs

It is slower and more expensive than building an adult product and sanding down the corners. You ship fewer features a quarter. You say no to growth mechanics that work brilliantly on adults and are indefensible on children. Your infrastructure bill rises the moment you self-host instead of piping data through three ad-tech vendors for free.

The alternative is a product that works right up until it doesn't — until a regulator, a journalist, or a parent finds the gap between what you claimed and what you actually architected. In most industries that gap is a PR problem. With children, it is a harm. That is the entire reason to move the cost forward, where it buys you an architecture, instead of paying it later as a recall.

## The takeaway

If you are building anything a child might touch:

- The AADC's 15 standards are your opening brief, not a compliance afterthought. - Age assurance has to persist across the session, not expire at signup. - Privacy defaults have to be high before the first user arrives. - Your AI needs a designed escalation path for the worst disclosure it will ever receive — because it will receive one — and that path has to include the discipline to stop talking and hand off.

None of this is a filter you can buy and bolt on. It is a set of decisions you make at commit one and keep making. If you're building in this space and want to compare notes on how the age-assurance model or the escalation pipeline is designed, get in touch — we'd rather trade architecture and hard trade-offs than pitch you a guarantee.