We Built a 5-Minute Watchdog So We'd Never Hear About a Crash From a Customer First

Somewhere in your business, something is running unattended right now — an invoicing script, a lead-capture form, a payment webhook, an automation that quietly keeps the lights on. If it dies at 3am, when do you find out? For most small businesses, the honest answer is: when a customer tells you. That's the moment your reliability problem becomes their trust problem.

The Blind Spot We Found in Our Own Stack

By mid-July 2026 we had four agents running our comms stack. They were stable — but stable isn't the same as watched. If one crashed overnight, we wouldn't know until a customer hit the broken path. So we built a fifth agent whose only job is to watch the other four and message us the moment one goes down.

We called it NARMER's herald duty — NARMER is our lead-notify agent, already wired into Telegram, email and Slack. Giving it a watchdog job meant every five minutes it checks whether its siblings are still alive.

How It Works

Every 300 seconds, the watchdog:

• Pulls the process list and checks each agent's last exit code
• Compares it against the previous check, stored locally
• Flags anything that went from running to dead, or from a clean exit to a failed one
• Sends an alert via Telegram (instant) and email (for the record)
• Cools down after one alert per agent per day, so a repeated failure doesn't spam us into ignoring it

Simple in concept. The three bugs we hit building it are the part worth stealing.

Bug One: A Restart Isn't a Crash

Our first alert fired within an hour of launch — "auth agent crashed." It hadn't. We'd just restarted it during a routine deploy. macOS sends SIGTERM on restart, which produces exit code 143. To a naive check, that's indistinguishable from a real failure.

Fix: a live process is healthy, full stop, no matter its last exit code. We only alert once the process is gone and its last exit was non-zero. One line of logic — if (job.PID !== undefined) return 'healthy' — killed every false alarm.

Bug Two: The Watchdog That Silently Stopped Writing

We stored the watchdog's state file in ~/Documents. No errors, no crashes — the file just quietly stopped updating. The cause: macOS's Transparency, Consent and Control privacy layer blocks scheduled jobs from writing to Documents, Desktop and Downloads, and there's no way to grant that permission programmatically. It fails dead silent.

Fix: move state out of the protected folders entirely — we use /var/tmp. If you're running any scheduled task on a Mac, assume Documents/Desktop/Downloads are off-limits before you find out the hard way.

Bug Three: A Placeholder Credential Can Disable Your Alarm

The watchdog authenticates to our vault to send alerts. During development we'd left a placeholder token in the config. We forgot to swap it before going live. The agent got rejected by the vault, failed to send the alert — and said nothing. Our safety net had a hole in it, and the only way we'd have found out was by not being warned when something actually broke.

Fix: validate every credential before trusting it, and if validation fails, fail loudly — don't degrade silently. We also added a pre-flight check that kills the job at deploy time if the token still contains the word "placeholder." A watchdog that can fail quiet is worse than no watchdog at all.

What This Means If You Run a Small Business

You don't need four agents and a vault to have this exact problem. Any SME running automated invoicing, a booking system, a payment integration, or even a simple scheduled backup has the same three failure modes hiding in it:

• A routine restart or update looking like a crash, or a real crash looking like nothing happened
• A permissions setting silently blocking the one file your system depends on
• An expired or placeholder credential quietly disabling the exact tool meant to warn you

None of these show up as an error message. They show up as a customer email asking why something didn't work — days after it broke. The watchdog has caught two real failures for us in the past month, both in our email agent, both fixed within minutes because we knew immediately instead of finding out from a customer.

That gap — between "it broke" and "we knew" — is the whole point. For a business your size, closing it doesn't need a big ops team. It needs one thing checking on the rest, on a short enough clock that a Saturday-night failure gets fixed before Monday's inbox tells you about it.

One Thing to Do Next

If you've got a system running unattended right now — booking, payments, invoicing, a lead form — and nothing is watching it, reach out at pharoahtechnology.co.uk and tell us what it is. We'll walk you through exactly how we'd build the watchdog for it.