What Automating This Blog Taught Me About Failure Modes

TL;DR — Over the past weeks I replaced “remember to write and publish” on this site with a scheduled pipeline: a daily automation that plans topics, verifies facts, writes, generates art, and publishes over SSH. It mostly works, and the ways it didn’t work taught me more about operational engineering than the parts that did. A scheduler that only exists while one machine is awake, a cache that insists on yesterday’s page, a design template quietly reused five times — every one of those bit us exactly once, and each produced a rule that now shapes the system. This is the honest field notes version, including the outages I caused myself.

The blog you’re reading runs on WordPress on a shared host, published through a path that evolved the hard way: browser REST when cookies allowed, then — after the plugin pipeline proved unreliable on long PHP jobs — a direct, scripted SSH workflow that uploads assets, inserts posts at the database level, and purges the page cache. Wrapped around it is a scheduled task that fires each morning, checks which posts are due, verifies topics haven’t been covered, and fills the calendar. The eight-part EFB series went out through the manual version; everything since runs through the automated one.

I want to write down what building and operating it taught me, because the failure modes turned out to be the standard ones — the same shapes SREs name in production systems — just at blog scale. Small systems are good for this: the feedback loop is hours, not quarters, and nobody pages you.

The Morning the Scheduler Slept In

The pipeline fires at 05:30 local time. On September 15 it didn’t — not because anything inside it failed, but because the machine it lives on was off. A scheduled task is not a commitment the universe honors; it’s a wish registered with a process that must be alive at the exact minute. That morning produced a silent gap: no daily post, no error, no log line. The monitoring that would have caught it was the same machine, equally asleep.

The fix wasn’t heroic. The run now begins with a backfill check: query the site’s REST API for the last three days of posts, compare against the slots that should exist, and write anything missing immediately — backdated to its proper slot. In effect, the pipeline stopped assuming it runs on time and started assuming it runs eventually. That’s the real lesson: schedules are a rendering hint, not a source of truth — the calendar state must be derivable from scratch on every run. It’s the same reason well-built deployment systems reconcile desired state instead of trusting that yesterday’s step ran.

The Failure Modes, Catalogued

None of these are novel. All of them are the standard content of the SRE literature on automation and toil — reconcile state, verify effects, design for retry — which is itself the point: the failure taxonomy of a hobby pipeline and of production infrastructure is the same taxonomy. Only the blast radius differs.

Backups Before Mutations, Always

The first time the pipeline edited already-published content — applying fact corrections to live posts — it did the correct nervous thing: wrote a copy of the current content to a backup file on the server before touching the database, then applied the edit, then verified. That habit came from watching too many “simple find-and-replace over 100 rows” stories end in restore-from-memory. Now every content-touching script follows the same shape: snapshot, mutate, verify, report. The backups have never been needed in anger, which is exactly how backups should work; the one time a regeneration produced misaligned artwork, the fix was a fresh render, not a restore — but the safety net changes how boldly you iterate, and that, quietly, changes the quality of what ships.

Verify Artifacts, Not Intentions

It is astonishing how much of reliable publishing reduces to refusing to trust intermediate success. A post insert that returns success has still, in my history, left a post without its featured image; an upload that “completed” has still served the previous file from cache. So the pipeline’s last step is always the same liturgy: fetch the live URL and require 200; query the REST API and require the correct date and metadata; fetch the image URL and require the new byte size; for generated art, render it and look at it — an image-analysis pass that catches what pixel probes and code review both miss. This is deliberately the same discipline as detection engineering: don’t validate your query syntax, validate that the alert fires. The system asserting “I did the thing” is worth little; the world showing the thing done is worth everything.

The visual check earned its place immediately: a first render of new banner art had a text element offset onto its container’s border. Every programmatic probe passed — the colors were right, the shapes were right — because the geometry I was probing wasn’t the geometry that was wrong. Only an actual look at the rendered image caught it. Tools that check what you remembered to check will never catch what you didn’t.

Idempotency Is Cheaper Than Caution

Re-running a script that appends a figure to a post gives you two figures; re-running one that regenerates thumbnails for an image gives you correct thumbnails twice. Early versions of my publishing scripts were the first kind, and caution was the only defense: check, re-check, run once. The mature version makes operations naturally idempotent — “prepend figure only if the URL isn’t already present,” “regenerate metadata unconditionally, since regeneration is safe.” It’s the database-migration mindset applied to content: cron will eventually run your job twice or not at all, and the system should shrug at both.

What I’d Tell Past Me

• State lives in the world, not in your run history. Derive what should exist from the site itself, every time — the site can’t remember intentions, but it also can’t lie about what’s there.
• Automate the verification before the action. The first version of any pipeline verifies by looking. Make looking a script.
• Templates are a debt. The fastest way to produce three identical banners is to have one excellent template. Vary structure deliberately or drift into uniformity — this is true of design systems and of alerting rules alike.
• Calendars and caches are the two liars. Anything time-based will miss; anything cached will show you the past. Plan for both from the first day, like we plan around clocks that disagree.
• Write the postmortem while it stings. The Sep 15 gap became the backfill check the same day. A blameless postmortem culture at team scale is just a personal habit of writing down what happened before you edit the memory.

Key Takeaways

• Scheduled tasks are only as alive as the host that runs them — design every pipeline to reconcile desired state and backfill gaps on each run, rather than trusting the schedule.
• Back up before every mutation of live content; the backup changes your iteration courage as much as your safety.
• Verify artifacts in the world (HTTP 200s, REST state, rendered pixels), never the success of the action that produced them.
• Idempotent mutations let you rerun freely; guarded, side-effect-free operations beat careful humans.
• Caches and calendars are the two systems that will lie to you politely — purge deliberately and verify with cache-busters.
• The failure taxonomy of a one-person pipeline matches production infrastructure exactly; small systems are the cheapest place to internalize it.

FAQ

What does the pipeline actually do?
Each morning it checks which content slots are due, searches the site to avoid repeats, verifies facts via web research, writes the post, generates a unique banner image, and publishes through an SSH/database path — then verifies everything live.

What was the worst failure?
The silent one: a day with no posts because the machine running the scheduler was simply off. No error, no alert — which is why the system now derives state from the site instead of its own history.

Why publish over SSH instead of the WordPress API?
Application passwords weren’t available for this site and long-running plugin jobs were being killed by the host, so a scripted upload-plus-database-insert path proved the most reliable option. Boring and direct wins.

How do you avoid the automation writing junk?
Hard gates, not vibes: minimum word counts, fact verification before writing, every reference URL checked live, dedupe searches against the site, and structural requirements (tables, takeaways, FAQ) enforced per post.

Isn’t auto-generated art going to repeat itself?
That literally happened — three identical banners. Now each post gets a newly written design function, and the run checks recent designs first; repetition is a bug, not a brand.

Would you automate editorial judgment too?
No. The pipeline automates mechanics and verification; topic selection, framing and what’s worth saying remain human decisions — the same line detection engineers draw between generating alerts and responding to them.

References

1. Google SRE Workbook — Eliminating Toil
2. Google SRE Book — Monitoring Distributed Systems
3. Google SRE Workbook — Postmortem Culture
4. crontab(5) — the scheduler’s honest manual page
5. systemd — timers and the alternative scheduler design
6. crontab.guru — cron expression reference

Field notes from operating hmmnm.com’s publishing pipeline, September 2026. Names of failures withheld to protect the operator, who was me.

Published by Hmmnm

Hands-on cybersecurity tutorials, CVE breakdowns, and guided learning paths — written and lab-tested by the Hmmnm team.

Learning Paths · About Hmmnm · Editorial policy