Automation That Survives Monday Morning

The automation demo always works. It runs once, on clean input, while someone is watching. Then it goes into production and encounters a duplicate email, a rate limit, a supplier who sends a PDF instead of a CSV, and a bank holiday — and it fails silently, which is the worst way for it to fail.

Three properties separate an automation that lasts from one that does not. None of them are about which tool you used.

1. Idempotency#

Running it twice must not do the thing twice.

Every automation gets rerun. A retry fires, someone clicks the manual trigger, a webhook is delivered twice because that is what webhooks do. If your workflow creates an invoice, running it twice creates two.

The fix is a deduplication key derived from the input, checked before the side effect:

idempotent.pypython
def process(message: Message) -> Result:
    key = sha256(f"{message.sender}:{message.subject}:{message.date}".encode()).hexdigest()

    existing = db.processed.find_one({"key": key})
    if existing:
        return Result(status="already_processed", id=existing["result_id"])

    result = do_the_work(message)
    db.processed.insert_one({"key": key, "result_id": result.id})
    return result

Write the key before the side effect where the store supports it, or use the external system's own idempotency key. The gap between doing the work and recording it is where duplicates live.

2. Explicit failure#

The default failure mode of most visual automation tools is a red execution nobody looks at. That is worse than crashing loudly, because the business assumes the process ran.

Three rules:

• Every failure notifies a person, not a log.
• Every failure message contains enough to act on: which record, which step, what error.
• Anything that cannot be processed goes to a queue a human can work through, not to /dev/null.

3. Observability#

You need to answer, without opening the tool: how many ran today, how many failed, how long since the last success, and what is stuck.

A single table gets you most of the way:

Where the tool choice actually matters#

It matters much less than the above. The orchestration layer — moving messages between systems — is genuinely faster to build and change visually. The business logic — validation, decisions, calculations — belongs in code with tests, whatever tool is calling it.

Get that split right and the tool becomes a replaceable detail. Get it wrong and your business rules live in a canvas nobody can review, test, or migrate.