The automation failed silently. A heartbeat catches it
Most automations do not fail loudly, they stop. A reference build with a heartbeat, a dead-letter branch and a queue check, with measured detection times.
An automation almost never fails with an error. It stops, and everything upstream keeps accepting work it will never finish. The first person to notice is usually a customer, days later. The cheapest way to close that gap is not better error handling. It is a liveness signal: one heartbeat write, one dead-letter branch, one queue check, each wired to an alert. In a reference build we ran for this piece, a stopped worker was caught in three seconds. Without the heartbeat, it was still invisible when we stopped watching.
That is the whole argument, and it is deliberately small. Error handling tells you what happened to a message you already know about. A heartbeat tells you that the thing supposed to process messages is no longer there.
The failure that produces no error
Picture the ordinary version. A worker process dies at 02:14. The webhook still returns 202 Accepted, because the queue took the message and the queue is doing its job. The dashboard stays green, because nothing threw an exception. The form on the website still says someone will be in touch. For three days, every lead lands in a table nobody is reading.
Nothing in that story is a bug in the queue. The queue delivered exactly what it promised: durable storage. The bug is that "the message was accepted" got treated as "the work will happen".
Teams usually respond by adding more error handling to the worker. That helps with messages that fail while being processed. It does nothing for a worker that is not running, because a stopped process does not raise an exception. It also does nothing for a consumer that is running but stuck, which is the more common production shape.
Your platform already retries. That is not monitoring.
Most queue products will retry a failed message a few times and then make a decision about it. The decision is the part worth reading.
Cloudflare Queues defaults to three retries per consumer. If no dead-letter queue is configured, the documentation is blunt: messages that repeatedly fail processing "will eventually be discarded." Discarded means no record, no count, no alert. A retry loop looks like health for as long as it lasts, and then the message is gone.
Amazon SQS behaves in the same family. You attach a redrive policy with a maxReceiveCount; messages that exceed it move to a dead-letter queue. If you never configure that policy, the message cycles until retention expires and disappears. The AWS docs also recommend an alarm on the dead-letter queue, which is the quiet admission that the queue by itself is storage, not a signal.
Temporal makes the same point from a different angle. Its Schedule-To-Start timeout, which is the limit on how long a task may wait before a worker picks it up, defaults to infinity. The docs recommend watching the schedule-to-start latency metric instead of relying on the timeout, because a fleet of workers that never starts a task does not produce a failed task. It produces nothing.
The pattern repeats across vendors: retries are a delivery mechanism, not an observability mechanism. They protect the message. They say nothing about whether anyone will ever look at it.
Three signals, three different questions
The mistake I see most often is instrumenting one of these and calling the workflow monitored. They answer different questions and fail in different ways.
| Signal | The question it answers | What it cannot tell you |
|---|---|---|
| Liveness (heartbeat) | Is the worker process alive and cycling? | Whether it is doing any useful work |
| Terminal failure (dead-letter) | Did a message exhaust its retries? | Whether a message ever arrived |
| Throughput (queue depth or age) | Is work moving through the system? | Whether the queue is empty because nothing is being produced |
A heartbeat is the only one that catches a dead process. A dead-letter branch is the only one that preserves the payload of a message that failed. Depth or age is the only one that catches a worker which is alive, beating, and consuming far too slowly.
Healthchecks.io describes the liveness check in the cleanest way I have seen: you set a period (the expected time between pings) and a grace time (how long to wait before calling it late). That is exactly the arithmetic a heartbeat alert needs. It also makes the tradeoff explicit: a short period catches failure quickly and pages you for normal jitter; a long period is quiet and slow. There is no setting that is both.
What we measured
To avoid arguing from intuition, we built the smallest version of the pattern and broke it on purpose. The build was a local reference implementation: a SQLite-backed queue, a worker loop, a downstream call that could be switched to return 500, and a monitor polling every 250 ms. The heartbeat fired every second and alerted after three missed beats. Retries used a 0, 1, 2, 4 second backoff, then the message moved to the dead-letter table. Each scenario ran twice on the same machine; the times below are rounded to a tenth of a second.
| Failure injected | Guard that was on | Time to alert | Same failure, guard off |
|---|---|---|---|
| Worker process stopped | Heartbeat, 3 missed beats | 3.0 s | No alert in the 12 s observation window |
Downstream returned 500 | 4 attempts, then dead-letter | 7.3 s | No alert; message dropped after the retries |
| Consumer far slower than arrivals | Queue depth ≥ 5 | 7.3 s | No alert; queue kept growing |
The number that matters is not 3.0 seconds. It is the right-hand column. In the unguarded build, none of the three failures produced a signal at all. A stopped worker and a worker with an empty queue are indistinguishable from the outside, which is precisely why "no errors in the logs" is such a weak statement.
The 7.3 seconds on the 500 case is also worth reading correctly. That is not detection speed; it is the length of the retry schedule before the platform gives up. The dead-letter branch did not make the failure happen faster. It made the failure visible at the moment the platform had already decided to abandon the message.
The smallest build that works
You do not need a platform team. You need four records and one monitor.
- A work queue with an attempt counter per message.
- A heartbeat row the worker writes on a fixed interval, ideally with a workflow name and a worker identity.
- A dead-letter record written when attempts are exhausted, with the original payload, the last error and a timestamp.
- A monitor that reads all three and does three comparisons: newest heartbeat too old, any new dead-letter row, queue depth or age above a threshold.
The parameters matter less than the fact that they are written down. A starting point that held up in the test:
beat interval 1 s alert after 3 missed beats monitor interval 250 ms retry backoff 0, 1, 2, 4 s dead-letter after 4 attempts backlog threshold 5 waiting messages
Two rules make the difference between a control and a decoration. First, the alert must name the workflow and the failure — "queue stuck" is not actionable at 3 a.m., "lead-router consumer has not beaten in 4 minutes, 12 messages waiting" is. Second, every alert needs a named owner and a replay path for the dead-letter record. A dead-letter queue that nobody drains is a tombstone with better branding.
Break it before you trust it
An alert you have never seen fire is a hope, not a control. The acceptance test is the same three injections, run on purpose, in this order:
- Stop the worker. Time how long until the alert arrives. It should be the heartbeat threshold plus one monitor interval, not "eventually".
- Make the downstream return
500. Confirm the message reaches the dead-letter record with its payload intact, and that the alert fires once rather than once per retry. - Slow the consumer below the arrival rate. Confirm the depth or age check fires while the heartbeat stays green. This is the case a liveness check alone will always miss.
Then run the boring control: a quiet period with normal traffic. If the alert fires there, the threshold is wrong, and a noisy control gets muted within a week.
What this still does not catch
The pattern is small on purpose, so it has edges. A heartbeat written by a scheduler rather than the consumer proves the scheduler is alive, not the worker. A webhook that fails before enqueue never creates a message, so no queue metric will see it; that needs a producer-side success rate. A dead-letter record protects the payload but says nothing about the customer who was waiting. And every threshold is a guess until you have measured a normal day.
None of those gaps argues for a bigger platform. They argue for naming the gap next to the control, so the next person knows what the green light actually means.
The Monday version
Pick the one workflow where a silent stop costs money — lead routing, order confirmation, invoice generation. Add the heartbeat write and the monitor first, because that is the only part that catches a dead process. Add the dead-letter branch second, so failures leave evidence. Add the depth check third, because it is the one that catches the slow rot. Then break all three and write down the times. That number is your real detection time, and it is the only one worth putting in a status report.
If you are running automations that answer customers and route revenue, the Automation & Chatbots service is where we build this in. If you already have the workflow and only want the reliability review, start here. The related reading is bot-to-human handoff, the chatbot quality scorecard and the agent evaluation harness. All three are about the same question from different sides: how do you know the thing you shipped is still doing its job?
Frequently Asked Questions
What is a heartbeat in an automation?
A heartbeat is a timestamp the worker writes on a fixed interval. A monitor alerts when the newest timestamp is older than the expected interval plus a grace period. It answers one question only: is the process still alive enough to write?
Does a dead-letter queue catch every failed automation?
No. A dead-letter queue records messages that exhausted their retries. It does not catch a worker that stopped consuming, a webhook that failed before the message was enqueued, or a queue that is empty because nothing is producing work.
How often should the heartbeat fire?
Faster than the business can tolerate silence, and slower than normal noise. Start with the shortest interval you can alert on without false positives, then set the alert to three missed beats. A one-minute heartbeat with a three-minute threshold is a reasonable default for lead routing.
