Alerts that don't spam, and the state machine behind them
Why a plain threshold check floods phones, and the four-state machine that sends one alert, a few reminders and one all-clear.
The first alert rule anyone writes sends a message when the latest reading is over a limit. It works in the demo. On a real site, a cold room door left open on a hot afternoon produces a reading every minute above the limit, and each becomes an email, a WhatsApp message, a Telegram message and a push notification. By the time a technician arrives, their phone holds forty copies of the same news, and the message they actually want, "it's back to normal", never comes, because the rule has no idea what normal is.
This is how I think about alerting in LiquidInsights, the IoT and energy operations platform I build at Savtech. The design below is simplified from that production platform to show the idea; it is not copied from it.
Why the naive check fails
The naive rule is one comparison: reading over the limit, notify every channel, on every reading, forever. Everything wrong with it comes from one fact: it has no memory.
- ›No debounce. A single spike from a door opening for ten seconds pages someone.
- ›No repeat policy. A sustained fault sends a message per reading. A sensor reporting every minute is sixty messages an hour per channel.
- ›Flapping. A value hovering around the limit crosses it on alternate readings, and each upward crossing looks like a brand new incident.
- ›No all-clear. Nothing knows the fault ended, so nobody is told.
- ›Retries duplicate. If the handler is retried after a timeout, the message goes out again.
Four states, not one boolean
Each rule, for each device it watches, sits in one of four states.
- ›Normal. The value is fine. Nothing to say.
- ›Pending. The value has crossed the limit but has not stayed there for the on delay. If it drops back, we return to Normal and nobody hears about it. This is the debounce.
- ›Active. The breach has lasted long enough. We send the ON alert once, then a repeat every N minutes if the rule asks for it.
- ›Clearing. The value is back in range but has not stayed there for the clear delay. If it breaches again, we go straight back to Active without a new alert. If it stays clear, we send the return-to-normal message and go to Normal.
If a rule has no on delay, a breach skips Pending and raises at once; with no clear delay, a clear skips Clearing the same way.
Key ideaAn alert is a change of state, not a reading above a limit.
Then the messages per incident are fixed by the rule, not by how often the sensor reports: one ON, a bounded number of repeats, one clear.
The transition function
I keep the transitions in a pure function. It takes the current state, the rule and a reading, and returns the next state plus at most one notification: Raised, Repeat or Cleared. No database, no clock, no I/O.
Two choices matter here. Time comes from the reading's timestamp, not from the server clock, so replaying a day of telemetry through the function gives the same alerts it gave live. And the sequence number goes up by one with every notification, which gives each message a stable identity that survives retries. More on that below.
Flapping and the deadband
A clear delay alone doesn't stop flapping if the value sits right on the limit. A temperature reading 79.9, 80.1, 79.9 would bounce between Active and Clearing forever. Two things handle it.
The deadband means "over the limit" and "back to normal" are different thresholds. With a limit of 80 and a deadband of 5, the alarm raises above 80 but only starts clearing below 75. Readings between the two keep whatever state we are in.
The silent re-arm means a breach during Clearing goes back to Active without sending anything. The technician already knows; the incident never ended.
A small unit test pins this down: limit 80, deadband 5, on delay two minutes, clear delay ten minutes, no repeats. Readings at minutes 0, 1, 2, 5, 6, 8, 9 and 20 have values 85, 79, 86, 87, 74, 81, 70 and 70.
The 79 cancels the spike's on delay. The 86 starts it again, and by minute 5 the breach has lasted three minutes, so the alarm is raised. The 74 is below 75, so Clearing starts; the 81 re-arms silently. The 70 at minute 9 starts Clearing again, and eleven minutes later the all-clear goes out.
The test asserts exactly one Raised and one Cleared, ending in Normal. Eight readings that the naive check would turn into four alerts become one alert and one all-clear.
Where the state lives
The state has to outlive the process. In memory, every deploy forgets which alarms are active: an alerted fault is raised again, and one that ended during the deploy never gets its all-clear.
So each (rule, device) pair has one row in SQL Server holding the phase, the window start, the last notification time, the sequence and the current alarm id. The evaluator loads the row, runs the pure function, and writes the row back only if something changed. Most readings end there: no write, no message.
A notification becomes an outbox row in the same transaction as the state change, so both are saved or neither is. Raising creates a new alarm id; clearing releases it. A row version makes two evaluations racing on the same rule and device fail instead of overwriting each other.
A few details that took me longer to get right than the state machine itself:
- ›One writer per device. Readings for the same device are evaluated in order by one worker. The row version is the safety net, not the plan.
- ›Late readings. A reading older than the last one evaluated for that rule is ignored. Otherwise a delayed batch from a gateway can drag an Active alarm back through Clearing.
- ›Delays only advance on readings. If a device reports every fifteen minutes, a two-minute on delay effectively becomes fifteen. That is usually fine, and it is honest: we only know the value when the device tells us.
One alarm, many channels, no duplicates
The evaluator never sends anything. It writes an outbox row, and a Hangfire job picks it up after commit. A recurring sweep catches any row whose job never got enqueued, for example if the process died between commit and enqueue.
The dispatcher fans each notification out to every channel and recipient subscribed to the rule. The trick is that uniqueness is enforced by unique indexes at every step, as the diagram shows, not by hoping the code runs once. A retried evaluation cannot create a second notification.
Each delivery row is inserted with status Sending before the call to the provider and marked Sent after. If the insert hits the unique index, that delivery is already done or in progress, and the job moves on. Channels are independent: a WhatsApp outage retries WhatsApp only and never re-sends the email.
There is one gap I accept on purpose. If the process dies after the provider accepted the message but before the row is marked Sent, the retry sends it again. External APIs don't give you exactly-once, and for an alarm I would rather send a rare duplicate than lose the message.
Tying the alarm to a work order
When an alarm is raised, the same job opens a work order, keyed by the alarm id so a retry can't create a second one. The filtered unique index adds a rule that matters for flapping: if a work order is still open for that rule and device, a new alarm attaches to it instead of opening another. A technician dealing with a faulty compressor should see one job with three alarm events on it, not three jobs.
Return to normal does not close the work order. It adds a note with the clear time. A value going back in range tells you the symptom stopped, not that someone fixed the cause, and closing the job is a human decision.
What I'd do differently
- ›Model "no data" as a state from day one. A sensor that stops reporting is often the real fault, and bolting it on as a separate rule type meant two sources of truth for one device's health.
- ›Keep a transition history table. The state row tells you where an alarm is, not how it got there. When someone asks why they got a message at 3am, an append-only log of every transition answers it in seconds.
- ›Make the deadband required. I let it default to zero early on, and zero deadband is exactly the configuration that flaps. Sensible per-unit defaults would have saved support time.
- ›Add a timer-driven sweep for delays. Advancing delays only on readings is simple and correct, but users read "clear delay: 5 minutes" literally. A periodic sweep over Pending and Clearing rows would match their expectation for slow-reporting devices.