Prometheus Alerting: Lessons from Six Months of False Positives

Three in the morning, the phone buzzes, nothing is on fire

The first version of my alert rules woke me up regularly. Not because the server was failing — because the rules were naive. A memory spike during the nightly media library scan looked like a leak to them. A restart counter that ticked once kept alarming forever. And a metric I trusted was measuring something different than I thought.

Today the server runs over seventy rules, and most weeks none of them speaks up without a reason. This post is a workshop: what specifically changed between the version that woke me and the version that wakes me only for real. A false alarm is not harmless — every one of them teaches you to ignore notifications, and monitoring nobody reads doesn't exist.

Before you write a rule, look at the metric

The most expensive lesson of the whole six months: a metric doesn't always measure what its name suggests. Before you write a threshold, type the metric into the Prometheus console and check three things: whether it exists under that name at all, what labels it carries, and whether its values match the reality you know from elsewhere.

The classic of this trap: container_memory_usage_bytes includes the filesystem cache, so it inflates "usage" by tens of percent — a container looked like it was eating 433 MB when it was really working with 326 MB. The right metric is container_memory_working_set_bytes. I told that story in full in the monitoring stack case study; here I'll leave just the workshop rule: a threshold on a metric you haven't inspected is guessing with a straight face.

for: — an alert is a state, not a spike

The second fix with the biggest payoff was the simplest one. An alert without a for: clause fires on the first sample that matches the condition — which means on every spike, every scrape restart, and every brief network hiccup.

The distribution across my seventy-plus rules speaks for itself: most rules wait 5 minutes, critical availability ones 2 minutes, and slow trends an hour (the three most common settings are 25 rules at five minutes, 14 at two, and 9 at an hour; the rest spread out from half a minute to six hours). There's also an exception with a history: the container high-memory alert waits half an hour, because the nightly media library scan can legitimately push usage above the threshold for tens of minutes — and the first version of that rule alarmed every night at the same time.

- alert: ServiceDown
  expr: up == 0
  for: 2m    # a scrape restart is not an outage
- alert: ContainerHighMemory
  expr: ...  # working_set > 85% of the limit
  for: 30m   # the nightly library scan legitimately raises memory

Choosing for: is always the same trade-off: too short — spikes wake you; too long — you're the last to know. A practical heuristic: how long can this problem last before it actually breaks something? For a dead reverse proxy, minutes. For a memory trend — hours.

predict_linear: alarm before it happens, not when it twitches

A static memory threshold has a built-in dilemma: set it low — it alarms during normal operation; set it high — it wakes you when it's already too late. For memory leaks a trend works better: predict_linear() extrapolates the last few hours and asks whether, at the current pace, the container will hit its limit.

My leak-detection rule went through three versions, and every fix was a lesson:

- alert: ContainerMemoryLeak
  expr: |
    predict_linear(container_memory_working_set_bytes{...}[6h], 6*3600)
      > container_spec_memory_limit_bytes * 0.95
    and container_memory_working_set_bytes{...}
      / container_spec_memory_limit_bytes > 0.5
    and delta(container_memory_working_set_bytes{...}[1h]) > 50 * 1024 * 1024

Three conditions joined with and, because each one lies on its own. The prediction alone, on a three-hour window, reacted to natural oscillations — a container drifting between 50 and 65% of its limit occasionally looked like a rocket. Widening the window to six hours calmed the trend, the "already using more than half the limit" condition filtered out freshly started containers, and the requirement of real growth (over 50 MB per hour) removed the ones that simply sit high and do nothing wrong.

Log-based alerts: three traps the tutorials skip

Metrics don't see everything — a process killed by the OOM killer leaves a trace in the logs, not on a chart. That's what rules in Loki are for. And here the traps were of an entirely different kind than in Prometheus, because pattern matching against text has its own ways of being spiteful:

Word boundaries. The pattern oom also matches "Room" and "Zoom" — any chat or device name in the logs can fire a memory alert. The rescue: \boom\b. Similarly, "killed" is too broad — it catches innocent sentences in application logs; you need to match specific phrases like "OOM-killer" or "was killed".

The feedback loop. My rule searching container logs for OOM patterns found them… in Loki's own logs, which after all process those patterns. The alert kept firing itself forever. Since then every log rule carries the exclusion container_name!="loki".

Know your labels. A rule is only as good as its selector. In my setup the job is called containerlogs and container names have no leading slash — but with a different promtail configuration the same rules would match nothing and stay silent forever. Before you trust a log rule, check in the console that its selector returns anything at all. Silence can be a configuration error, not health.

The validator passes, the semantics lie: what an audit found

After half a year of tuning I ran a full audit of every rule — not because something was broken, but because everything looked like it was working. What turned up were bugs no syntax validator will ever catch, because syntactically everything was fine.

An inhibition rule dead from day one. Alertmanager lets you suppress related alerts: when a whole service is down, I don't want a separate alarm about its high CPU. My rule used target_match with the value 'HighCPUUsage|HighMemoryUsage' — and target_match matches exactly, character by character. The regex variant is called target_match_re. A literal containing | is a perfectly valid string, so the validator stayed quiet, and for half a year the rule suppressed nothing. On top of that, the alert it was supposed to target was actually named differently. Two bugs in four lines of configuration, zero warnings.

One failure, three critical notifications. Alertmanager groups notifications by alert name, among other things — every name is a separate e-mail. Over six months of adding availability rules I never noticed that a single failing metrics exporter met the conditions of three different rules with three different names. One outage, three critical notifications about the same event. The fix has two parts: rules that were 1:1 duplicates simply went away, and where a dedicated alert carries a better description than the generic one, the dedicated alert now inhibits the generic one (inhibition with equal on a shared job or instance label).

Excluding something that doesn't exist. The generic availability rule excluded the job blackbox-http — which didn't exist; it had been renamed at some point and the exclusion stayed behind. A dead exclusion is worse than no exclusion: it suggests the case is handled, so nobody ever looks there.

Operational takeaways from this round: routing can be tested behaviourally — amtool config routes test with the labels of a sample alert shows which receivers it will reach. Inhibitions need a separate check: whether the alert names and labels in source_match, target_match and equal actually occur in the rules — and in a test environment, on a pair of synthetic alerts as well. Plus a deployment detail that cost me a quarter of an hour: after editing a file mounted into a container as a single bind mount, the container keeps seeing the old version, because the editor swaps the inode — only a container restart helps, not a reload.

The audit also threw in a finding from outside alerting itself: Loki had been writing its index in a format that had meanwhile been marked as deprecated (boltdb-shipper). Migrating to its successor (TSDB) turned out safer than it sounds — a new entry in schema_config with a future date, old data stays readable by the old engine until retention expires, and the rollback is simply removing the entry before that date arrives.

The last mile: from JSON to a human sentence

Even a well-tuned alert reaches your phone as a robot-to-robot message: labels, values, identifiers. The final leg of the pipeline used to be handled by two local language models via Ollama — and the division of labour between them is a lesson of its own, measured rather than assumed.

The plan was simple: one model classifies the priority and writes the message. The tests said otherwise. The small model classified flawlessly and fast, but its Polish turned out crippled. Bielik — the Polish model from SpeakLeash — wrote naturally, but flattened classification: everything landed mid-scale, a critical disk included. Neither could do both at once. The solution: a two-stage pipeline — one model classifies, the other writes the final message in Polish.

Deployment added traps of its own. Through the raw generate endpoint, Bielik would sometimes echo the prompt back instead of answering — what helped were chat-style calls with example answers in the system prompt (few-shot); without them reliability was a lottery, with them every test passed. A single example, in turn, anchored the style — the model copied its closing line into every message; two examples with different endings solved it. And the overriding rule of the whole stage: fail-open. When the model fails or echoes (a simple content guard catches it), the alert goes out with a fallback message. Nice language is optional — delivery is not.

All of that worked. Then I measured what it actually cost — and took it back apart.

The escape hatch that never fired

From the start I knew I did not want to wake the models on every alert. Recurring patterns were supposed to carry a fixed class and a ready-made message, bypassing the AI entirely. I wrote that list, put the nineteen most frequent alerts on it, and spent six months convinced it worked. I even described it in the first version of this post.

It did not work. The condition that skipped the model read "skip if the message is empty" — and an alert from the list always had a message, because that was its entire point. The flag meaning "this alert needs no AI" appeared in that condition exactly zero times. The shortcut bypassed the first model of two. The second one was the slow one.

Numbers from a single run say more than a description. Two alerts about an unreachable endpoint — an entry from the list, in theory the "fast path":

On top of that, the model overwrote my own message with a worse version. My hand-written "Endpoint unavailable" became a sentence with an English name and a raw address inside it — breaking the very system prompt that forbade exactly that. I was paying five minutes of CPU to degrade text I had already written myself.

Scale over thirty days, forty-five notifications:

Every third notification hit a timeout. Alerts were not lost — the whole path is fail-open, so they went out with the fallback message — but they were late.

Inversion: the model as an option, not the default

The real cause was not that one condition. The alert rules lived in the Prometheus config, and the list of "known" patterns lived in code inside the automation. Nothing tied them together. Adding a rule did not force adding it to the list, so every new rule fell into the expensive path by default. The drift was not an oversight — it was built into the design. When I counted, eleven of the fourteen alerts that actually fire day to day were off the list.

The fix is not a patch but an inversion of the default. The message and priority are now composed from annotations on the rule itself — where the rule lives:

- alert: LowDiskSpace
  expr: ...
  labels:
    severity: critical
  annotations:
    summary: "Low disk space on {{ $labels.mountpoint }}"
    summary_pl: "Kończy się miejsce na dysku {{ $labels.mountpoint }} — zostało mniej niż 15%"
    priority: "HIGH"
    # ai_enrich: "true"   # model ONLY on explicit request

The model is enabled only by a label on a specific rule. A new rule can no longer slip quietly into the expensive path — at worst it comes out in English because nobody wrote the localised version. That is a visible, cheap defect, unlike a five-minute delay.

The one-off cost: writing localised descriptions for all seventy-three rules. It sounds like a lot, but it is the same work the model was doing on every single alert firing — except now it is done once, and better. Before this, five rules had a localised description.

There is a second reason, and it matters more than CPU load: coupling. Model inference is a long, indivisible task — once started, you wait it out. As long as alerts went through the same engine as other work, they queued behind it. Which means that in exactly the scenario where notifications matter most — an outage and a burst of alerts at once — they waited longest.

A late alert is worse than an awkwardly worded one. A notification path should be independent of anything that can occupy a resource for a while — and model inference is exactly that kind of occupation. This is an architectural argument, not a cost-saving one: at a handful of alerts a day, electricity is not the stake here.

The delivery path itself — who gets notified, over which channel, what happens during quiet hours — is covered in the post about n8n patterns, and choosing models for CPU without a GPU in the post about running a local LLM.

Checklist before deploying a rule

• Inspect the metric in the console — it exists, it has those labels, it measures what you think.
• Counter? Compare the increase (increase(...[5m]) > 0), never the raw value — otherwise the alert never clears.
• Add a for: matched to the question "how long can this problem last before it hurts".
• Trend instead of threshold where problems build slowly — predict_linear with conditions that cut off oscillations.
• Log rules: word boundaries in patterns, exclude the logging system's own logs, a verified selector.
• The rule description = the first diagnostic step, not a restatement of the name. At three in the morning you'll thank yourself for a ready-made command.
• Every false alarm corrects the rule and leaves a dated comment. A silence is not a fix.
• A quarterly semantics audit — do the alert names in inhibition rules and the jobs in exclusions still exist; check routing with amtool config routes test on a sample of labels.
• Write the localised description on the rule, not in the delivery path. One source of truth instead of two that will drift apart.
• Measure the distribution of delivery times, not just the count of successful runs — otherwise you will not notice that your fast path is fast only on the diagram.

Six months later, monitoring wakes me a few times a quarter — and every time rightly so. That's the proper measure: not the number of rules or dashboards, but trust in the phone buzzing in the middle of the night.