Stop Re-Investigating the Same Alert
Your AI SRE shouldn’t burn tokens on every Slack follow-up.
The incident patterns below are composite and anonymized. Counts are rounded. Names, IDs, and infrastructure details are fictionalized to protect customer confidentiality.
TL;DR
- Duplicate investigations on the same alert are expected when chat mentions do not reuse completed RCAs
- In one anonymized week: ~100 investigations, ~15 alerts with multiples, one alert with 20+ full digs
- Fix: reuse-first launch policy — answer from the prior summary unless the operator asks to re-investigate
- Service metric: investigations per alert per week, not fluency of the latest write-up
Explain like I’m five
If someone already wrote the book report, don’t write it again every time a classmate asks “what was that book about?” Hand them the report. Only rewrite if they say “start over.”
The Acme Commerce pattern
Composite story, drawn from production debug export analysis:
Alert: worker-service errors on invoice jobs missing a required tenant_id field after a bad release.
Behavior: The alert fired on an hourly cycle. On-call @mentioned the bot in #incidents-prod with “why is this still firing?” Each mention launched a full investigate workflow. Session after session rediscovered the same KeyError pattern and the same release candidate.
Outcome: Roughly twenty completed investigations on one alert ID in a week. Token spend and wall time scaled with chatter, not with new evidence.
The agent was not “wrong.” The service was. It treated every human message as a request for a new deep dive.
What reuse-first looks like
When a recent terminal RCA exists for the alert (default cooldown on the order of hours):
- Do not launch a new investigate-alert workflow
- Answer from the prior investigation summary
- Include the watch / session link so the human can open the receipts
- Only start fresh when the operator explicitly asks — “re-investigate,” “from scratch,” “force new”
That is the difference between a chatbot that always digs and a service that remembers what it already rendered for this alert.
Slash-style escapes (/reinvestigate) make intent obvious. Soft “please check again” language should still hit the reuse path unless the operator opts out.
If you lead an SRE team
- Chart investigations per alert ID weekly. Spikes mean the product is redoing work, not that on-call is curious
- Train the channel: follow-ups get the prior summary; say “re-investigate” when you want a new dig
- Count time-to-first-useful-answer on first investigate, then reuse latency on follow-ups — different SLAs
If you ship the agent platform
- Short-circuit on recent terminal status before spawning collectors
- Encode “fresh dig” as an explicit intent regex or slash command, not as ambient enthusiasm in the prompt
- Stamp prior investigation id + finished time into the reuse prompt so the model cannot invent a new story from thread vibes alone
Related
- Series opener: Service Rendered Efficiently
- Next: Slack Is a Triage Board, Not a Log Dump
- Token budgets: LLM tokenomics
- Checklist: SRE as service
Acknowledgments. Reuse-first launch policy lessons from shipping Aiden SRE chat investigation. Customer details composite.
Building AI for incident triage without the demo theater? Find me on GitHub or LinkedIn.
FAQ
Why do AI SRE agents re-investigate the same alert?
Hourly alert cycles and Slack follow-ups often launch a full investigate workflow even when a completed RCA already exists. Without a reuse-first policy, every @mention looks like a new job.
What should operators do instead of re-running investigate?
Default to the prior summary and watch link within a cooldown window. Explicitly ask to re-investigate or start from scratch only when they need a fresh deep dive.
What metric should SRE leads track?
Investigations per alert ID per week — not just model accuracy. Twenty full digs on one alert is usually a product failure, not a smarter-prompt opportunity.
Stay in the loop — production notes on AI agents, workflows, and SRE.
Low volume — new posts and curated reading lists. Unsubscribe anytime.