Skip to main content

Telco: the agent that has read the runbook

Track: Telco · In the room: pre-assigned teams · Seed use case: incident triage across runbooks, grounded with AI Search.

Who this is for​

Teams from telecom network operations, with essentially no Cloudflare developer-platform footprint today — not a handicap here. No legacy triage tooling to migrate off, just a greenfield hour and a problem every telco shares: an alert fires, and whoever picks it up has to reconstruct, from memory or a stale wiki, which of the fifty runbooks applies.

The pain is never "we lack monitoring" — telco networks drown in telemetry. It's the translation step between alert and action: minutes spent searching for the right procedure while a cell site, core function, or customer service stays degraded. That's exactly what a grounded agent is for.

Synthetic data only

Invented node ids, alert payloads, runbooks. Nothing resembling real network topology, customer impact, or internal procedure goes into a shared account today.

Three agent use cases​

1. Incident-to-runbook triage — the seed use case​

  • Trigger. A monitoring alert fires — threshold breach, flapping interface, degraded node health. Pushed to the agent; nobody opens a dashboard.
  • Decides. Which runbook procedure matches the alert's signature, how severe it is, and whether the remediation is safe to apply unattended or needs a human watching.
  • Tools. triage_incident matches the alert against the runbook corpus, returns a diagnosis with the passage relied on. propose_remediation turns that into a concrete step — restart a process, fail over a link, rate-limit a neighbour — with citation attached.
  • Human in the loop? Always on the remediation, never on the read. Citing the right procedure is pure gain. Touching a live network element is irreversible in the way that matters: it can turn degraded service into an outage if the diagnosis was wrong, so it waits for a named engineer.

2. Recurrence and flap detector​

  • Trigger. The same triage runs repeatedly because the same node keeps alerting.
  • Decides. Whether this is the fourth unrelated alert today or the fourth occurrence of the same underlying fault — a node worth an engineer's afternoon vs. a dozen separate tickets.
  • Tool. check_recurrence compares the alert against that node's recent history, raises priority once a pattern is confirmed rather than on every instance.
  • Human in the loop? Not for the correlation itself — flagging a pattern is informational. Yes for anything it recommends, like escalating a fault into a planned maintenance window, since that reprioritises other engineers' work.

3. Change-risk pre-check​

  • Trigger. A planned configuration change is about to be pushed to a node or group of nodes.
  • Decides. Whether the node has open or recent incidents that make the change riskier than the ticket assumes, and whether the runbook for this node's function class has a known interaction with the proposed change.
  • Tool. precheck_change cross-references the change against incident history and the relevant runbook section, returns a go / hold-for-review verdict with reasoning.
  • Human in the loop? Yes, as advice a change-approval board can override. The agent never blocks outright — it adds a data point that accelerates a safe change or gives a risky one a second look.

The 60-minute cut​

Build use case 1: an alert arrives, a runbook is cited, a remediation waits for an engineer.

In scope:

  1. Six synthetic alerts (alert id, node id, signal name, description) — two clearly match one runbook each, one matches nothing, the rest overlap enough to need real disambiguation.
  2. Five synthetic runbook entries — invented content — each with id, fault signature, severity, remediation step, and whether that step is safe to auto-apply.
  3. One tool matching an alert to the best-fitting runbook entry, stating honestly when nothing matches well enough to cite.
  4. A Workflow proposing the remediation, pausing on step.waitForEvent() for anything not marked auto-applyable.
  5. A one-page queue: every alert, its cited runbook id, verdict, approval state.

Out of scope, and say so: real alerting integration, real network protocols, ticketing sync, authentication, any topology graph.

The demo that wins. Fire two similar-looking alerts; show the agent citing a different, correct runbook passage for each. Try applying the riskier remediation yourself and get refused until an engineer approves. A room of network engineers trusts a system that shows its citation more than one that shows a confidence score.

Primitives — exactly three​

  1. AI Search over your own documents. The pattern Cloudflare's AI Search product formalises: index a corpus, retrieve the passage that supports an answer, cite it — rather than asking a model to remember runbooks it never saw. For the hour, simulate the pattern with a hardcoded corpus and matching tool, as the prompt below does; wiring the real product onto your own docs is the natural follow-up.
  2. Workflows. Applying a remediation to live infrastructure has to be exactly-once and able to wait for an engineer on another call. step.waitForEvent() holds the run open at no cost; step.do() checkpointing means a retry can't restart the same node twice.
  3. Persistent Durable Object state. An alert only means something in context of that node's history — routine on one node, a red flag on another that's alerted three times today. An Agent instance addressed by node id, with embedded SQLite via this.sql, is where that history lives, strongly consistent and hibernating between alerts.

Skip schedules — an incident is pushed by monitoring, not polled for. Skip Browser Rendering — no public page here, only internal telemetry and your own runbook corpus.

Paste-ready starter prompt​

Scaffold the starter first:

npm create cloudflare@latest -- team-telco-incident-triage --template cloudflare/agents-starter
cd team-telco-incident-triage

Then paste this into your coding agent:

I'm working in a fresh clone of Cloudflare's agents-starter template. The agent code is in
src/server.ts and the React client is in src/client.tsx. I have 60 minutes. Keep the
existing chat agent working — extend it, don't replace it.

Build a network-incident triage agent that cites a runbook before recommending anything.

1. Add a hardcoded array of 5 synthetic runbook entries to a new file. Each has: id (like
   RB-204), a short fault-signature description, severity, a remediation string, and
   autoApplyable (boolean). Invented content only.

2. Add a hardcoded array of 6 synthetic alerts to src/server.ts: alertId, nodeId, signal
   name, description, and timestamp. Make two obviously match one runbook entry each, one
   match nothing well, and the rest ambiguous enough to need real comparison.

3. Add a tool called triage_incident. Given an alertId, it compares the alert description
   against every runbook entry using a Workers AI model and returns JSON: matchedRunbookId
   (or null if nothing fits well), confidence 0 to 1, diagnosis (two sentences that quote the
   matched entry's fault signature), and remediation. If matchedRunbookId is null, say so
   honestly instead of guessing.

4. Add a Cloudflare Workflow called IncidentRemediation in a new file, with steps as
   step.do():
     a) triage — run the triage logic and store the result
     b) await_approval — skip this step entirely if the matched runbook's autoApplyable is
        true. Otherwise use step.waitForEvent() with a generous timeout, waiting for an event
        of type "remediation-decision"
     c) apply — record that the remediation was applied (or declined), and by whom
   Import WorkflowEntrypoint, WorkflowStep and WorkflowEvent from "cloudflare:workers", and
   add the workflow binding to wrangler.jsonc.

5. Store each alert's triage result in the Agent's SQLite storage using this.sql, keyed by
   nodeId, and check that node's last 24 hours of results before triaging a new alert for it
   — if 2 or more prior results reference the same runbook id, flag isRecurring true.

6. Add routes:
     POST /alert/:alertId    starts an IncidentRemediation workflow for that alert
     GET  /queue             HTML page: every alert, its cited runbook id (or "no match"),
                              severity, recurrence flag, and approval state
     POST /decide/:alertId   sends the "remediation-decision" event with approved true or
                              false and an engineer name from the body

Requirements: TypeScript. Route every Workers AI call through AI Gateway by passing
{ gateway: { id: "agenthack" } } as the options argument. Never auto-apply a remediation that
isn't marked autoApplyable. Run `npx wrangler types` after changing wrangler.jsonc. Then
deploy with `npx wrangler deploy` and give me curl commands that prove an autoApplyable
remediation goes straight through and a non-autoApplyable one waits for /decide.

The Worker name team-telco-incident-triage follows the shared-account rule <team_prefix>-<slug>. Eight teams deploy into one account today, so if the Scoping agent issued you a different prefix, use that one instead.

What "production" looks like​

  1. Index your real runbooks in Cloudflare AI Search, not a hardcoded array. Point an instance at actual procedure documentation and require every diagnosis to carry the real source passage. Measure precision on historical alerts before trusting a live one — a wrong citation is worse than none, because it's confidently wrong.
  2. Make the approval a real identity, not a form field. Put your identity provider in front of the decision endpoint so an approval carries a named engineer, their on-call role, and the exact diagnosis shown, retained next to the agent's recommendation.
  3. Feed resolved incidents back into the corpus. An incident resolved with a procedure that wasn't in the runbook is a documentation gap, not a one-off fix. Closing that loop turns a static corpus into a living one, cheaper than any amount of prompt tuning.

Next: Level Up explains each primitive, the cheat sheet has the snippets.