Skip to main content

Platform and SaaS: the agent that runs the next experiment

Track: Platform and SaaS · In the room: pre-assigned teams · Seed use case: experiment-analysis agent that designs its own follow-up test.

Who this is for​

Teams from an experimentation/A-B-testing platform, with Workers and KV already in production, running a business built on other companies' experiments. An experiment result is not an endpoint — it's a fork: ship it, kill it, or run the obvious next test — and today that fork is a human staring at a dashboard deciding which. The pain isn't running experiments, that's solved and it's your product; it's the gap after a result lands, where someone has to read it, understand the implication, and decide what's next — usually the busiest analyst on the team, which is why the backlog of "obvious" follow-up tests that never got run keeps growing.

Synthetic data only

Invented experiments, metrics, and conversion numbers. Nothing resembling a real customer's experiment data or business result goes into a shared account today.

Three agent use cases​

1. Experiment-analysis agent — the seed use case​

  • Trigger. Experiment reaches statistical readiness — enough samples, or a time box elapsed. The agent notices on its own; no dashboard check needed.
  • Decides. Whether the result is significant or inconclusive, what it implies about the hypothesis, and the natural next test — same win on a different surface, same null result with a different lever, or a segment worth splitting out.
  • Tools. analyze_experiment interprets the result and drafts the implication in plain language; design_followup proposes the next experiment's hypothesis, variant, and target metric, grounded in this experiment and its predecessors.
  • Human in the loop? Reading and interpreting needs no approval — pure analysis. Launching the follow-up always does: it spends real traffic and time against a metric the business watches, and a bad hypothesis costs weeks, not minutes.

2. Under-powered-test flagger​

  • Trigger. An experiment is running, partway through its planned duration.
  • Decides. Whether current traffic allocation and effect size make a real answer likely, or the test is quietly heading for an inconclusive result that wastes its time box.
  • Tool. check_power estimates whether the test is on track and, if not, proposes a fix — more traffic, a longer window, or a bigger minimum detectable effect.
  • Human in the loop? Only on the fix. Flagging is informational and safe; changing a live experiment's allocation or duration is the experiment owner's call — it affects everyone sharing the traffic pool.

3. Cross-experiment conflict detector​

  • Trigger. A new experiment targets a page or flow another experiment already runs against.
  • Decides. Whether the two would interact — same page, overlapping audience, metrics that could contaminate each other — and how serious the overlap is.
  • Tool. check_conflicts compares the proposal against every running experiment and lists what overlaps and why it matters.
  • Human in the loop? Yes, a launch reviewer's decision. The agent never blocks a launch — it ensures the reviewer sees the overlap before go-live, not after the data is contaminated.

The 60-minute cut​

Build use case 1: notice on a clock, analyse with memory, gate the launch.

In scope:

  1. Four synthetic experiments (id, hypothesis, variant description, target metric, control/variant samples and conversions) — one clear win, one clear loss, one inconclusive, one segment-specific win.
  2. Scheduled check via this.schedule() for experiments whose sample count crosses a threshold you pick, every 60 seconds so it fires on stage (say "production would check hourly").
  3. One tool interpreting the result, one drafting a follow-up proposal, stored together as the experiment's analysis.
  4. A Workflow proposing the follow-up, pausing on step.waitForEvent() before it's "launched."
  5. A one-page view of every experiment: verdict, proposed follow-up, approval state.

Out of scope, and say so: a real statistics engine, real traffic splitting, integration with an actual experimentation platform, authentication, multi-armed bandits.

The demo that wins. Let the schedule fire on stage and watch an experiment cross the readiness threshold on its own — show the interpretation citing a prior iteration ("the last test on this flow showed X, this confirms it for mobile"), then the follow-up sitting in PENDING APPROVAL until you launch it live.

Primitives — exactly three​

  1. Schedules (this.schedule()). An experiment becomes ready on its own clock — samples or a time box — not a pushed event. Keeps the readiness check next to the experiment history it needs to interpret the result.
  2. AI Search over your own documents. A follow-up is only useful if it isn't a test you already ran. Grounding design_followup in an indexed archive of past write-ups stops it confidently re-proposing something the team already rejected — a larger model can't cite a test it was never shown.
  3. Workflows. Launching a follow-up spends real traffic and time, and must be exactly-once — a retry must not launch the same test twice. step.waitForEvent() holds the proposal open until an experiment owner approves it, at no cost while waiting.

Skip Browser Rendering — nothing external to render, the loop reasons over your own numbers and document archive. Skip Durable Object state as a named choice — you get it free from the Agent class holding each experiment's history.

Paste-ready starter prompt​

Scaffold the starter first:

npm create cloudflare@latest -- team-platform-experiment-followup --template cloudflare/agents-starter
cd team-platform-experiment-followup

Then paste this into your coding agent:

I'm working in a fresh clone of Cloudflare's agents-starter template. The agent code is in
src/server.ts and the React client is in src/client.tsx. I have 60 minutes. Keep the
existing chat agent working — extend it, don't replace it.

Build an experiment-analysis agent that designs its own follow-up test and never launches
one without approval.

1. Add a hardcoded array of 4 synthetic experiments to src/server.ts. Each has: experimentId,
   hypothesis, variantDescription, targetMetric, controlSamples, controlConversions,
   variantSamples, variantConversions, and a readySampleThreshold. Make one a clear win, one
   a clear loss, one inconclusive (samples below threshold), and one a win with a note that
   it is segment-specific. Invented data only.

2. Add a hardcoded array in a separate file of 3 synthetic "past experiment" summaries — id,
   hypothesis, and outcome — that a real archive would hold. Invented content only.

3. Add a tool called analyze_experiment. Given an experimentId, if combined samples are below
   readySampleThreshold, return status "not_ready" and stop. Otherwise use a Workers AI model
   to return JSON: verdict (win, loss, inconclusive), liftPercent, confidence (low, medium,
   high), and a two-sentence plain-language interpretation.

4. Add a tool called design_followup. Given an experimentId and its analysis, compare against
   the past-experiment archive and use a Workers AI model to return JSON: proposedHypothesis,
   proposedVariant, targetMetric, and a citation naming which past experiment (if any)
   informed the choice, or null if none did. Never fabricate a citation to an experiment not
   in the archive.

5. Use this.schedule() to run a sweep every 60 seconds. For each experiment not yet analysed
   whose combined samples exceed readySampleThreshold, run analyze_experiment then
   design_followup, and store both keyed by experimentId in the Agent's SQLite storage via
   this.sql, updating rather than duplicating on re-runs.

6. Add a Cloudflare Workflow called FollowupLaunch in a new file, with steps as step.do():
     a) propose — record the proposed follow-up
     b) await_approval — step.waitForEvent() with a generous timeout, waiting for an event of
        type "followup-decision"
     c) record — record whether the follow-up was launched or declined, and by whom
   Import WorkflowEntrypoint, WorkflowStep and WorkflowEvent from "cloudflare:workers", and
   add the workflow binding to wrangler.jsonc.

7. Add routes:
     POST /launch/:experimentId   starts a FollowupLaunch workflow for that experiment's
                                   proposal
     POST /decide/:experimentId   sends the "followup-decision" event with approved true or
                                   false
     GET  /experiments            HTML page: every experiment, its verdict, its proposed
                                   follow-up and citation, and its launch approval state

Requirements: TypeScript. Route every Workers AI call through AI Gateway by passing
{ gateway: { id: "agenthack" } } as the options argument. Run `npx wrangler types` after
changing wrangler.jsonc. Then deploy with `npx wrangler deploy` and give me curl commands
that show the schedule firing and one experiment going through propose, decide, and record.

The Worker name team-platform-experiment-followup follows the shared-account rule <team_prefix>-<slug>. Eight teams deploy into one account today, so if the Scoping agent issued you a different prefix, use that one instead.

What "production" looks like​

  1. Replace the hardcoded numbers with a real statistics engine. Sequential testing and confidence intervals are solved elsewhere — the agent interprets and proposes, not computes significance, so wire it to whatever engine backs your platform's real results.
  2. Index your actual experiment archive, not three invented entries. design_followup's value scales with how much real history it can cite — start with a year of write-ups and measure how often engineers accept the proposal as-is versus rewrite it.
  3. Track the acceptance rate of proposed follow-ups over time. That number shows how much of the analyst's job the agent is doing versus how much noise it generates — and it's the number that justifies widening its autonomy, or doesn't.

Next: Level Up explains each primitive, the cheat sheet has the snippets.