Skip to main content

Platform and SaaS: the agent that runs the next experiment

Track: Platform and SaaS · In the room: Optimizely · Seed use case: experiment-analysis agent that designs its own follow-up test.

Who this is for​

Optimizely. A meaningful Cloudflare developer-platform footprint already — Workers and KV in production — running a business built entirely on other companies' experiments. You already understand better than almost anyone in the room that an experiment result is not an endpoint, it is a fork: ship it, kill it, or run the obvious next test. Today that fork is a human staring at a dashboard deciding which of the three it is.

The pain is not running experiments — that part is solved and it is your product. The pain is the gap after a result lands: someone has to read it, understand what it actually implies, and decide what to test next, and that someone is usually the busiest analyst on the team, which is why the backlog of "obvious" follow-up tests that never got run is longer than anyone wants to admit.

Synthetic data only

Invented experiments, invented metrics, invented conversion numbers. Nothing that resembles a real customer's experiment data or a real business result goes into a shared account today.

Three agent use cases​

1. Experiment-analysis agent — the seed use case​

  • Trigger. An experiment reaches statistical readiness — enough samples collected, or a pre-agreed time box elapsed. The agent notices on its own; nobody opens a dashboard to check.
  • What the agent decides. Whether the result is significant or still inconclusive, what it implies about the underlying hypothesis, and what the natural next test is — the same win tried on a different surface, the same null result tried with a different lever, or a segment worth splitting out on its own.
  • What it does — the tools. analyze_experiment interprets the result and drafts the implication in plain language. design_followup proposes the next experiment's hypothesis, variant, and target metric, grounded in what this experiment and its own predecessors already showed.
  • Human in the loop? Reading and interpreting needs no approval — that is pure analysis. Launching the follow-up test does, without exception, because it spends real traffic and real time against a metric the business is watching, and a bad follow-up hypothesis costs weeks, not minutes.

2. Under-powered-test flagger​

  • Trigger. An experiment is running and partway through its planned duration.
  • What the agent decides. Whether the current traffic allocation and effect size make it likely the test will ever reach a real answer, or whether it is quietly heading for an inconclusive result that wastes the rest of its time box.
  • What it does — the tool. check_power estimates whether the test is on track and, if not, proposes a specific fix — more traffic, a longer window, or a bigger minimum detectable effect.
  • Human in the loop? Only on the fix. Flagging is informational and safe. Changing a live experiment's traffic allocation or duration is a decision an experiment owner makes, because it affects everyone else's tests sharing the same traffic pool.

3. Cross-experiment conflict detector​

  • Trigger. A new experiment is proposed that targets a page or flow another experiment is already running against.
  • What the agent decides. Whether the two experiments would interact — same page, overlapping audience, metrics that could contaminate each other — and how serious the overlap is.
  • What it does — the tool. check_conflicts compares the proposed experiment against every currently running one and returns a specific list of what overlaps and why it matters.
  • Human in the loop? Yes, as a launch reviewer's decision. The agent never blocks a launch — it makes sure the person who can decide whether an overlap is acceptable actually sees it before go-live, not after the data is already contaminated.

The 60-minute cut​

Build use case 1. Cut it to: notice on a clock, analyse with memory, gate the launch.

In scope:

  1. A hardcoded array of 4 synthetic experiments, each with an id, a hypothesis, a variant description, a target metric, sample counts for control and variant, and conversion counts. Make one a clear win, one a clear loss, one inconclusive, and one a win specific to a segment.
  2. A scheduled check using the Agents SDK this.schedule() that looks for experiments whose sample count has crossed a threshold you pick. Set it to every 60 seconds so it fires during your demo, and say out loud that production would check hourly.
  3. One tool that interprets the result and one that drafts a follow-up proposal, stored together as this experiment's analysis.
  4. A Workflow that proposes the follow-up and pauses on step.waitForEvent() before it is considered "launched."
  5. A one-page view: every experiment, its verdict, its proposed follow-up, and its approval state.

Out of scope, and say so: a real statistics engine, real traffic splitting, integration with an actual experimentation platform, authentication, and multi-armed bandits. If you are implementing a significance test from scratch, you have lost the hour — a Workers AI model reasoning over the summary numbers is honest for sixty minutes.

The demo that wins. Let the schedule fire on stage and watch an experiment cross the readiness threshold on its own. Show the agent's interpretation citing this experiment's own prior iteration — "the last test on this flow showed X, this one confirms it for mobile" — then show the follow-up sitting in PENDING APPROVAL until you launch it live.

Primitives — exactly three​

  1. Schedules (this.schedule() on the Agents SDK Agent class). An experiment becomes ready on its own clock — enough samples, or a time box — not because someone pushes an event. Scheduling from inside the agent keeps the readiness check next to the experiment history it needs to interpret the result correctly.
  2. AI Search over your own documents. A follow-up hypothesis is only useful if it isn't a test you already ran. Grounding design_followup in an indexed archive of past experiment write-ups is what stops the agent from confidently proposing something the team tried and rejected two quarters ago — a larger model with a longer prompt cannot cite a test it was never shown.
  3. Workflows. Launching a follow-up spends real traffic and real time, and it must be exactly-once — a retry must not launch the same test twice. step.waitForEvent() holds the proposal open until an experiment owner approves it, with no cost while it waits.

Skip Browser Rendering here — there is nothing external to render, the whole loop reasons over your own experiment numbers and your own document archive. Skip persistent Durable Object state as a named primitive — you get it for free by building on the Agent class to hold each experiment's own history; spend your third choice on grounding the follow-up in your archive instead.

Paste-ready starter prompt​

Scaffold the starter first:

npm create cloudflare@latest -- team-platform-experiment-followup --template cloudflare/agents-starter
cd team-platform-experiment-followup

Then paste this into your coding agent:

I'm working in a fresh clone of Cloudflare's agents-starter template. The agent code is in
src/server.ts and the React client is in src/client.tsx. I have 60 minutes. Keep the
existing chat agent working — extend it, don't replace it.

Build an experiment-analysis agent that designs its own follow-up test and never launches
one without approval.

1. Add a hardcoded array of 4 synthetic experiments to src/server.ts. Each has: experimentId,
   hypothesis, variantDescription, targetMetric, controlSamples, controlConversions,
   variantSamples, variantConversions, and a readySampleThreshold. Make one a clear win, one
   a clear loss, one inconclusive (samples below threshold), and one a win with a note that
   it is segment-specific. Invented data only.

2. Add a hardcoded array in a separate file of 3 synthetic "past experiment" summaries — id,
   hypothesis, and outcome — that a real archive would hold. Invented content only.

3. Add a tool called analyze_experiment. Given an experimentId, if combined samples are below
   readySampleThreshold, return status "not_ready" and stop. Otherwise use a Workers AI model
   to return JSON: verdict (win, loss, inconclusive), liftPercent, confidence (low, medium,
   high), and a two-sentence plain-language interpretation.

4. Add a tool called design_followup. Given an experimentId and its analysis, compare against
   the past-experiment archive and use a Workers AI model to return JSON: proposedHypothesis,
   proposedVariant, targetMetric, and a citation naming which past experiment (if any)
   informed the choice, or null if none did. Never fabricate a citation to an experiment not
   in the archive.

5. Use this.schedule() to run a sweep every 60 seconds. For each experiment not yet analysed
   whose combined samples exceed readySampleThreshold, run analyze_experiment then
   design_followup, and store both keyed by experimentId in the Agent's SQLite storage via
   this.sql, updating rather than duplicating on re-runs.

6. Add a Cloudflare Workflow called FollowupLaunch in a new file, with steps as step.do():
     a) propose — record the proposed follow-up
     b) await_approval — step.waitForEvent() with a generous timeout, waiting for an event of
        type "followup-decision"
     c) record — record whether the follow-up was launched or declined, and by whom
   Import WorkflowEntrypoint, WorkflowStep and WorkflowEvent from "cloudflare:workers", and
   add the workflow binding to wrangler.jsonc.

7. Add routes:
     POST /launch/:experimentId   starts a FollowupLaunch workflow for that experiment's
                                   proposal
     POST /decide/:experimentId   sends the "followup-decision" event with approved true or
                                   false
     GET  /experiments            HTML page: every experiment, its verdict, its proposed
                                   follow-up and citation, and its launch approval state

Requirements: TypeScript. Route every Workers AI call through AI Gateway by passing
{ gateway: { id: "agenthack" } } as the options argument. Run `npx wrangler types` after
changing wrangler.jsonc. Then deploy with `npx wrangler deploy` and give me curl commands
that show the schedule firing and one experiment going through propose, decide, and record.

The Worker name team-platform-experiment-followup follows the shared-account rule <team_prefix>-<slug>. Eight teams deploy into one account today, so if the Scoping agent issued you a different prefix, use that one instead.

What "production" looks like​

  1. Replace the hardcoded numbers with a real statistics engine. Sequential testing and proper confidence intervals are a solved problem elsewhere — the agent's job is to interpret and propose, not to compute significance from scratch, so wire it to whatever engine already backs your platform's real results.
  2. Index your actual experiment archive, not three invented entries. The value of design_followup scales directly with how much real prior history it can cite — start with the last year of write-ups and measure how often engineers accept the proposed follow-up as-is versus rewrite it.
  3. Track the acceptance rate of proposed follow-ups over time. That number tells you precisely how much of the analyst's job this agent is actually doing versus how much it is generating noise for someone to reject — and it is the number that justifies widening its autonomy, or doesn't.

Next: the Level Up page explains each primitive, and the cheat sheet has the snippets. Both are in the sidebar.