A/B Test Engines

Open Analytics, pick one project in the project menu (rules cannot be saved under All projects), then the Experiments tab. Under Managed experiment rules:

1. Experiment key: a stable name, such as products-engine-test. Your code sends it on every call.
2. Traffic %: the share of visitors in the test. The rest get no variant.
3. Status: Active. Draft, Paused and Archived assign nobody.
4. Variant 1 and Variant 2: a key (baseline, candidate), the index each one searches, and a weight. The weights must add up to 100.
5. Save rules.

The latest saved rule for a key is the one in force. Name the control variant baseline (or control): the dashboard compares the other variant against it.

Call POST /v1/experiments/assign with the experiment key and a subjectId that stays the same for a visitor: your user id, or an anonymous id you keep in a cookie. The same subject always gets the same variant, even in a new session. A search key is enough, so this can run in the browser.

const res = await fetch("https://headband.dev/v1/experiments/assign", {
  method: "POST",
  headers: { Authorization: `Bearer ${SEARCH_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({
    experimentKey: "products-engine-test",
    subjectId: "user_123",
    sessionId: "sess_456",
  }),
});
const { data } = await res.json(); // experiment answers are wrapped in "data"
const index = data.assigned ? data.indexUid : "products_meili"; // not in the test: baseline

{
  "data": {
    "assigned": true,
    "experimentKey": "products-engine-test",
    "variant": "candidate",
    "indexUid": "products_typesense",
    "sessionId": "sess_456",
    "config": { "status": "active", "trafficPercent": 100, "killSwitch": false, "updatedAt": "..." }
  }
}

When assigned is false, reason says why: outside_rollout (the visitor is not in Traffic %), not_runnable (the rule is not active, or its kill switch is on) or no_weight. Serve your baseline and do not tag the search. An experiment key with no saved rule answers 404 experiment_config_not_found. A project key finds the rule in its own project; an org key must also send projectId or indexUid.

In our run, 40 visitors against a rule with Traffic % 30 and two variants at 50: 11 were assigned (5 baseline, 6 candidate), and each got the same variant when asked again with a new session.

Search the index the assignment named, and add experimentKey, variant and sessionId so Headband counts the search for that variant. Send both experimentKey and variant, or neither. POST /v1/search and POST /v1/ai/search take them; multi-search and hybrid search do not.

await fetch("https://headband.dev/v1/search", {
  method: "POST",
  headers: { Authorization: `Bearer ${SEARCH_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({
    index: data.indexUid,
    q: "running shoes",
    experimentKey: data.experimentKey,
    variant: data.variant,
    sessionId: data.sessionId,
  }),
});

Report what the visitor did with POST /v1/experiments/events. The key needs analytics.write, which every search key has.

async function event(eventType, extra = {}) {
  await fetch("https://headband.dev/v1/experiments/events", {
    method: "POST",
    headers: { Authorization: `Bearer ${SEARCH_KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify({
      eventType,
      experimentKey: data.experimentKey,
      variant: data.variant,
      indexUid: data.indexUid,
      sessionId: data.sessionId,
      ...extra,
    }),
  });
}

await event("experiment_view", { query: "running shoes" });
await event("search_result_click", { query: "running shoes", metadata: { documentId: "42" } });
await event("conversion", { conversionValue: 12900, metadata: { orderId: "ord_789" } });

experimentKey, variant and indexUid are required, and the index must exist. conversionValue is a whole number, such as cents, and metadata an object of at most 4 KB. An accepted event answers 202. Events do not check that a rule exists, so a key with a typo is recorded as its own experiment.

In Analytics, with the project picked, the Experiments tab lists each experiment by variant: searches, events, zero results, average hits, average latency, conversions and conversions per search. Above the table, a recommendation compares the candidate with the baseline:

These are advisory promote/rollback recommendations: Headband never changes the rule or moves traffic by itself. The tab reads the most recent 1,000 analytics events in the period you pick, searches included, so on a busy project it covers only that window. Keep your product analytics as the record for revenue.

• To stop at once, tick "Enable kill switch flag for this experiment" or set Status to Paused, and save. Assignment then answers assigned=false with not_runnable, and your code serves the baseline.
• To ramp up, raise Traffic % and save. Visitors already in the test keep their variant.
• To finish, point your app at the winning index, or switch the baseline index to the winning engine, then archive the rule.

You can skip hosted assignment and split traffic yourself, for example with a hash of the visitor id. Tag searches and send events the same way, with your own variant names; the Experiments tab reports them whether or not a rule exists. To read the saved rule from your code instead, call GET /v1/experiments/config below.

@headband/react 0.2.0 adds assignExperiment for the hosted endpoint, getExperimentConfigs plus deterministic local assignment helpers, and recordExperimentEvent. It is on npm.

The Experiments tab reports searches, zero results, hits, latency and conversions. Track anything else, such as answer acceptance, as a custom event or in your own analytics.