Name: launchdarkly-experiment-setup
Author: launchdarkly

Set up and run experiments in LaunchDarkly. Create experiments with metrics, treatments, and flag config, start iterations to collect data, swap design between iterations, and stop with a winner.

LaunchDarkly Experiment Setup You're using a skill that guides you through setting up and running experiments in LaunchDarkly. Your job is to design the experiment, create it with the right metrics, treatments, and flag config, start data collection, evolve the design between iterations when needed, and stop with a winner. Never create a metric or experiment until the user approves a complete design, never start an iteration until the user says to start, and never change a flag unless the user asks. See Before you create anything for required pieces, suggestions, and labels. See the workflow for gathering and approval, Step 1 for the event check, and Step 4 for starting. Prerequisites This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment. Required MCP tools: create experiment — create a new experiment with its initial iteration (hypothesis, metrics, treatments, flag config). start experiment iteration — begin collecting data for an experiment's current draft iteration. get experiment — check experiment status, treatments, metrics, and current iteration. Optional MCP tools: list experiments — browse existing experiments in the project. update experiment — update fields on the experiment or its current iteration. Honours mutableFieldsByStatus , so what's editable depends on whether the iteration is not started , running , or stopped . Returns rejected inputs under skipped . save and start experiment iteration — the API recommended way to change locked fields on a running experiment. Stops the current iteration, creates a new draft with the supplied field updates, and starts it in one call. stop experiment iteration — stop the running iteration. You must declare a winner: pass the winningTreatmentId (and a winningReason ). If no variation outperformed, pick the baseline/control as the winner. list metrics , create metric , list metric events — manage metrics referenced by the experiment. get flag — read one flag's targeting in an environment: whether it is on, the variation a rule serves, its off variation, and whether that rule is a percentage rollout. list flags — search flags in a project by name or key. create flag — create a feature flag in a project. It is off in every environment until turned on. get flag status across envs — read a flag's status in each environment ( new , active , launched , or inactive ). search contexts — find stored contexts in an environment by kind, to see which context kinds are in use. Core Concepts What Are Experiments? Experiments in LaunchDarkly measure the impact of feature flag variations on key metrics. An experiment consists of: Treatments : the flag variations being compared (control vs. test). Each treatment has an allocationPercent ; the values across treatments should sum to 100. Metrics : what you're measuring (conversion rate, latency, revenue, etc.). One must be the primary metric. Flag config : the flagKey , ruleId , and flagConfigVersion of the targeting rule that drives the experiment. Iteration : a single data collection window. Created in not started status, becomes running when started, transitions to stopped when ended. Holdout (optional): a project level group of users excluded from the experiment for baseline measurement ( holdoutId ). Experiment Lifecycle 1. Create the experiment with its first iteration ( create experiment ). 2. Start the iteration to begin data collection ( start experiment iteration ), only after the user explicitly says to start. 3. Monitor results as data accumulates ( get experiment ). 4. Evolve the design mid experiment if needed — change locked fields like treatments , metrics , or methodology by calling save and start experiment iteration , which stops the current iteration, creates a new draft with your changes, and starts it. That call starts data collection, so make it only after the user explicitly says to start. 5. Stop the iteration when you have a winner or a clear call ( stop experiment iteration ). 6. Ship the winning variation. Core Principles 1. Metrics first : ensure the metrics you'll reference exist before creating the experiment. 2. Clear hypothesis : every iteration requires a hypothesis string from the user. See Before you create anything. 3. Proper controls : exactly one treatment must have baseline: true . 4. Sufficient sample size : let iterations run long enough to support a decision. See Step 7. 5. One change at a time : test one variable per experiment for clear attribution. Before you create anything These three must come from the user. Never fill them in: Flag and variations : which flag to test, and which of its variations are in the experiment. Hypothesis : what they expect to improve, and by how much if they know. Primary metric : the single metric or funnel that decides the result. If any of these three are missing, leave them blank until the user supplies them. They go in the gather message. See the workflow. Once the user names the flag, see Step 2. Control variation : which of those variations is the baseline ( baseline: true ). If the user named it, use that. If they did not: Ask, and leave it blank, when the flag is brand new, the targeted rule serves a percentage rollout, or the flag is off and get flag status across envs reports that environment as inactive . A flag is brand new if you created it in this conversation, or get flag status across envs reports that environment as new : the flag was created fewer than seven days ago and has never been evaluated there. A percentage rollout is a targeted rule that splits traffic across variations instead of serving one. get flag shows which. inactive means the flag has not been evaluated for at least seven days, so no one is currently being served the off variation. Otherwise suggest one. If the flag is on, suggest the variation the targeted rule serves today. If the flag is off, suggest its off variation. Randomization unit : the context kind the experiment randomizes on. If the user named it, use that. If they did not, call search contexts for this project and environment. Suggest user only when a kind of user returns contexts and filter kind notEquals "user" returns totalCount 0, so user is the only context kind in use. If search contexts is unavailable, returns no contexts, or any other kind is stored, ask and leave it blank. Label every suggested value in the design as suggested, with a one line reason, for example Control (suggested, what users see today): Start free trial . The user can change any suggestion before approving. Approving the design accepts the suggestions still shown in it. Settings the user did not mention can stay on LaunchDarkly's defaults. List each default you are using in the design: Traffic split : equal allocationPercent across the variations the user named (two variations is 50/50). The values must sum to 100. Statistics : LaunchDarkly default (frequentist, sequential testing). Do not send methodology unless the user chooses one. If they choose one, send that value and show it here instead of this default. A Bayesian choice is methodology "bayesian" . Data source : dataSource "launchdarkly" . Reshuffle : canReshuffleTraffic true . Leave analysisConfig unset unless the user set it, and say that in the design. The randomization unit is never one of these defaults. Workflow If nothing is missing, skip the gather message and go straight to the design. Otherwise gather every missing piece in one message before you show any design, then wait: the flag and variations, the hypothesis, or the primary metric, when one is missing the named flag, when get flag cannot find it: say so and offer three choices in one message: correct the key (list similar flags from list flags ); create the flag themselves, then tell you to continue; or have you create it. For that last choice, propose the key, name, type, and variations from their design, say it will be created off, and wait for an explicit yes before calling create flag . If launchdarkly flag create is installed, follow it for that step. After the flag exists, continue setup. Never create a flag unless they pick that choice and approve the details. the control or the randomization unit, when the rules above say to ask instead of suggest for a new primary metric, its kind ( custom , pageview , or click ), what it is based on (event key, URL match, or click selector), and how it is measured (share of units who converted, a total count, or an average), when the user has not already made that clear list metrics and get flag are fine while you gather this. create metric and create experiment are not. If the user's wording already makes one of these clear, do not ask about it. "The share of users who sent the cta clicked event" already names the measure, the kind, and the event key. Show the design only when nothing is blank. Never show a design with a missing value. Include every piece gathered above, the other treatments, flag config ( flagKey , ruleId , flagConfigVersion ), and every default you are applying. Statistics reads LaunchDarkly default (frequentist, sequential testing) unless the user chose a methodology, in which case show their choice. Label each suggested value. See Before you create anything. STOP HERE until the user approves the design. Create the metric and the experiment only after the user explicitly approves a complete design they have seen. If the user changes anything after seeing it, show the updated design and ask once more. Step 1: Prepare Metrics 1. Use list metrics to find existing metrics. The user names the primary metric. 2. If it already exists, keep its name and key for the design review. 3. If it does not exist, a miss only means the metric is new. The user has often named the metric and not its kind or event key. Gather those before the design. Until the user has said the kind and, for a custom event, the event key: Do not call list metric events . Do not check the metric name, or any other key the user did not give, as the event key. Do not skip the flow check by treating the metric as pageview or click . Do not block creation for an event you have not identified. 4. See the gather rule for a new metric's kind, what it is based on, and how it is measured. 5. After the user has said the new metric is custom and named its event key, call list metric events for that event key. Either way, the metric goes in the design once nothing else is blank. If the event is not reaching LaunchDarkly, include a warning in that design: the event has not arrived in that environment yet. List at most 10 custom event keys list metric events returns for that environment, preferring keys similar to the one the user gave, so a typo is easy to spot. Say the user can correct the event key, or approve as is if the tracking is not deployed yet. If they correct the event key, that is a change: call list metric events again for the new key and show the updated design, per the approval rule. On approval, create the metric and the experiment as normal, even if the event has not arrived. 6. Pageview and click metrics have no custom event key. Skip list metric events only after the user has said the kind is pageview or click . 7. Pass the user named primary metric as primarySingleMetricKey or primaryFunnelKey . Goal Metric type Example key Conversion Custom conversion checkout completed Performance Custom numeric page load time ms Engagement Custom conversion feature clicked Revenue Custom numeric order value Step 2: Identify the Targeting Rule You need the ruleId and current flagConfigVersion of the flag rule that will drive the e