When running experiments, you can create lists of users called "holdouts" who will be excluded. You can keep these same users out of either specific experiments or all of your testing entirely.
To create a new holdout:
- Navigate to Experiments and click on the Holdout groups tab.
- Click New holdout.
- Enter a descriptive name for your holdout, fill out the optional description to help you remember its purpose, and set the percentage of users you want to include in the holdout.
The holdout should be large enough to be statistically significant, but small enough as to not significantly reduce the pool of users available for experiments. We recommend between 1% and 10%.
Once created, you can assign the holdout to new or existing experiments. The holdout becomes locked once any experiment using it is launched and you will be unable to modify it. This is to prevent data inconsistencies that could affect results.
When assigned to an experiment, your holdout will be treated as another variant in experiment analysis. You'll be able to see the conversion rate, delta %, credible interval, and win probability alongside other experiment variants.
