Outliers are often excluded from the analysis with a Outlier filtering/detection such as do A/B testing platforms handle outliers, example:
Dynamic Yield: