Winsorization

Winsorization is a common technique for removing noise in experiment results, specifically from outliers.

Winsorization refers to the practice of measuring the percentile Px of a metric and setting all values over Px to Px.

Statsig computes the Px value using all non-zero and non-null unit-level values of the metric; metrics are aggregated from rows or events, and then the Px’th unit’s value is used as the threshold to adjust other units’ values.

Statsig Cloud

At Statsig, the default percentile for winsorization is 99.9%. This reduces the influence of extreme outliers caused by factors such as logging errors or bad actors.

Winsorization is applied to sum, event count, mean, ratio, and funnel metrics, including imported metrics. Winsorization will not be applied to Participation or User Accounting metrics.

Warehouse Native

Statsig Warehouse Native lets you configure this per metric - and choose explicitly the upper and/or lower bounds to apply.

Winsorization is applied to to sum, event count, mean and ratio metrics.

Metric Capping

This is a very simple, but effective technique to handle outliers. With this capability, you can define max values for a metric for whatever unit type(s) are configured for this metric. Any value surpassing the set cap will automatically be adjusted downward to match it.

For instance, if you determine that purchases greater than $10,000 per day on your E-commerce platform should not skew analysis, any transaction exceeding this threshold will be adjusted downward to this limit, ensuring the integrity of your experiment analysis. Capped metrics are available for Event Count and Aggregation (sum) metric types.