Roblox adds early warnings for harmful experiments
Five-minute updates can flag serious problems during an experiment’s first day. Use the warnings to consider stopping; daily results still guide rollout.
Roblox has launched Early Harm Detection for experiments. During the first 24 hours, or until daily results arrive, playtime, average revenue per user and payer conversion update every five minutes. Open the experiment’s Metrics page in Creator Hub to inspect them. The October 1 announcement describes an earlier chance to catch damaging changes.
What triggers a warning?
The entire confidence interval must fall below the harm threshold: −10% for playtime or −20% for average revenue per user or payer conversion. An estimate crossing the threshold is not enough. Below 10,000 enrolled players across a variant and its control, Roblox shows the measured change but makes no harm decision. The documentation explains these limits.
Critical-harm notifications go to the game owner and experiment owner by email and Creator Hub, with an optional webhook. The team decides whether to stop. Incorrect enrollment proportions are also checked every five minutes during the first day, then daily; Roblox recommends stopping and restarting experiments with that warning.
Give the first day an owner
Our recommendation is to assign someone to watch a new experiment and record who can stop it. Before starting, write down the change being tested and what players should experience differently.
Test any shared alert route before relying on it. When a warning arrives, inspect the affected metric, check the variant’s actual gameplay and record the decision. Keep checking player reports and broken interactions even when no warning appears.
Early harm results support stopping decisions, not choosing a winner. Roblox says to use daily results and the planned experiment duration for rollout decisions. A quiet first day is an incomplete observation, especially when too few players have enrolled to trigger a warning.