I randomly split eligible users into two groups.
I define a clear hypothesis for the experiment.
I check statistical uncertainty and guardrail metrics before deciding.
More conversions don't automatically mean a winning experiment.
more time spent on an app isn’t always a good thing if users are struggling to complete their tasks.
the single key metric that best captures the core value your product delivers to users and helps guide long‑term product decisions.
I use A/B testing to move product decisions away from pure opinion and into data‑driven insight, especially for high‑impact UI tweaks like button copy on the signup page.
My hypothesis is: Changing the button copy to “Get Started” will increase signup conversion.
Variant A shows the button labeled “Sign Up”.
Variant B shows the button labeled “Get Started”.
- Primary metric: signup conversion rate.
After measuring conversion and uncertainty, I decide whether to roll out, iterate, or stop the experiment.
More conversions don’t automatically mean a winning experiment; sample size, duration, statistical significance, and overall user experience must be evaluated.
- Guardrail metrics: any usability or satisfaction signals that could decline.
I introduced the concept of a North Star Metric (NSM) as the single key metric that best captures the core value my product delivers to users and serves as the primary decision‑filter for long‑term product strategy.
I adopt the NSM as the primary guiding metric for product teams to ensure alignment around user value.
I keep additional metrics to monitor side effects—trust, retention, task success—because the NSM alone can be gamed.
Example NSM: Netflix → Hours of content watched.
Example NSM: Spotify → Listening engagement.
Example NSM: Airbnb → Nights booked.
Example NSM: Duolingo → Active learners who keep learning.