I've long argued that the biggest problem with orthodox NHST for A/B testing is that you actually don't care about 'significance of ...
I've long argued that the biggest problem with orthodox NHST for A/B testing is that you actually don't care about 'significance of effect' as much as you do 'magnitude of effect'. Furthermore, p-values tell you nothing about the range of possible improvements (or lack thereof) you're facing. Maybe you are willing to risk potential losses for potentially huge gains, or maybe you can't afford to lose a single customer and would rather exchange time for certainty. My favored approach I've outlined here[0]. Where the problem is basically considered one of Bayesian parameter estimation. Benefits include: 1. Output is a range of possible improvements so you can reason about risk/reward for calling a test early. 2. Allows the use of prior information to prevent very early stopping, and provide better estimates early on. 3. Every piece of the testing setup is, imho, easy to understand (ignore this benefit if you can comfortably derive Student's T-distribution from first principles) [0] https://www.countbayesie.com/blog/2015/4/25/bayesian-ab-test...
Posted by CountBayesie on Jun 3, 2015, 5:30 PM
Tags
Related issues
- Linter Product Plan and Progress
- > - Apple: When you start an Airplay stream on your phone directed at an Apple TV, the Apple TV doesn't send a switch input command.
- Thanks for sharing your ideas.
- I wish there was a really good source for analysis on the Greek situation that ignored value judgements of what "should" happen.
Raise real-world problems, find people facing the same issue, and discuss solutions.6,000 issues0 comments
onboarding (EdTech)
Linter Product Plan and Progress
> - Apple: When you start an Airplay stream on your phone directed at an Apple TV, the Apple TV doesn't send a switch input command.
Thanks for sharing your ideas.
I wish there was a really good source for analysis on the Greek situation that ignored value judgements of what "should" happen.
I've long argued that the biggest problem with orthodox NHST for A/B testing is that you actually don't care about 'significance of ...
I've long argued that the biggest problem with orthodox NHST for A/B testing is that you actually don't care about 'significance of ...
I've long argued that the biggest problem with orthodox NHST for A/B testing is that you actually don't care about 'significance of effect' as much as you do 'magnitude of effect'. Furthermore, p-values tell you nothing about the range of possible improvements (or lack thereof) you're facing. Maybe you are willing to risk potential losses for potentially huge gains, or maybe you can't afford to lose a single customer and would rather exchange time for certainty. My favored approach I've outlined here[0]. Where the problem is basically considered one of Bayesian parameter estimation. Benefits include: 1. Output is a range of possible improvements so you can reason about risk/reward for calling a test early. 2. Allows the use of prior information to prevent very early stopping, and provide better estimates early on. 3. Every piece of the testing setup is, imho, easy to understand (ignore this benefit if you can comfortably derive Student's T-distribution from first principles) [0] https://www.countbayesie.com/blog/2015/4/25/bayesian-ab-test...
Comments (0)
No comments yet. Start the discussion.