Sample Size

#since here the variable is a proportion, we can use power.prop.test 
power.prop.test(
                p1=0.1, #this is the conversion rate of the site prior to running the test. Aka control conversion rate
                p2=0.11, #this is the minimum conversion rate of the test that would make it worth it to make the change
                power=0.8, sig.level = 0.05 # these are power and significance level
                )

     Two-sample comparison of proportions power calculation 

              n = 14750.79
             p1 = 0.1
             p2 = 0.11
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

NOTE: n is number in *each* group

The output is pretty self-explanatory:

What we are looking for is the value of n, all others are parameters we put as input into the function. So this is telling us that we need ~15K in both test and control for us to be able to detect an increase of at least 1% in our metric.

Note that you can also pass a vector and see how sample size would change with different values for p2 (our test minimum conversion rate to make the change). You can then show that to the product manager to help them choose the best minimum effect. Something like this:

require(ggplot2)
#possible p2 values. We choose from 10.5% to 15% with 0.5% increments
possible_p2 = seq(.105, .15, 0.005)
possible_p2

 [1] 0.105 0.110 0.115 0.120 0.125 0.130 0.135 0.140 0.145 0.150

#now let's estimate sample size for all those values and plot them
sample_size = c()
for (i in 1:length(possible_p2))
{ ppt = power.prop.test(
                p1=0.1, #this is the conversion rate of the site prior to running the test. Aka control conversion rate
                p2=possible_p2[i], #this is the minimum conversion rate of the test that would make it worth it to make the change
                power=0.8, sig.level = 0.05 # these are power and significance level
                )
sample_size[i] = ppt$n
}
qplot(sample_size, possible_p2, geom="line")

Here you can see very well that, as we increase p2, the required sample size goes down quickly. It makes sense. If we are looking for a very small gain, we need a lot of data to find it. But if we are looking for a huge gain like from 10% to 15% conversion rate, few data will be enough to check if that’s happening or not.

Finally, now that you know how many people you need in test and control, you can figure out for how many days you need to run the test based on your daily traffic.

A widely used approach to this is:

• If you need more than 2 weeks to collect that many people, just split the traffic 50/50 and run it until you have the required sample size
  
  

• Otherwise, still run it for two weeks to be able to reliably capture weekly patterns. But test it on the smallest percentage of users that would allow to collect enough data in 2 weeks
  
  

Example:

1. Let’s assume our site has 1K visitors per day. We found out we need ~15K users in both test and control to detect an improvement of at least +1%. So we would split the traffic 50/50 for test and control. Each day we would have 500 users in test and 500 in control. And we would run the test for 1 month, i.e. 30*500 = 15K
   
   

2. Let’s assume our site has 100K visitors per day. By splitting test and control 50/50, we would get all users we need in 1 day. But that would be too noisy, Sunday is different from Monday. And if we ran it for two weeks with a 50/50 split, we would collect way more data than we need. And that’s also not efficient, after all we could use that traffic to test other things. So we do something like this:
   
   
   • We want to have 15K people in two weeks. That means ~1K people per day. Since we have 100K daily visitors, 1% of our users will enter the test and see the different version of the site. Because of this, large companies typically run tests on 1% or less of their users