I built the Test Confidence Calculator to do just that. You an plug in your numbers and run “what if” type analysis to determine what kind of volume and CR you need to reach statistical significance. http://www.keystonesolutions.com/community/2011/05/optimization-test-confidence-calculator/

Hope you find this helpful.

-jason

Great page, but I found myself totally lost right at the point all the reading through formulas was supposed to pay off! I’m trying to find a way to estimate how many Visitors I’ll need to get significant results if the trends from the last two months continue – that is, I have lots of data, but no clear winner between a couple of different experiences.

Can anyone spell out for me what this last formula is saying:

Student’s T-Test = 1 – TDIST(ABS(0.84),(4003 + 4008 -2),2) = 0.60

?

Is there any way to say (sticking with your example) if the CR’s stay at approximately the same level they have been, how many visits would one need to be confident in the differences that one is seeing?

Thanks for any pointers.

Tom

Love worked examples like this, but I would just point that you are starting off with a binomial distribution (conversion rate) but then use a normal distribution for some other calculations (T-Test and confidence intervals). This is totally fine as the normal distribution does approximate the binomial for large number of trials and conversion rates that are not too close to 0 or 1.

The takeaway is just that you should not jump to conclusions about the control and alternative until you have a decent number of trials.

The way variance was written tripped me up at first though.

Variance (control) = 9.41(1 – 9.41) = 0.09

would probably be more clearly (and accurately) written as

Variance (control) = .0941(1 – .0941) = 0.09

Otherwise, I think I could easily provide this info in a group setting to a varied group which I think makes it very valuable!

Thanks

I agree with your reply to Whitney–to increase confidence (and statistical power), you need to increase the sample size.

–Jeff

Thanks for the comment. To be clear, this isn’t my methodology. This is the underlying set of calculations that Omniture Test & Target uses within their reporting interface.

I think we can both benefit from talking to each other. Nice post. I’m not on board with how you come to “confidence”, but everyone’s got a different angle.

Cheers,
Jorge

Specifically – can you explain how the LIFT confidence intervals are reported? ie. if it’s showing a mean lift of 3% but the little hover option is showing -2% to 8% – how are they getting the -2% – 8% range? From our calcuations, they are taking the test group high bound less control group low bound for the high end and the opposite for the low end.

But – how exactly is that statistical confidence? I need to be able to say “We are seeing a 3% lift +/- X with 95% confidence (or whatever). Question is – how do we get to 3% +/- X% using T&T’s data? Right now, we are forced to calculate outside of T&T using daily aggregates – not ideal. But T&T’s lift intervals are just too wide and do not seem representative of the true error of the lift.

But, I dont think you can reach a higher level of significance by running the test for more days.

It all depends on the data you get. I mean, even after running it for more days, lets say the t-statistic comes out to be the same still, there’s nothing much statistics can do it.