Z Test of Proportions - Solve for minimum proportion to reach significance

Question

The following equation provides the test statistic used in the Test of Proportions.

For a proposed a/b test, I am attempting to show the minimum value needed for the treated group (p2) to show a statistical significance at a 95% confidence level. In other words, I am trying to solve this equation for p2. Given that I know my total population size, the percent that will be treated, and Z-value, this would seem to be straightforward. However, I am getting stuck on the algebra.

I have written an R script that will run through a range of values for p2 until a p-value of a given confidence is met, but that is a sloppy way of solving the problem.

Answer 1

I wouldn't bother doing this algebraically (or however that's spelled).

Notice that if

Z = diff(p) / se(p)

then

0 = diff(p) / se(p) - Z

The uniroot function can then do the work for you. you provide the values of everything but p2, and uniroot will seek out the value that resolves to 0.

zdiff <- function(p2, p1, n1, n2, alpha = 0.025) 
{
  ((p1 - p2) - 0) / sqrt(p1 * (1-p1) / n1 + p2 * (1-p2) / n2) - qnorm(alpha, lower.tail = FALSE)
}

uniroot(f = zdiff,
        p1 = .5, n1 = 50, n2= 50, 
        interval = c(0, 1))

$root
[1] 0.311125

$f.root
[1] -1.546283e-06

$iter
[1] 5

$init.it
[1] NA

$estim.prec
[1] 6.103516e-05

So with equal sample sizes of 50 and p1 = .5, p2 would have to be less than 0.311125 to generate a statistically significant result at the two-sides alpha = 0.05 level.