Search code examples
rstatab-testingabtest

What should the default power be in power.prop.test?


Is there an industry standard for the power section of the power.prop.test?

I am using the function to find p2 but not sure what is the standard for power.

power.prop.test(
  n= 6289195,
  p1=0.004, 
  power=0.8, 
  sig.level=0.05, 
  tol=.Machine$double.eps^.8)

For example, should it be 0.8 or 0.9?


Solution

  • This is a practical statistics question rather than an R question, but a power of 0.8, i.e. 80%, is common. Since it is common (rather like 95% confidence), people think they understand what it is saying and do not query its choice as much as they might another values.

    You need to remember that it is an arbitrary target: if you changed it in your example, then the main impact would be to give you a different result for p2. Really you should be explicitly balancing the cost of increasing the sample size with against the different costs of Type I and particular Type II errors

    A common reference is to Cohen J. (1988). Statistical Power Analysis for the Behavioral Sciences. New York, NY: Routledge Academic, section 2.4, which says:

    It is proposed here as a convention that, when the investigator has no other basis for setting the desired power value, the value .80 be used. This means that b is set at .20. This arbitrary but reasonable value is offered for several reasons (Cohen, 1965, pp. 98-99). The chief among them takes into consideration the implicit convention for a of .05. The b of .20 is chosen with the idea that the general relative seriousness of these two kinds of errors is of the order of .20/.05, i.e., that Type I errors are of the order of four times as serious as Type II errors. This .80 desired power convention is offered with the hope that it will be ignored whenever an investigator can find a basis in his substantive concerns in his specific research investigation to choose a value ad hoc.

    Other examples of 0.8 found in a quick search:

    • The R stats reference page for power.prop.test uses power=0.8 as an example
    • A University of Ottawa medicine page says "A power of 80% is often chosen; hence a true difference will be missed 20% of the time. This is a compromise because raising power to 90% power will require increasing the sample size by about 30%"
    • The Statistics Done Wrong site and book says "A scientist might want to know how many patients are needed to test if a new medication improves survival by more than 10%, and a quick calculation of statistical power would provide the answer. Scientists are usually satisfied when the statistical power is 0.8 or higher, corresponding to an 80% chance of concluding there’s a real effect. However, few scientists ever perform this calculation, and few journal articles ever mention the statistical power of their tests."