If computed the relative rejection frequency, how to measure if significantly different from significance levels? (Normality tests in R)

Question

professionals and students,

I have significance levels 10%,5% & 1% and I have computed the relative rejection frequency thanks to an answer on my previous question.

replicate_sw10 = replicate(1000,shapiro.test(rnorm(10)))
table(replicate_sw10["p.value",]<0.10)/1000

> FALSE  TRUE 
> 0.909 0.091

But if I have done this for various sample sizes (T=10,30,50,100,500) and stored it manually via excel. Maybe there is an ever easier way to compute this in a function/list.

However how do I measure if it significantly different from significance levels? (The hint is the following: the rejection of a test can be modelled as a Bernoulli random variable)

Best regards

Answer 1

So, the easiest way to do this is.. so if you perform 1000 test, you would expect approximately 0.1 of your test to have a pvalue < 0.1. It's like a bernoulli trial like you said, and you can use a binomial test to see the probability of something as extreme as your result:

set.seed(100)
replicate_sw10 = replicate(1000,shapiro.test(rnorm(10)))
obs_significant = sum(replicate_sw10["p.value",]<0.1)

binom.test(obs_significant,n=1000,p=0.1)

    Exact binomial test

data:  obs_significant and 1000
number of successes = 118, number of trials = 1000, p-value = 0.06479
alternative hypothesis: true probability of success is not equal to 0.1
95 percent confidence interval:
 0.09865252 0.13962772
sample estimates:
probability of success 
                 0.118