In general, if we have k independent significance tests at the α level, the probability p that we will get no significant differences in all these tests is simply the product of the individual probabilities: (1 - α)k. For example, with α = 0.05 and k = 10 we get p = 0.9510 = 0.60. This means, however, we now have a 40% chance that one of these 10 tests will turn out significant, despite each individual test only being at the 5% level. In order to guarantee that the overall significance test is still at the α level, we have to adapt the significance level α′ of the individual test.
This results in the following relation between the overall and the individual significance level:
|(1 - α′)k = 1 - α.|
This equation can easily be solved for α′:
|α′ = 1 - (1-α)1/k,|
which for small α reduces to:
|α′ = α / k|
This is a very simple recipe: If you want an overall significance level α and you perform k individual tests, simply divide α by k to obtain the significance level for the individual tests.
© djmw, November 7, 2001