If you use exploratory analyses to generate hypotheses, be sure to test those hypotheses on data sets other than the one used for exploratory analysis. If not, you may simply be 'massaging' the data for a (probably false) signal.

If using data transformations or discarding data, ensure that there is solid rationale to do so. As indicated, the Bonferroni and Sidak tests are pretty easy to calculate manually.'Data dredging' (sometimes called 'data fishing') is a real risk which may invalidate any conclusions you draw from your analysis.Įxploratory analyses are used to find subsets of data that confirm (or are more likely to confirm) an a priori hypothesis which may not be generalisable to the whole (statistical) population.Įxploratory analyses are used to generate a hypothesis from a given data set which is tested using the same data set.Įvaluate if your data supports the results of a hypothesis based on previous knowledge and research. There is an extensive literature on all the terms and tests used above, including good material at Wikipedia. As you can tell the Sidak test makes very little difference vs. In this same example it would be calculated as follows: 1 -(1-5%)^(1/10) = 0.51%. There is a second method that uses some compounding and is technically fractionally more accurate called the Sidak test. This adjustment is called the Bonferroni test. Now you adjust your p-value alpha threshold as follows: 5%/10 = 0.5%.

So, let's say you initially started with a p-value alpha threshold of 5% and you are testing 10 hypothesis. It consists in adjusting your p-value alpha thresholds to reflect the number of hypotheses you are testing. The second solution is actually a lot easier. This means it is somewhat less restrictive than a true Post Hoc test because in this case, it factors that among your 20 different hypotheses there may be just a few that you were truly more focused upon from the start. Dunnett test is also called a Planned Comparison test. The first solution is to use a Post Hoc test like Tukey's Honestly Significant Difference (HSD) test or Dunnett test. One of the hypotheses will be confirmed at the 0.05 p-value level.

If you test 20 different hypotheses simultaneously, there is a pretty high likelihood that you will have a false positive.

What you call data dredging is a very well known and tricky issue in statistics.

Data dredging is ok