It is now possible to use a statistical significance test to assess whether a Necessary Condition observed in a dataset is based on random chance or not.
Necessary Condition Analysis (NCA) can be used to determine if a certain condition is necessary but not sufficient for a certain outcome. For example, intelligence is necessary for creativity (Karwowski et al., 2016) and trust is necessary for successful buyer-supplier collaborations (Van der Valk et al., 2016).
The strength of a necessary condition is captured by the necessity effect size (Dul, 2016). Its calculation is based on a scatterplot of two variables (X and Y) in which a ceiling line is drawn on top of the data in the upper left corner of the graph (see the figure below). The line separates the area with observations from the area without them. The larger the area without observations compared to the area with observations, the larger the effect size and the stronger the necessary condition.
Just like the effect sizes generated by other analytical tools—such as regression analysis or structural equation modeling—we need a test to check whether the necessity effect we find is real or based on random chance. Until now, it was not possible to conduct such a test. In a recent paper, however, Jan Dul, Erwin van der Laan, and Roelof Kuik introduce a permutation test that makes it possible to test the statistical significance of the NCA effect size.
The Permutation Test
The permutation test reshuffles the values of X and Y to create a set of random samples in which X and Y are unrelated to each other. It simulates the distribution of the null hypothesis that X is not a necessary condition for Y. To test the alternative hypothesis (X is a necessary condition for Y) we can compare the observed effect size with the distribution of random effect sizes in which X and Y are unrelated.
The p-value is the probability that the observed effect size is equal or larger than the random effect size. In other words, it is the probability that the observed effect size is due to random chance. If that probability is small—for example less than 5% (p < 0.05)—we cannot reject the null-hypothesis that X and Y are unrelated. The alternative hypothesis—X is necessary but not sufficient for Y—is supported.
Steps in the Permutation Test
The permutation test consists of five steps (Dul, van der Laan, & Kuik, 2018):
- Calculate the NCA effect size for your sample.
- Formulate the null-hypothesis (X and Y are not related).
- Create a large set of random resamples (e.g., 10.000) using the permutations of X and Y.
- Calculate the effect size of all the resamples.
- Compare the observed effect size with the distribution of random effect sizes.
The test will be part of version 3.0 of the NCA R-package. More information about the permutation test and its implementation can be found here.