Increasing the number of permutations in a statistical test can actually make the final results less powerful.
Monte Carlo permutation tests exhibit non-monotonicity, meaning more samples do not always lead to more reliable power. Scientists have long operated under the belief that more data always improves the accuracy of a study. This mathematical discovery proves that adding permutations can sometimes decrease the chance of finding a true effect. It highlights a dangerous trap for researchers who try to fix weak results by simply running more simulations. Precision in science depends more on the structure of the test than the sheer volume of data processed.
More Permutations Do Not Always Increase Power: Non-monotonicity in Monte Carlo Permutation Tests
arXiv · 2605.03886
Monte Carlo permutation tests are a cornerstone of valid, model-free statistical inference. A widely held practical intuition is that increasing the number of sampled permutations improves test performance, in particular that statistical power tends to increase with the Monte Carlo budget. In this paper, we show that these intuitions are false in general. Leveraging the saw-toothed structure of power arising from distributional discreteness, we provide a simple structural explanation for why pow