Comparison of Quantiles of Two Groups with Permutation Tests
Zeynel Cebeci, A. Firat Ozdemir, Engin Yildiztepe
5 May 2025
1 Introduction
This vignette provides a comprehensive guide for performing permutation tests to compare quantiles of two groups in R. A permutation test is a non-parametric statistical method that involves rearranging the data points to generate a distribution of a test statistic under the null hypothesis. This technique is particularly useful when the assumptions required for traditional parametric tests are not met, providing a robust alternative for statistical inference.
In experiments involving two groups, it is essential to assess whether there are significant differences between their distributions. Traditional methods might focus on comparing means or medians, but quantile comparison offers a deeper insight into the distributional characteristics of the groups. Quantiles, such as the median (50th percentile) or quartiles (25th and 75th percentiles), divide the data into equal parts and highlight specific points in the dataset that are critical for understanding the distribution’s shape and spread.
This vignette demonstrates how to implement permutation tests for quantile comparison using the permtest function in the package groupcompare. By following the steps outlined, users will be able to conduct thorough statistical analyses and draw meaningful conclusions about their data.
2 Required Packages and Data Sets
2.1 Install and load the package gpcomp
The recent version of the package from CRAN is installed with the following command:
install.packages("groupcompare", dep=TRUE)If you have already installed groupcompare, you can load it into R working environment by using the following command:
library("groupcompare")2.2 Data sets
The dataset to be analyzed can be in wide format where the values for Group 1 and Group 2 are written in two separate columns or in long format where the values are entered in the first column and the group names are entered in the second column. In the following code chunk, a dataset named ds1 is created using the ghdist function to simulate the G&H distribution. The generated dataset contains data for two groups named A and B, each consisting of 25 observations, with a mean of 50 and a standard deviation of 2. In the example, by assigning zeros to the skewness (g) and kurtosis (h) arguments, the simulated data is intended to have a normal distribution. As expected, the means and variances of groups A and B are done equal. In the example, the generated dataset is in wide format, and immediately after, it is converted to long format using the wide2long function to create the dataset ds2. This provides an idea of the long data format, and as can be seen, in the long data format, the first column contains the observation values, while the second column contains the group names or codes. As understood from the example, different groups can be created by changing the means, variances, skewness, and kurtosis parameters.
set.seed(12) # For reproducibility purpose
grp1 <- ghdist(50, 50, 2, g=0, h=0)
grp2 <- ghdist(50, 45, 4, g=0.8, h=0)
ds1 <- data.frame(grp1=grp1, grp2=grp2)
head(ds1)## grp1 grp2
## 1 47.03886 44.83214
## 2 53.15434 44.56903
## 3 48.08651 47.20590
## 4 48.15999 65.17133
## 5 46.00472 42.15702
## 6 49.45541 48.99942 # Data in long format
ds2 <- wide2long(ds1)
head(ds2)## obs group
## 1 47.03886 grp1
## 2 53.15434 grp1
## 3 48.08651 grp1
## 4 48.15999 grp1
## 5 46.00472 grp1
## 6 49.45541 grp13 Data Visualization
For statistical tests, data visualization is performed before the analysis to provide insights about the structure or distribution of the data. In the comparison of two groups using parametric tests such as the t-test, visualization provides preliminary information on whether the assumptions of the test are met. The bivarplot function in the following code chunk facilitates the examination and comparison of group data using various plots.
bivarplot(ds2)
4 Resampling Statistics with Permutations
The permtest function of the package groupcompare, given an example of its usage in the following code chunk, compares the groups in the dataset using permutations and returns the results. In the result object, pval contains the p-values of for the differences of each statictic related to two compared groups. (Please note that the value of R should be higher (e.g. 3000) in real-world applications.)
results <- permtest(ds2, statistic=calcquantdif, alternative="two.sided", R=500)In the code chunk above, ds2 is the name of dataset in long data format in which the group names locate in the second column. As a mandatory argument, the statistic is the name of the function that calculates and returns the quantile differences of the compared groups. In the example, calcquantdif is a function that calculates and returns the differences between group quantiles. Among its arguments, alternative shows the type of null hyphothesis. The default is two.sided but it can be set to less or greater for a single-tail alternative hyphothesis as well. R shows the number of repetitions for bootstrap. The default number of permutations is 5000 but it is recommended to increase up to a number as high as 20000.
5 Results and Interpretation
Below, the code chunk displays the structure of results object and the results stored obtained above. Here, results$tstar stores the statistics obtained in permutations and can be used for further visualization and analysis. As it is seen, the p-values are 2.20e-16 for the difference between all of the quantiles, meaning all the quantiles are siginificantly different.
str(results)## List of 7
## $ t0 : Named num [1:9] 5.62 4.95 4.86 4.98 4.5 ...
## ..- attr(*, "names")= chr [1:9] "P10" "P20" "P30" "P40" ...
## $ tstar : num [1:500, 1:9] -1.325 0.523 -0.974 -0.109 -0.641 ...
## ..- attr(*, "dimnames")=List of 2
## .. ..$ : NULL
## .. ..$ : chr [1:9] "P10" "P20" "P30" "P40" ...
## $ pval : Named num [1:9] 2.2e-16 2.2e-16 2.2e-16 2.2e-16 2.2e-16 ...
## ..- attr(*, "names")= chr [1:9] "P10" "P20" "P30" "P40" ...
## $ alternative: chr "two.sided"
## $ R : num 500
## $ pvalsk : logi [1:2] NA NA
## $ call : language permtest(x = ds2, statistic = calcquantdif, alternative = "two.sided", R = 500)results$t0## P10 P20 P30 P40 P50 P60 P70 P80
## 5.6160059 4.9501201 4.8594475 4.9845448 4.4981128 3.5381490 2.8770037 0.9401363
## P90
## 0.1316084head(results$tstar)## P10 P20 P30 P40 P50 P60
## [1,] -1.3245136 -1.7530604 -2.6483186 -1.0072093 -1.23359354 -0.229271264
## [2,] 0.5231460 -0.3013630 -0.3761942 -0.6354308 -1.42081451 -0.866892293
## [3,] -0.9744594 0.1857390 -0.3761942 0.4736984 -0.01297126 0.259526066
## [4,] -0.1085207 0.6999764 1.2457520 0.1484691 -0.53111609 0.013482962
## [5,] -0.6413481 0.2435761 1.3410715 0.6504253 -0.06140080 0.009553991
## [6,] -0.2829107 0.5266653 1.1272236 0.9468752 0.61473382 -0.139181597
## P70 P80 P90
## [1,] -0.5960080 -0.64748245 -1.03102517
## [2,] -0.8817546 -0.46472389 -0.78346152
## [3,] 0.5389690 0.51634212 0.10362627
## [4,] 0.3276260 0.03798994 1.15707616
## [5,] 0.4385084 0.14315489 -0.08459354
## [6,] -0.4630248 -0.63551890 -1.18610636results$pval## P10 P20 P30 P40 P50 P60 P70 P80
## 2.20e-16 2.20e-16 2.20e-16 2.20e-16 2.20e-16 2.20e-16 2.20e-16 6.20e-02
## P90
## 9.46e-016 Other Tests
In addition to permutation test, performing bootstrap can be useful for validation of the results when comparing the quantiles of two groups. For this purpose, the bootstrap function in the groupcompare package can be run. For details, you can refer to the usage documentation of the package as well as the vignette titled Quantiles Comparison of Two Groups with Bootstrap.
While p-values for the difference of percentiles of two groups have been calculated here, significance differences for other group statistics can be determined by passing different function names to the statistic argument. For example, when the calcstatdif function is assigned to the statistic argument in the example above, p-values can be calculated for the differences between the means, medians, IQRs, and variances of the two groups.