Permutation tests for heterogeneity comparisons in presence of categorical variables with application to university evaluation

In social sciences researchers often meet the problem of determining if the distribution of a categorical variable is more concentrated in population X1 than in population X2. For example the effectiveness of two different PhD programs can be evaluated in terms of the heterogeneity of the set of job opportunities. The job opportunities are nominal categorical variables and populations X1 and X2 include all PhD holders for program 1 and program 2. We may define that a PhD program is better than another if it is able to offer a larger variety of job opportunities. Several other examples can be mentioned to highlight the importance of heterogeneity comparison problems in social sciences; moreover this problem occurs also very often in genetics, biology, medical studies and other sciences. The nonparametric solution of this problem has similarities to that of permutation testing for stochastic dominance on ordered categorical variables, i.e. testing under order restrictions. If ordering of probability parameters in H0 is unknown and it has to be estimated by sampling data, only approximate nonparametric solutions are possible within the permutation approach. Main properties of test solutions and some Monte Carlo simulations in order to evaluate the tests behaviour under H0 and H1, will be presented. A real problem concerned with University evaluation is also discussed.