Identifying significantly different pairs from multcompare output

Data attached.

Using R2019a

I am trying to identify from the 'c' output of multcompare the significantly different pairs (i.e. without having to use the interactive figure)

from https://uk.mathworks.com/help/stats/multcompare.html

In this example the confidence interval does not contain 0, so the difference is significant at the 5% significance level. If the confidence interval did contain 0, the difference would not be significant

This suggests that significantly different pairs are not the rows in c where the lower bound (column 3) < 0 & the upper bound (column 5) > 0.

However in the attached example below this doesn't quite appear to be the case.

load exampleinput.mat

[p, tbl, stats] = anova1(exampleinput.D,exampleinput.form,"ON");
[c,m,~,gnames] = multcompare(stats,"Alpha",0.05,"CType","bonferroni")

significantpairs = c(~(c(:,3) < 0 & c(:,5) > 0),:)
pairs = [gnames(significantpairs(:,1)), gnames(significantpairs(:,2))]

The above code nearly gets it right.

the interactive graph identifies FCK and WBCK as being different, the code does not.

The code identifies FCK and SECK as being different, the graph does not.

The 5 other pairs are correctly identified.

-----------

I have tried to extract the different pairs by writing a function utilising this method, as suggested here. This didn't quite work.

Changing CType to 'tukey-kramer' (suggested here by Peter König), yields similiar results, some pairs are correctly identified, some are not.

-------

So, the question is: which to trust: the interactive graph, or the code? Or rather, what is the interactive graph doing that I'm not?

Thank you.

Answer 1

[Scott MacKenzie]

The significantly different pairs are identified by the p-values in column 6 of the c matrix generated by multcompare. Here's an example using some test data (attached). There are four columns of data representing marks on four tests. The researcher wishes to know which test pairs have marks that significantly differ from each other.

% load test data: 16 marks on 4 tests
y = readmatrix('testdata.txt'); % 16x4

% do the anova 
[~, ~, stats] = anova1(y, {'t1' 't2' 't3' 't4'}, 'off');

% pass stats object into multcompare
[c, ~, ~, ~] = multcompare(stats, 'display', 'off')

c = 6×6

1.0000 2.0000 -3.9088 -0.8750 2.1588 0.8711 1.0000 3.0000 -7.5338 -4.5000 -1.4662 0.0013 1.0000 4.0000 -4.8463 -1.8125 1.2213 0.3983 2.0000 3.0000 -6.6588 -3.6250 -0.5912 0.0130 2.0000 4.0000 -3.9713 -0.9375 2.0963 0.8464 3.0000 4.0000 -0.3463 2.6875 5.7213 0.1002

The summary results (using the default alpha of .05) are in the 6th column above. The marks are significantly different between test 1 and test 3 (p = .0013) and between test 2 and test 3 (p = .0130).

The default test is Tukey-Kramer. A different test can be selected via the ctype option. Generally, the tests yield the same or similar results.

NOTE: There is more than one version of multcompare. The output is a bit different for the version of multcompare that receives a repeated measures model as input, rather than a stats object (as demonstrated above). Check the documentation for details