The study showed no significant difference between icon displays and tables for time to make the decision (P=0.81) or for the quality of the decision. In view of this, the abstract and discussion are deceptive.

Elting, Linda S Martin, Charles G Cantor, Scott B Rubenstein, Edward B (Department of Medical Specialties, University of Texas MD Anderson Cancer Center, 1515 Holcombe Boulevard–Box 40, Houston, TX 77030-4095, USA lelting@notes.mdacc.tmc.edu).

Editor—The statement in the abstract regarding the superiority of the icon display is correct, as is the P value ascribed to the comparison. This reflects the overall comparison among the four displays, using Cochran’s Q test of the repeated measures of correct decisions. This is reported in the abstract and the results section because it was the planned analysis of our primary hypothesis. Exploratory, pairwise analyses were also reported. Coincidentally, the P values for the McNemar tests of the difference between icon displays and the bar charts or pie graphs were also 0.03; the P value for the pairwise comparison between the icon and table displays was 0.17.

The interpretation of the overall, repeated measures test of the primary hypothesis is straightforward. The accuracy rate with icons was superior to that with the other display methods, and the observed difference was unlikely to have occurred by chance. As is commonly the case with exploratory analyses, however, interpretation of the pairwise comparisons is an exercise in explaining the results of underpowered tests. The study was not powered to test these hypotheses, and the P value of 0.17 for the difference between the icon displays (82%) and table displays (68%) reflects the small sample size. (The overall test included 136 observations, and the pairwise tests included only 68.)

One could argue that a difference of 82% versus 68% and a P value <0.20 in a small sample warrant further study and that in a larger sample this difference would be significant. As the P value suggests, however, one could argue equally strongly that there is a 17% probability that the observed difference occurred by chance alone.

Unfortunately, no matter how one explains the results of underpowered tests, in statistics, as in sheep herding, it’s the size of the flock that counts. The prudent reader will cry wolf only in cases justified by adequate sample sizes. Accordingly, we remind readers to interpret the pairwise, exploratory analyses in our study with caution because of their low power. Although underpowered for hypothesis testing, the pairwise comparisons are useful for hypothesis generation. As we have stated, accuracy rates with table displays were intermediate. Future studies should examine this issue.