Prodigal

Evaluating the performance of microbial genefinders is a difficult task, due mainly to the lack of experimentally verified genes in Genbank. A reasonable place to start is the Ecogene set of verified starts. These start sites have been experimentally determined via N-terminal sequencing. We included this set in its entirety, with the exception of those genes containing introns. This set also includes several nonstandard starts (ATA), but we kept those genes in to see if at least the 3' end would be detected.

Prodigal performs exceptionally well on this data set, locating all of the genes in the set, and getting about 96% of the starts correct. In reality, this number is even higher, since 2 of the starts are ATA, which no program will find. A detailed comparison of Prodigal's performance with other genefinding programs can be found in the forthcoming publication.

In addition to doing detailed comparisons with a few key genomes, we also compared Prodigal's annotations with those in Genbank for every organism. In the following table, "Matching Genes" indicates that the same gene was predicted in both the Genbank annotations and with Prodigal (although not necessarily with the same start site). "Matching Starts" indicates the number of genes that are exactly the same between the two (both start and stop). "SN", or Sensitivity, is the number of correct genes divided by the number of genes in the Genbank file. "SP", or Specificity, is a measure of the false positive rate, and is the number of 'Matching Genes' divided by the number of genes predicted by Prodigal. "Accuracy" is defined as (SN+SP)/2.

In performing these comparisons, and looking at how other genefinding programs did with the same data, we discovered that Prodigal predicts consistently fewer genes (lower false positive rate) and gets substantially more starts correct. While other genefinding programs occasionally find 10-20 more genes in a genome than Prodigal, they do so at the cost of 50-100+ more false positives. The performance gap especially widens in high GC genomes, such as Anaeromyxobacter dehalogenans 2CP-C or Pseudomonas aeruginosa PAO1, where Prodigal's start site predictions often exceed those of other programs by as much as 5-10%. It should be noted that Genbank files in general contain many mistakes, and are not the most reliable source of comparison; however, when one implements a change and it results in an improvement across a large number of Genbank files, this can be seen as a likely improvement to the algorithm.

Given Prodigal's consistent performance across all types of genomes, we have come to the conclusion that comparison to Genbank files is often more useful in measuring the accuracy of the Genbank annotations (!) than of Prodigal. Those genomes with a total accuracy (3') of less than 93% or so, and especially those with a total accuracy (5'+3') of less than 85%, could likely benefit from simply substituting the current annotations with the Prodigal predictions.