Northeast Fisheries Science Center Reference Document 06-28
Precision
Exercises Associated
with SARC 42 Production Aging
by Sandra J. Sutherland,
Nina L. Shepherd,
and Sarah E. Pregracke
National
Marine Fisheries Serv., Woods Hole Lab., 166 Water St., Woods Hole MA
02543-1026
Print
publication date November 2006;
web version posted January 23, 2007
Citation: Sutherland
SJ, Shepherd NL, Pregracke SE.
2006. Precision exercises with SARC 42 production aging.
US Dep Commer, Northeast Fish Sci Cent Ref Doc 06-28; 6 p.
Download complete PDF/print version
INTRODUCTION
In production aging programs, age reader accuracy can be thought of
as how often the “right” age is obtained, and precision
as how often the “same” age is obtained (Campana 2001). It
is possible that, over time, an age reader may inadvertently change
the criteria that are used for determining ages, thereby introducing
a bias into the age data. This bias can be measured with accuracy
tests, which consist of the age reader blindly examining known- or
consensus-aged fish from established reference collections. An
age reader may also make periodic mistakes, which introduces random
errors into the data. The degree of this error can be measured
with precision tests, which consist of the age reader blindly re-aging
fish which they have already aged. Both accuracy and precision
must be considered within a quality-control monitoring program.
Acceptable levels of aging accuracy and precision are influenced by
factors such as species, age structure, and age reader experience. Although
percent agreement is strongly affected by these differences, the staff
of the Fishery Biology Program at the Northeast Fishery Science Center
(NEFSC) have long considered levels above 80% to be acceptable. The
total coefficient of variation (CV) is less affected by these differences
and, thus, is a better measure of aging error. In many aging
labs around the world, total CVs of under 5% are considered acceptable
among species of moderate longevity and aging complexity (Campana 2001),
such as the species considered here.
At the NEFSC Fishery Biology Program, the approach to age-data quality
control and assurance has historically been a two-reader system. In
this approach, there are both a primary and a secondary age reader
for each species. The primary age reader conducts all production
aging, in which a large number of samples are aged over a short period
of time using established methods (Penttila and Dery 1988). The
secondary age reader then ages a portion of those same samples using
similar methods. The ages determined by the two readers are compared,
and if they agree sufficiently (above 80% agreement), the production
ages are considered valid. If not, the sources of disagreement
must first be resolved. This interreader approach is still used
in the course of training new readers in order to ensure consistency
in application of aging criteria and in inter-laboratory sample exchanges. Budgetary
and staffing constraints have made this approach less feasible, however,
by reducing the number of species for which there are two competent
age readers at this laboratory.
In response, the NEFSC Fishery Biology Program has updated our approach
to quality control and assurance. Intrareader tests of aging
accuracy and precision, as described above, allow us to quantify the
amount of inherent aging error and bias in the ages determined by each
of our staff members. These values provide a measure of the reliability
of the production age data used in stock assessments, and they may
be directly incorporated into population models as a source of variability.
For the 42nd Northeast Regional Stock Assessment Review
Committee (SARC 42) meeting (NEFSC 2006), exercises were undertaken
to estimate the precision of production aging by the Fishery Biology
Program for silver hake (Merluccius bilinearis) and Atlantic
mackerel (Scomber scombrus). This report lists the results
of those exercises. No accuracy tests were conducted, as the
NEFSC aging laboratory does not yet have reference collections for
these species.
METHODS
For precision tests on both species, subsamples were randomly selected
from the production sample and re-aged by the same age reader. When
re-aging fish, the age reader had knowledge of the same data as during
production aging (i.e. fish length, date captured, and area captured)
but no knowledge of previous age estimates. During age-testing
exercises, no attempts were made to improve results with repeated readings. There
was also no attempt to revise the production ages in cases where differences
occurred.
Results are presented in terms of percentage agreement, total coefficient
of variation (CV), age-bias plots, and age-frequency tables (Campana
et al. 1995; Campana 2001). Also, Bowker’s test of symmetry
(Bowker 1948; Hoenig et al. 1995) was used in cases where the percent
agreement was less than 90%. This statistic tests whether there was
a systematic difference between the two readings.
For mackerel, random subsamples were drawn from the 2002 NEFSC spring
bottom trawl survey, and NEFSC commercial port samples from the second
quarter of 2005 and the fourth quarter of 2002. For the silver
hake exercises, a subsample was selected from the 2004 NEFSC spring
bottom trawl survey.
The SARC 42 scheduling of both mackerel and silver hake, which are
normally aged by the same primary age reader, required that the secondary
age reader perform production aging for silver hake. Therefore,
an interreader comparison was undertaken for this species to compare
the production ages from the secondary reader against test ages by
the primary reader, using 2004 NEFSC spring bottom trawl survey samples.
RESULTS AND DISCUSSION
The total sample sizes associated with the precision exercises were
N = 100 for mackerel and N = 99 for silver hake. Results are
summarized in Table 1.
For mackerel (Figure 1), a high level of precision was attained, with
95% agreement and a total CV of 0.7%. No bias was apparent. This
indicated an adequate level of consistency in age determinations for
this species.
For silver hake (Figure 2), an agreement level of 92% was attained,
with a low total CV (1.8%). No bias was apparent. This
indicated an adequate level of precision by this age reader.
The comparison between the two silver hake age readers (Figure
3,
N = 99) resulted in lower consistency, with 77% agreement and a 5.2%
CV. A Bowker’s test of symmetry revealed a significant
bias (χ2 = 19.0, P < 0.005,
5 df), primarily due to disagreements at ages 3 and 4. There
was no trend in bias, but ages determined by the two age readers differed
significantly at age 4.
Recent age determinations appear to have been reliably
precise for the species in SARC 42 assessments. Despite this high level of precision, ages generated for silver hake collected
on the NEFSC surveys between autumn 2002 and spring 2005 (inclusive) may differ
from ages determined for samples from previous years. This type of uncertainty can
be reduced or eliminated once the Fishery Biology Program has assembled
reference sample collections for all species which we age regularly. These
collections will then be available to train new age readers, to refresh
current age readers’ skills, and to measure the accuracy of each
reader’s ages on a regular basis.
REFERENCES
Bowker AH. 1948. A test for symmetry in contingency tables. Journal
of the American Statistical Association 43:572–574.
Campana SE. 2001. Accuracy, precision, and quality control
in age determination, including a review of the use and abuse of age
validation methods. Journal of Fish Biology 59:197-242.
Campana SE, Annand MC, McMillan JI. 1995. Graphical and
statistical methods for determining the consistency of age determinations. Transactions
of the American Fisheries Society 124:131-138.
Hoenig JM, Morgan MJ, Brown CA. 1995. Analysing differences
between two age determination methods by tests of symmetry. Canadian
Journal of Fisheries and Aquatic Science 52:364–368.
Northeast Fisheries Science Center (NEFSC). 2006. 42nd
northeast regional stock assessment workshop stock assessment
report. NEFSC Reference Document 06-09. 308 p. Available
at http://www.nefsc.noaa.gov/nefsc/publications/crd/crd0609/
Penttila J and Dery LM. 1988. Age determination methods
for northwest Atlantic species. National Oceanic and Atmospheric
Administration Technical Report NMFS 72; 135 p. Available at http://www.nefsc.noaa.gov/fbi/age-man.html
