This is the accessible text file for GAO report number GAO-04-606 
entitled 'Pacific Groundfish: Continued Efforts Needed to Improve 
Reliability of Stock Assessments' which was released on June 14, 2004. 

This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao. gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Requesters: 

June 2004: 

PACIFIC GROUNDFISH: 

Continued Efforts Needed to Improve Reliability of Stock Assessments: 

GAO-04-606: 

GAO Highlights: 

Highlights of GAO-04-606, a report to congressional requesters

Why GAO Did This Study: 

Because of concerns raised about the accuracy of National Marine 
Fisheries Service (NMFS) stock assessments, GAO reviewed the 
assessments for five species of Pacific groundfish: Pacific hake and 
four types of rockfish—bocaccio, canary, darkblotched, and yelloweye. 
Specifically, for these five species GAO (1) assessed the reliability 
of NMFS’ stock assessments, (2) identified which relevant 
recommendations from NMFS’ stock assessment improvement plan have been 
implemented and which have not, and (3) identified the costs associated 
with planned and ongoing improvements to groundfish stock assessments. 

What GAO Found: 

The reliability of the NMFS assessments is questionable for the five 
species GAO reviewed, although the assessments were based on the best 
information available at the time they were conducted. According to 
NMFS officials and a National Research Council report, to obtain 
reliable results each stock assessment should include at least one 
NMFS data source of sufficient scope and accuracy because such data 
are derived from unbiased, statistical designs. However, in the 
yelloweye assessment, no NMFS data were used, and in the darkblotched, 
canary, and bocaccio assessments, the NMFS data were limited because 
the NMFS’ surveys were conducted in trawlable waters only. A 2003 NMFS 
report concluded that darkblotched groundfish are less abundant and 
bocaccio and canary are more abundant in untrawlable waters. Also for 
all five species, NMFS lacks a standard approach for ensuring the 
reliability of non-NMFS data used in stock assessments. Some assessors 
reviewed the quality of non-NMFS data; others did not. The assessors 
who reviewed the quality of the non-NMFS data found errors that made 
some of the data unusable or that could have impaired the reliability 
of certain stock assessments. Finally, for four species, the stock 
assessment reports were questionable because they did not present the 
uncertainty associated with the population estimates. For example, the 
canary stock assessment review panel recommended that standard 
estimates of uncertainty be included in the assessment report because 
without them it is difficult to determine their reliability. 

NMFS has taken steps to implement some of the recommendations contained 
in the NMFS stock assessment improvement plan, but much remains to be 
done. NMFS has concentrated its efforts mostly on improving data 
quantity. For example, NMFS increased the frequency of groundfish stock 
assessments and extended the geographic ranges of the shelf and slope 
surveys to cover over 300 more miles along the southern California 
coast. However, because of staffing and funding limitations, NMFS has 
not yet implemented many of the recommendations aimed at obtaining more 
types of data and improving data quality. For example, NMFS has not 
collected enough ecosystem data, and the frequency and range of 
recruitment surveys (estimated production of new members of a fish 
population) are limited. Finally, because of other program priorities, 
NMFS has not implemented the recommendation to create a comprehensive 
plan that combines the improvement plan and its complementary plans. 

NMFS records indicate at least $8.9 million is needed to complete 
ongoing and planned stock assessment improvements—$2.6 million that 
NMFS’ Northwest Fisheries Science Center requested but did not receive 
in fiscal years 2001 to 2003, and $6.3 million requested for fiscal 
years 2004 and 2005. It will cost about (1) $7.7 million to improve the 
types of data used, such as more untrawlable water and recruitment 
surveys and (2) $1.2 million to improve the quality of data used in 
stock assessments, such as enhanced calibration of vessel equipment and 
standardized trawl survey procedures. The actual cost of the remaining 
improvements may be even higher than the $8.9 million estimated because 
the estimates primarily reflect the amount of money that agency 
officials believed could be realistically obtained, rather than what 
the improvements might cost. 

What GAO Recommends: 

GAO recommends that the Secretary of Commerce require the Director of 
NMFS to take actions to improve the reliability of stock assessments, 
such as continuing efforts to improve the quality and types of data 
used in groundfish assessments, establishing a standard approach that 
requires that data used in stock assessments be evaluated for 
reliability, and requiring stock assessment reports clearly present the 
uncertainties in the assessments. 

NOAA generally agreed with the report’s accuracy and agreed with the 
report’s recommendations, but expressed concern that the report’s 
conclusion could be misconstrued to infer that the assessments are 
unreliable for use in managing the west coast groundfish fishery. 

www.gao.gov/cgi-bin/getrpt?GAO-04-606. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Anu K. Mittal at (202) 
512-3841 or MittalA@gao.gov. 

[End of section]

Contents: 

Letter: 

Results in Brief: 

Background: 

Reliability of NMFS' Pacific Groundfish Stock Assessments Is 
Questionable: 

Some Recommended Stock Improvements Have Been Implemented, but Much 
Remains to be Done: 

Remaining Improvements Estimated to Cost at Least $8.9 Million: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendixes: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Comments From the Department of Commerce: 

GAO Comments: 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Staff Acknowledgments: 

Tables: 

Table 1: Stock Assessment Reliability Issues for Five Pacific 
Groundfish Species: 

Table 2: Pacific Groundfish Stock Assessment Improvement Funds 
Requested and Received by Data Type and Quality, Fiscal Years 2001-
2005: 

Figures: 

Figure 1: Photograph of a Pacific Hake: 

Figure 2: Photograph of a Bocaccio Rockfish: 

Figure 3: Photograph of a Canary Rockfish: 

Figure 4: Photograph of a Darkblotched Rockfish: 

Figure 5: Photograph of a Yelloweye Rockfish: 

Abbreviations: 

NMFS : National Marine Fisheries Service: 

NOAA: National Oceanic and Atmospheric Administration: 

Letter June 4, 2004: 

The Honorable Gordon H. Smith: 
United States Senate: 

The Honorable Ron Wyden: 
United States Senate: 

Pacific groundfish, 82 species of bottom-dwelling fish, such as several 
species of rockfish and sole, have contributed to the economies of 
fishing communities in California, Oregon, and Washington, generating 
over $220 million in income in 2001. However, some populations of 
Pacific groundfish have declined sharply because of natural conditions, 
such as climate, as well as man-made conditions, such as overfishing. 
In an effort to achieve sustainable populations, the National Marine 
Fisheries Service (NMFS) significantly reduced the allowable harvest 
for nine species of Pacific groundfish. NMFS made this decision based 
on its stock assessments--which include estimates of species' total 
biomass (weight of the total population)--and recommendations from the 
Pacific Fishery Management Council, one of eight councils responsible 
for managing fisheries in federal waters. 

NMFS generally uses internal scientists or contracts with outside 
experts to conduct stock assessments. These assessors use multiple 
types of data, including data from external sources, such as state-
collected data on the poundage and species of fish brought ashore by 
commercial and recreational fishermen. NMFS' Northwest Fisheries 
Science Center (Northwest Center), which coordinates Pacific groundfish 
stock assessments, also collects its own data through fish population 
surveys, particularly two key surveys: (1) the shelf and slope bottom 
trawl survey, in which a trawl net is dragged along the ocean floor, 
and the resulting catch is sorted, counted, and biologically examined 
(e.g., age and sex) and (2) the acoustic survey, in which sound waves 
are used to measure density of schools of groundfish in mid-level 
waters; bottom trawls are not feasible in these waters. Fish survey and 
other data are entered into computerized mathematical models (stock 
assessment models), which estimate the total biomass for the fish 
species being assessed. However, concerns have been raised about the 
accuracy of stock assessments and therefore whether they can be relied 
upon to specify the amount of fish that can be harvested to ensure a 
sustainable population. 

In 1999, NMFS commissioned a national task force to review the stock 
assessment process across the agency. Task force members were from NMFS 
headquarters and five of the six NMFS science centers, including the 
Northwest Center. In the October 2001 Marine Fisheries Stock Assessment 
Improvement Plan,[Footnote 1] the task force identified a lack of 
adequate data as the greatest impediment to producing reliable stock 
assessments and recommended actions to improve the quantity, quality, 
and types of data used in the models that estimate biomass. The task 
force also identified a need to better communicate and quantify 
uncertainties in these models. 

In this context, you requested that we review NMFS' stock assessments 
for five species of Pacific groundfish: Pacific hake as well as four 
types of rockfish--bocaccio, canary, darkblotched, and yelloweye. 
Specifically, for these five species you asked us to (1) assess the 
reliability of NMFS' stock assessments, (2) identify which relevant 
recommendations from the stock assessment improvement plan have been 
implemented and which have not, and (3) identify the estimated costs 
associated with planned and ongoing improvements to groundfish stock 
assessments. 

To address these issues, we reviewed key laws and stock assessment 
studies (2002 and 2004 studies for Pacific hake, 2002 and 2003 studies 
for bocaccio, 2002 study for canary, 2000 and 2003 studies for 
darkblotched, and 2001 and 2002 studies for yelloweye), along with the 
types of fish population surveys used and the controls over stock 
assessment data. We spoke with officials from NMFS, the Oregon 
Department of Fish and Wildlife, the Washington Department of Fish and 
Wildlife, the California Department of Fish and Game, the Pacific 
Fishery Management Council, the Pacific States Marine Fisheries 
Commission, environmental groups, and industry associations, as well as 
fishermen and academics. We also reviewed NMFS' budget documents to 
develop cost estimates. We did not simulate NMFS' stock assessment 
models or evaluate the mathematical and statistical methodologies used 
in the models. We conducted our review from May 2003 through April 2004 
in accordance with generally accepted government auditing standards. 
See appendix I for additional details on our scope and methodology. 

Results in Brief: 

For a variety of reasons, the reliability of the NMFS assessments is 
questionable for the five species we reviewed, although the assessments 
were based on the best information available at the time they were 
conducted. First, according to NMFS officials and a 1998 National 
Research Council report, to obtain reliable results, each stock 
assessment should include at least one source of NMFS-collected data of 
sufficient scope and accuracy because such data are derived from 
unbiased, statistical, and scientific designs. However, the four 
rockfish assessments did not meet this standard, although the 
assessment for Pacific hake did. NMFS did not use any of its own data 
in the yelloweye assessment, and the NMFS data used in the bocaccio, 
canary, and darkblotched assessments were limited because NMFS 
conducted its surveys in trawlable waters only; these fish also inhabit 
both trawlable and untrawlable waters. In the absence of data from 
rocky, untrawlable waters, the assessors estimated overall biomass 
using the NMFS data collected from the trawlable areas. However, in 
2003, NMFS found that darkblotched are less abundant and bocaccio and 
canary are more abundant in untrawlable waters. As a result, some 
rockfish populations may be overstated, while others may be 
understated. Second, for all the species, NMFS lacks a standard 
approach for ensuring the reliability of non-NMFS data used in the 
stock assessments. As a result, some assessors reviewed the quality of 
these data while others did not. The assessors who reviewed the quality 
of the non-NMFS data found errors that they believed made the data 
unusable or impaired the reliability of the stock assessment. Finally, 
the reliability of four of the stock assessment reports--bocaccio, 
canary, darkblotched, and yelloweye--is questionable because they did 
not present the uncertainty associated with the population estimates. 
The National Research Council, NMFS, and GAO agree that estimates of 
uncertainty should be included in stock assessment reports because 
without them it is difficult to determine their reliability. 

NMFS has taken steps to implement the NMFS stock assessment improvement 
plan recommendations to improve stock assessments, but much remains to 
be done to enhance their reliability. While the task force considered 
the lack of adequate data--in quantity, quality, and type--as the 
greatest impediment to accurate stock assessments, the Northwest Center 
has concentrated most of its efforts on improving data quantity. For 
example, starting in 2003, the Northwest Center extended the geographic 
ranges of the shelf and slope surveys, expanding them to cover over 300 
additional miles along the southern California coast. It also increased 
the frequency of the groundfish shelf and slope survey from triennially 
to annually and the frequency of the Pacific hake acoustic survey from 
triennially to biennially. However, NMFS has not yet fully implemented 
many of the recommendations aimed at obtaining more types of data and 
improving data quality. For example, although Northwest Center 
officials recognize that recruitment surveys (estimated production of 
new members of a fish population) should be conducted biannually along 
the entire coast, these surveys are currently conducted only annually 
and only in selected regions. While Northwest Center officials also 
recognize that actions, such as calibrating vessel equipment to ensure 
comparable survey data, are needed to improve data quality, these 
actions have not yet been fully implemented. Finally, NMFS has not 
implemented the recommendation to create a comprehensive plan that 
combines the improvement plan and related plans to develop integrated 
program initiatives. Although NMFS officials recognize a need for such 
a plan, other priorities have precluded them from developing it. 

According to NMFS funding and budget documents, NMFS needs at least 
$8.9 million to complete ongoing and planned improvements to its stock 
assessments of Pacific groundfish--$2.6 million that the Northwest 
Center had requested but did not receive in fiscal years 2001 through 
2003, and $6.3 million requested for fiscal years 2004 and 2005. 
Specifically, according to these records, the Northwest Center has the 
following funding needs: 

* $7.7 million to improve the types of data used, including $2.4 
million for surveys of untrawlable waters, $2.1 million to expand 
acoustic and recruitment surveys, and $3.2 million to collect ecosystem 
data and: 

* $1.2 million to improve the quality of data used in stock 
assessments, including $600,000 to enhance the calibration of vessel 
equipment, $525,000 to develop and implement methods to more accurately 
distinguish among groundfish species, and $75,000 to standardize trawl 
survey procedures. 

The actual cost of implementing improvements to Pacific groundfish 
stock assessments may be even higher than the $8.9 million estimated 
because the Northwest Center's budget requests primarily reflect the 
amount of money the Center believed it could realistically obtain, 
rather than what the improvements would cost, according to NMFS 
officials. 

Background: 

The Magnuson-Stevens Fisheries Conservation and Management Act granted 
responsibility for managing marine resources to the Secretary of 
Commerce. [Footnote 2] The Secretary delegated this responsibility to 
NMFS, which is part of Commerce's National Oceanic and Atmospheric 
Administration (NOAA). The act also established eight regional fishery 
management councils, each responsible for making recommendations to the 
Secretary of Commerce about managing fisheries in federal waters. The 
eight fishery management councils--consisting of fishing industry 
participants, state and federal fishery managers, and other interested 
parties--and their areas of responsibility include the following: 

* Caribbean Council, covering waters off the U.S. Virgin Islands and 
the Commonwealth of Puerto Rico;

* Gulf of Mexico Council, covering waters off Texas, Louisiana, 
Mississippi, Alabama, and the west coast of Florida;

* Mid-Atlantic Council, covering waters off New York, New Jersey, 
Delaware, Maryland, Virginia, and North Carolina;

* New England Council, covering waters off Maine, New Hampshire, 
Massachusetts, Rhode Island, and Connecticut;

* North Pacific Council, covering waters off Alaska;

* Pacific Council, covering waters off California, Oregon, and 
Washington;

* South Atlantic Council, covering waters off North Carolina, South 
Carolina, Georgia, and the east coast of Florida; and: 

* Western Pacific Council, covering waters off Hawaii, American Samoa, 
Guam, the Commonwealth of the Northern Mariana Islands, and uninhabited 
U.S. territories in the Western Pacific. 

In addition to these eight councils, NMFS has six regional science 
centers, which are responsible for generating the scientific 
information necessary for the conservation, management, and use of each 
region's marine resources. The six fishery science centers and their 
areas of responsibility are as follows: 

* Alaska Center, covering the coastal oceans off Alaska and parts of 
the west coast of the United States;

* Northeast Center, covering waters along the Northeast Continental 
Shelf from the Gulf of Maine to Cape Hatteras, North Carolina;

* Northwest Center, covering the northeast Pacific Ocean, primarily 
waters off the coasts of California, Oregon, Washington, and British 
Columbia;[Footnote 3]

* Pacific Islands Center, covering the central and western Pacific 
Ocean;

* Southwest Center, primarily covering waters off the coast of 
California and areas throughout the Pacific and Antarctic 
Oceans;[Footnote 4] and 

* Southeast Center, covering waters along the continental southeastern 
United States as well as Puerto Rico and the U.S. Virgin Islands. 

The Magnuson-Stevens Act, as amended by the Sustainable Fisheries 
Act,[Footnote 5] also established national standards for fishery 
conservation and management. These standards deal with preventing 
overfishing, using scientific information, using fishery resources 
efficiently, minimizing bycatch,[Footnote 6] and minimizing 
administrative costs. The regional councils use these standards to 
develop appropriate plans for conserving and managing fisheries under 
their jurisdiction, including measures to prevent overfishing and to 
rebuild overfished stocks as well as measures to protect, restore, and 
promote the long-term health and stability of the fishery. 

In 1982, the Pacific Fishery Management Council (Pacific Council) 
released its initial Fishery Management Plan for groundfish. The 
Pacific Council's goal is to have long-range plans for managing 
groundfish fisheries that will promote a stable planning environment 
for the seafood industry, including marine recreation interests, and 
will maintain the health of the resource and environment. To help 
achieve these goals, stock assessments are conducted on groundfish 
species to estimate fish populations. Since 1995, the Northwest Center 
has had lead responsibility for conducting stock assessments on Pacific 
groundfish. The Northwest Center receives assistance from other NMFS 
science centers, such as the Southwest Center, which conducted the 
bocaccio assessment. 

Stock assessments are the biological evaluation of the status of fish 
stocks. Stock assessments provide official estimates in key areas, such 
as the size of the stock population, the size of the spawning 
population, the amount of fish that have died (fish mortality), and the 
estimated number of fish at a particular young age (recruitment). Stock 
assessments form the scientific basis used by regional councils to 
determine biologically sustainable harvests and guide the monitoring 
and rebuilding of overfished and threatened stocks. For example, 
regional councils use stock assessments and other indicators of 
biological productivity to recommend to NMFS a maximum, or total 
allowable catch, in a particular fishery--typically for a year. Stock 
assessments are a key tool for managing fisheries. Without stock 
assessments, fishery managers would have limited information about the 
status of fisheries in making decisions about setting harvest levels 
and developing plans to rebuild overfished stocks. 

For each species, the assessor reviews previous stock assessments, 
gathers available data about the species being assessed, runs the data 
through computer-generated models, and estimates the species' total 
biomass. Stock assessors use NMFS-collected data, such as stock surveys 
conducted on NOAA vessels or contracted commercial fishing vessels, as 
well as data collected by non-NMFS sources, such as commercial and 
recreational catch data collected by state agencies. The following six 
key types of data are collected: 

* Stock abundance--surveys of how many fish constitute a stock's total 
size or weight. 

* Commercial and recreational fisheries data--the amount and 
composition of fish caught from a particular stock, whether caught 
intentionally by commercial and recreational fishermen or 
unintentionally caught and discarded. Data sources include fishing 
logbooks, dockside samples, and onboard observations, among others. 

* Life history--biological data, such as the age and sex composition of 
the stock, age at first maturity, fertility, average lifespan, and 
natural mortality. 

* Ecosystem relationships--data on the relationship between a fish 
stock and its physical environment, as well the relationship of a fish 
stock to other species. 

* Recruitment research--data on the abundance of juvenile and larval 
fish (fish at their earliest stage), which helps scientists forecast 
the size of a particular stock in the future. 

* Synoptic oceanographic sampling--data on the ocean ecosystem, such as 
water temperature or salinity, plankton composition, or ocean currents. 

For each stock assessment, a review panel, consisting of NMFS 
scientists and outside experts, independently reviews the methodology 
of the assessment and works with the assessor to ensure their comments 
are adequately addressed. 

Through 2003, 24 of the 82 species of Pacific groundfish have had a 
full quantitative stock assessment. Relying on these assessments, NMFS 
has declared that nine species of Pacific groundfish are overfished, 
including the five species of Pacific groundfish we reviewed in this 
report--Pacific hake as well as bocaccio rockfish, canary rockfish, 
darkblotched rockfish, and yelloweye rockfish. Rockfish are long-lived, 
late-maturing and slow-growing species, making them particularly 
susceptible to overfishing. More specifically Pacific hake, also called 
Pacific whiting, is generally found off the west coast of North 
America. It is one of many species of hake distributed in the Atlantic 
and Pacific oceans. Fishing of Pacific hake primarily takes place off 
the coasts of northern California, Oregon, Washington, and British 
Columbia. Fishermen use mid-water trawls and generally fish over the 
ocean bottom at depths of 100 to 500 meters. Pacific hake was declared 
overfished in 2002 because the 2002 stock assessment estimated Pacific 
hake biomass at 700,000 metric tons. By 2004, the biomass was 
estimated at between 2.7 and 4.2 million metric tons, and Pacific 
hake is no longer considered overfished. Figure 1 shows a picture of a 
Pacific hake. 

Figure 1: Photograph of a Pacific Hake: 

[See PDF for image] 

[End of figure] 

Bocaccio rockfish generally inhabit waters off the coast of northern 
Baja, Mexico to Alaska. Bocaccio, commonly sold at market as "red 
snapper," are commercially fished using trawls, hook-and-line and 
gillnets. Adult bocaccio commonly live over rocky areas or open areas 
of the ocean's floor to about 320 meters. Bocaccio were formally 
declared to be "overfished" in 1999. The 2003 stock assessment 
estimated the bocaccio biomass at 7,133 metric tons. Figure 2 shows a 
picture of a bocaccio rockfish. 

Figure 2: Photograph of a Bocaccio Rockfish: 

[See PDF for image] 

[End of figure] 

Canary rockfish inhabit the northeastern Pacific Ocean, from northern 
Baja, Mexico to the western Gulf of Alaska. Adult canary rockfish are 
primarily found along the continental shelf, from 46 to 457 meters 
deep. Canary rockfish are harvested commercially using trawl nets and 
hook-and-line and are also considered an important species for 
recreational fishermen. NMFS declared canary rockfish as overfished in 
2000. The 2002 stock assessment estimated the canary rockfish biomass 
at 6,197 metric tons. Figure 3 shows a picture of a canary rockfish. 

Figure 3: Photograph of a Canary Rockfish: 

[See PDF for image] 

[End of figure] 

Darkblotched rockfish are found in the waters from Santa Catalina 
Island, California to the Bering Sea on soft bottom areas at about 29 
to 549 meters deep. Commercial fishery concentrations are located off 
the coasts of California and Oregon. Darkblotched rockfish are caught 
primarily by commercial trawls and contribute to both commercial and 
recreational fishing. NMFS determined that darkblotched rockfish was 
overfished in 2000, when the last full stock assessment was conducted; 
it was updated in 2003. This update estimated the darkblotched rockfish 
biomass at 7,266 metric tons in 2001. [Footnote 7] Figure 4 shows a 
picture of a darkblotched rockfish. 

Figure 4: Photograph of a Darkblotched Rockfish: 

[See PDF for image] 

[End of figure] 

Yelloweye rockfish almost exclusively inhabit rocky areas from northern 
Baja, Mexico to the Aleutian Islands, Alaska. Yelloweye rockfish, found 
in depths ranging from 15 to 550 meters, are caught using both trawl 
nets and line gear and are highly prized by both commercial and 
recreational fishermen. Stock assessments for yelloweye rockfish were 
first conducted in 2001, and NMFS determined that the species was 
overfished in 2002. The 2002 stock assessment estimated the yelloweye 
biomass at 2,325 metric tons in 2001. [Footnote 8] Figure 5 shows a 
picture of a yelloweye rockfish. 

Figure 5: Photograph of a Yelloweye Rockfish: 

[See PDF for image] 

[End of figure] 

Reliability of NMFS' Pacific Groundfish Stock Assessments Is 
Questionable: 

The reliability of NMFS' stock assessments is questionable for the 
Pacific hake and four rockfish species we reviewed, although they were 
based on the best information available at the time the assessments 
were conducted. The reliability of the stock assessments we reviewed is 
questionable because (1) four of the assessments did not have at least 
one NMFS-collected data source of sufficient scope and accuracy; (2) 
NMFS lacked a standard process for assessing the reliability of non-
NMFS data used in all five assessments; and (3) for four of the 
assessments, the stock assessment reports did not adequately identify 
the uncertainty of the biomass estimates. (See table 1. ) To address 
these limitations, the Northwest Center plans to increase the scope and 
accuracy of its collected data, as additional funds become available; 
is implementing changes that will help ensure the reliability of non-
NMFS data; and plans to update the stock assessment model to provide 
uncertainty ranges for the 2005 stock assessments. 

Table 1: Stock Assessment Reliability Issues for Five Pacific 
Groundfish Species: 

Species: Pacific hake; 
NMFS-collected data available? Yes; 
Standard data reliability testing conducted by NMFS? No; 
Uncertainty ranges for biomass estimates provided? Yes. 

Species: Bocaccio; 
NMFS-collected data available? Limited[A]; 
Standard data reliability testing conducted by NMFS? No; 
Uncertainty ranges for biomass estimates provided? No. 

Species: Canary; 
NMFS-collected data available? Limited[A]; 
Standard data reliability testing conducted by NMFS? No; 
Uncertainty ranges for biomass estimates provided? No. 

Species: Darkblotched; 
NMFS-collected data available? Limited[A]; 
Standard data reliability testing conducted by NMFS? No; 
Uncertainty ranges for biomass estimates provided? No. 

Species: Yelloweye; 
NMFS-collected data available? No; 
Standard data reliability testing conducted by NMFS? No; 
Uncertainty ranges for biomass estimates provided? No. 

Source: GAO analysis of information provided by NMFS. 

[A] NMFS-collected data are available for only the trawlable portion of 
the species' habitat. 

[End of table]

Absence of NMFS Survey Data Adversely Affects the Assessments' 
Reliability: 

Stock assessors use a variety of data, including NMFS data and non-NMFS 
data, in developing their assessments. [Footnote 9] Two key pieces of 
NMFS survey data are the shelf and slope bottom trawl survey and the 
acoustic survey. Other NMFS data that assessors sometimes use include 
larval surveys (data for fish in their earliest stage) and recruitment 
surveys. The non-NMFS data assessors use include commercial catch data 
and recreational catch data. 

A 2002 National Research Council report found that the inclusion of 
NMFS survey data was the best option for a reliable estimate of 
abundance because such surveys use an unbiased statistical design, 
control sampling locations, and provide for quality assurance. [Footnote 
10] According to Northwest Center officials, to obtain reliable 
results, each stock assessment should include at least one source of 
NMFS-collected data of sufficient scope and accuracy because such 
surveys are unbiased and scientifically designed. 

Northwest Center officials raised concerns about basing assessments 
solely on non-NMFS data such as commercial and recreational catch data. 
Catch data do not provide the species' relative or absolute biomass, 
according to NMFS officials. For example, catch data alone is 
insufficient because: 

* fishermen are not randomly sampling the ocean but are fishing areas 
that they are allowed to fish and they believe have the most fish;

* fishing restrictions, such as a total allowable catch, can limit the 
amount of fish being caught; and 

* catch data have often been inaccurate for a variety of reasons, such 
as imprecise accounting for dead fish tossed back into the ocean. 

Although the assessors used several different data sources, four of the 
five assessments did not use NMFS survey data or the NMFS data used 
covered only a portion of the species' habitats. In the yelloweye 
assessment, no NMFS survey data were available because yelloweye live 
almost exclusively in the rocky habitat that NMFS trawl surveys cannot 
cover. As a result, the yelloweye assessment was based solely on non-
NMFS data. Similarly, the NMFS survey data used in the bocaccio, 
canary, and darkblotched assessments were limited in scope because the 
surveys were conducted only in trawlable waters. Bocaccio, canary, and 
darkblotched live in both the trawlable and untrawlable habitats. Using 
data from trawl surveys conducted from 1977 through 1998, NMFS reported 
in 2003 that 77 percent of the survey area was trawlable and 23 percent 
was untrawlable. [Footnote 11] Lacking data on species in the 23 percent 
of the ocean floor that is untrawlable, the assessors estimated the 
overall biomass using the NMFS data collected from the trawlable areas. 
However, the abundance in the trawlable area is not representative of 
the abundance in the untrawlable area. The 2003 NMFS report also found 
that darkblotched groundfish are less abundant in untrawlable waters, 
while canary and bocaccio species are more abundant in untrawlable 
waters. As a result, some rockfish populations may be understated while 
others may be overstated. According to stock assessors, relying solely 
on survey data from trawlable waters increases the uncertainty of stock 
assessments. 

In contrast, the fifth groundfish species, Pacific hake, lives 
primarily in mid-water habitat; and so the concerns about the lack of 
NMFS data in rocky, untrawlable habitats are not applicable. 

NMFS Lacks a Standard Process for Ensuring the Reliability of Non-NMFS 
Data Used in the Stock Assessments: 

NMFS does not have a standard process for evaluating whether the non-
NMFS data used in its stock assessments are reliable. We believe that 
certain internal control activities, such as a standard process for 
ensuring data reliability, can help ensure that information used to 
make management decisions is complete and accurate. [Footnote 12] 
Lacking a standard process, some assessors reviewed the quality of the 
raw non-NMFS data, while others did not. Assessors who reviewed for 
data quality found mistakes that they believed made some of the data 
unusable or that could have impaired the accuracy of the stock 
assessments. For example, the assessor for the 2002 yelloweye stock 
assessment found numerous errors in the recreational catch data, such 
as attributing the catch from an entire fishing vessel to a single 
fisherman, and thus did not use the data because doing so could have 
resulted in overestimating the biomass. According to another stock 
assessor, commercial catch data frequently have inconsistencies. 
Specifically, the assessor said California, Oregon, and Washington 
require fishermen to enter catch and location information into 
logbooks, but logbooks are often incomplete and inaccurate. While the 
stock assessment review panels evaluated the assessments, the panels 
did not evaluate the quality of the raw data used in the assessments. 

According to a Northwest Center official, several assessors have raised 
concerns about data quality and accessibility in feedback meetings. In 
response to these concerns, the Northwest Center official has recently 
begun assigning data stewards to each data set used in its assessments. 
Data stewards are responsible for helping assessors compile relevant 
data and for conducting quality assurance and quality control checks on 
the data. The Northwest Center plans to conduct a data quality workshop 
in July 2004 to formally establish the roles and responsibilities of 
the data stewards, with the intent of standardizing the data evaluation 
process. 

Stock Assessments for Four Species Did Not Provide Uncertainty 
Estimates: 

In 1998, the National Research Council recommended that NMFS include 
realistic measures of uncertainty in its stock assessments. [Footnote 
13] NMFS' 2001 stock assessment improvement plan also recognized the 
need to better quantify and communicate the uncertainty in assessments. 
In a review of the 2002 canary assessment, the stock assessment review 
panel recommended that standard estimates of uncertainty be included in 
future assessments because it is difficult to determine the reliability 
of the stock assessment without them. [Footnote 14] Similarly, we 
believe that estimates based on samples should have a range of 
uncertainty to show the amount of variability in the 
estimates. [Footnote 15] However, the bocaccio, canary, darkblotched, 
and yelloweye assessments did not present a measure of uncertainty 
associated with the biomass estimates. 

Without uncertainty ranges, it is difficult for regional councils and 
NMFS to know how much confidence they can have in relying on the 
estimates for determining stock abundance and hence for setting 
allowable harvests of the fish. For example, lacking uncertainty 
ranges, the 2002 bocaccio stock assessment estimated a bocaccio biomass 
of 2,914 metric tons in 2002. The 2003 assessment of bocaccio biomass, 
however, estimated 6,506 metric tons in 2002--more than doubling the 
previous estimate because of additional and updated data. With such 
wide variations, it is important to provide uncertainty ranges, 
otherwise management may make inappropriate decisions. 

While assessors told us that their stock assessments included some 
information about differences in estimated biomass when using different 
data sources (sensitivity analyses), the mathematical model that NMFS 
uses to estimate biomass (Stock Synthesis model) does not calculate 
uncertainty ranges. NMFS officials told us that NMFS is updating the 
model so that it can compute uncertainty ranges; NMFS expects to use 
the updated model for all 2005 stock assessments. The Pacific hake 
assessor used a mathematical model (AD Model Builder) that could 
compute uncertainty ranges and included these ranges in the Pacific 
hake assessment report. 

Some Recommended Stock Improvements Have Been Implemented, but Much 
Remains to be Done: 

NMFS has taken some steps recommended in the Marine Fisheries Stock 
Assessment Improvement Plan to improve the quantity, quality, and type 
of data used in Pacific groundfish stock assessments, but much remains 
to be done to make the assessments more reliable. The Northwest Center 
has concentrated most of its efforts on implementing recommendations 
aimed at obtaining more data. Recommendations aimed at increasing the 
types of data and improving their quality have not yet been fully 
implemented for a variety of reasons, such as staffing and funding 
limitations. In addition, other program priorities have precluded NMFS 
from implementing the recommendation to create a comprehensive plan 
that incorporates the improvement plan and related plans so that it can 
develop integrated program initiatives to improve stock assessments. 

The Northwest Center Has Taken Some Steps Recommended for Improving 
Stock Assessments: 

The October 2001 stock assessment improvement plan identified three 
scenarios (tiers) to consider when analyzing the resources needed to 
improve stock assessments. The three tiers of assessment improvements 
are as follows: 

* Tier 1--improve stock assessments using existing data without 
initiating new data collection programs. 

* Tier 2--conduct baseline monitoring of species, which in most cases 
requires sampling the species at least every 1 to 3 years, and 
preferably at least every 1 to 2 years. 

* Tier 3--implement "next generation" stock assessments by explicitly 
incorporating ecosystem considerations, such as multispecies 
interactions and environmental effects in assessments. 

The improvement plan also made a number of recommendations to improve 
stock assessments. The recommendations fall into the following four 
categories: 

* Data collection--pursue new initiatives to expand data collection 
efforts that at a minimum bring stock assessment science to Tier 2. In 
addition, continue to develop partnerships and cooperative research 
programs with other entities, such as state agencies, commercial and 
recreational fishing organizations, and individuals to improve the 
quantity, quality, and type of data collected. 

* Communication--educate constituents about NMFS' strategies for 
improving stock assessments. 

* Training--implement comprehensive training and staff development 
programs for NMFS' analytical and quantitative staff, and augment 
existing programs that support graduate students interested in stock 
assessment science. 

* Planning--develop integrated program initiatives by preparing a 
comprehensive plan that combines the improvement plan with its 
complementary plans. 

Improvement in the quantity of data collected for use in stocks 
assessments is a key component to achieving Tier 2 status. The 
improvement plan states that as the quantity of the data increases, the 
assessments become more reliable because the data cover a longer period 
of time, producing better population trend information. Northwest 
Center officials said that the quality of the data improves with more 
frequent surveys and more randomly selected survey locations that, over 
the long term, provide a better understanding of the variability 
inherent in the population distribution and abundance. A better sense 
of trends and variability allow for improved short-term predictions of 
the status of the species. 

The Northwest Center has concentrated most of its efforts on 
implementing improvements in data quantity, such as more frequent 
acoustic, and shelf and slope bottom trawl surveys. The following 
illustrate some of the actions the Northwest Center took in 2003 to 
improve data quantity: [Footnote 16]

* Increased the frequency of the Pacific hake acoustic survey from 
triennially to biennially. Beginning in 2003, the survey was 
restructured into a single, integrated survey with Northwest Center and 
Canadian officials jointly planning all survey elements. Officials from 
the Northwest Center and Canada now jointly conduct all of the acoustic 
surveys. 

* Increased the frequency of the groundfish shelf and slope bottom 
trawl survey from triennially to annually, leveraging available 
resources by cooperatively working with the fishing industry. 
Specifically, contracting with private commercial fishing vessels to 
conduct the surveys. According to Northwest Center officials, working 
collaboratively with the fishing industry has afforded fishermen the 
opportunity to become stakeholders in the data collection process. 

* Extended the geographic range of the groundfish shelf and slope 
bottom trawl survey. The surveys are now coastwide from Cape Flattery, 
Washington to the Mexican border, adding over 300 more miles along the 
southern California coast. Previous surveys ended at Morro Bay, 
California. 

Efforts continue to communicate the strategies needed to improve stock 
assessments and to augment existing programs aimed at developing future 
stock assessment scientists. For example, through the Pacific Fishery 
Management Council, Northwest Center staff meet with their 
constituents, such as representatives from state agencies and the 
fishing industry, to discuss groundfish management issues. In addition, 
the Northwest Center organized a series of public meetings to discuss 
new initiatives that affect the stock assessment program. For example, 
the Northwest Center held public meetings in several communities along 
the Pacific coast to discuss implementation of the Observer Program--a 
program designed to collect information about discarded fish for the 
non-hake west coast groundfish fleets. [Footnote 17]

Finally, the Northwest Center now participates in NMFS' National Sea 
Grant program to augment a Northwest Center-supported graduate study 
program at the University of Washington that trains stock assessment 
scientists. The Sea Grant program provides fellowships for students 
interested in marine research, such as stock assessment methodology and 
marine resource economics. The Northwest Center plans to employ two Sea 
Grant students during the summer of 2004. 

Further Actions Are Needed to Improve Stock Assessments: 

While the Northwest Center has implemented some recommendations aimed 
at improving stock assessments, it has not yet fully implemented many 
others. These recommendations include collecting additional types of 
data, such as ecosystem and recruitment information; improving data 
quality, such as calibrating survey vessel equipment; and increasing 
training opportunities for Northwest Center staff. Also, NMFS has not 
acted on the task force recommendation to combine the improvement plan 
and its complimentary plans into a comprehensive plan that provides 
integrated program initiatives. 

According to the improvement plan, additional types of data will allow 
NMFS to further test and validate model assumptions, thereby increasing 
the reliability of the stock assessments. The improvement plan further 
states that information derived from ecosystem research and recruitment 
surveys is essential if assessments are to meet the national standards 
of "next generation" assessments or Tier 3 status. According to 
Northwest Center officials, ecosystem information and coastwide 
recruitment surveys are two of the most critical data sets needed to 
ensure continuous improvement of groundfish stock assessments. The 
Northwest Center conducts ecosystem research as part of its Science for 
Ecosystem-based Management Initiative. Understanding the complex 
ecological relationships between fish and the environment within which 
they exist provides a better understanding of the effects of the 
ecosystem on the groundfish fisheries and the scientific knowledge 
needed to make informed ecosystem-based management decisions. Although 
research is ongoing to develop ecosystem information, only a limited 
amount of the data is collected and used in stock assessments. For 
example, ecosystem data are collected during shelf and slope bottom 
trawl surveys as time and resources allow. However, this information is 
not widely incorporated into stock assessments. For the five species we 
reviewed, only the boccacio assessment used ecosystem data--information 
on the temperature of the ocean's surface. 

According to Northwest Center officials, the collection of ecosystem 
data is limited because the relatively small size of the commercial 
vessels used in the shelf and slope bottom trawl surveys cannot support 
the number of researchers needed to effectively conduct comprehensive 
ecosystem research and collection activities. Furthermore, the 
implementation of comprehensive ecosystem research and data collection 
programs is contingent upon the funding of a dedicated research vessel 
for west coast surveys. Northwest Center officials said they are to 
receive a dedicated research vessel sometime during calendar years 2008 
or 2009, at the earliest. 

Better recruitment information for Pacific groundfish is also needed 
because such information provides an early predictor of fish abundance, 
especially for species such as hake, where there is a great variation 
in recruitment. Northwest Center officials said that current 
recruitment surveys are limited because existing funds support only 
yearly surveys in selected areas. To achieve the best early predictions 
of stock status, these officials said, recruitment surveys should be 
coastwide and conducted twice a year. According to Northwest Center 
officials, 13 full-time staff are needed to expand these and other 
high-priority data collection efforts, such as surveys in untrawlable 
habitat and expanded acoustic surveys. 

The lack of quality data was identified in the improvement plan as an 
impediment to producing reliable stock assessments. For example, when 
equipment on different survey vessels are not calibrated, the data are 
not comparable, and trends may not be accurately determined. The 
Northwest Center is continuing its efforts to calibrate survey vessel 
equipment. 

The improvement plan also recommended that NMFS provide additional 
training to ensure that qualified NMFS staff are available now and in 
the future to conduct stock assessments and related activities. For 
example, the plan recommended the development of a comprehensive 
training program and more professional developmental opportunities for 
NMFS' scientific staff. Northwest Center officials said they try to 
meet the training and professional development needs of their 
scientific staff. However, to date they have focused on developing 
external training programs, such as the University of Washington 
graduate program, to develop stock assessment scientists for the future 
and have yet to develop a comprehensive training program for in-house 
stock assessment scientists. 

Finally, the improvement plan recommended that NMFS prepare a 
comprehensive plan that combines the improvement plan with other 
complementary plans, such as the NOAA Fisheries Data Acquisition Plan 
and the NMFS Social Sciences Plan. A comprehensive plan would allow 
NMFS to better integrate and coordinate program initiatives for 
improving stock assessments. For example, the acquisition plan--the key 
complementary plan to the improvement plan--identifies the need for 
fishery research vessels to satisfy NMFS' data collection needs. 
Although the improvement plan includes the number of staff that would 
participate in data collection surveys, it does not contain the capital 
and operating costs of the research vessels. Similarly, the staffing 
requirements for augmenting the social sciences capabilities of NMFS to 
conduct economic analyses is represented in the sciences plan and not 
in the stock improvement plan. A NMFS official said that other program 
priorities, such as conducting more stock assessments and improving 
data collection activities, have precluded them from developing a 
comprehensive plan. 

Remaining Improvements Estimated to Cost at Least $8.9 Million: 

According to NMFS funding and budget requests, the Northwest Center 
needs at least $8.9 million to complete ongoing and planned 
improvements to the stock assessments for Pacific groundfish. However, 
the actual cost of implementing remaining improvements to Pacific 
groundfish stock assessments may be even higher because the Northwest 
Center's budget requests primarily reflect the amount of money the 
Center believed it could realistically obtain, rather than the actual 
cost of the improvements. 

Remaining Improvements Are Estimated to Cost at Least $8.9 Million, but 
Estimate Is Likely Understated: 

According to NMFS, the Northwest Center needs at least $8.9 million to 
complete ongoing and planned improvements for Pacific groundfish stock 
assessments: $2.6 million that NMFS' Northwest Center requested but did 
not receive between fiscal years 2001 to 2003 and $6. 3 million the 
Center requested for fiscal years 2004 and 2005. Specifically, as shown 
in table 2, the Northwest Center records have identified the following 
funding needs: 

* $7.7 million to improve the types of data used, including $2.4 
million for surveys of untrawlable waters, $2.1 million to expand 
acoustic and recruitment surveys, and $3.2 million to collect ecosystem 
data; and: 

* $1.2 million to improve the quality of data used in stock 
assessments, including $600,000 to enhance the calibration of vessel 
equipment; $525,000 to develop and implement methods to collect 
information on stock identification, structure, and movement; and 
$75,000 to standardize trawl survey procedures. 

Table 2: Pacific Groundfish Stock Assessment Improvement Funds 
Requested and Received by Data Type and Quality, Fiscal Years 2001-
2005: 

Dollars in millions. 

Types of data used: Surveys for untrawlable waters: Nearshore surveys 
(2003); 
Requested: $1.10; 
Funded: $0.50; 
Unfunded requests: $0.60.

Types of data used: Surveys for untrawlable waters: Untrawlable habitat 
surveys[A] (2003); 
Requested: $0.30; 
Funded: $0.20; 
Unfunded requests: $0.10.

Types of data used: Surveys for untrawlable waters: Longline groundfish 
fishing tag surveys[A] (2003); 
Requested: $0.40; 
Funded: $0; 
Unfunded requests: $0.40.

Types of data used: Surveys for untrawlable waters: Augment trawl 
survey in nearshore (2004); 
Requested: $0.51; 
Funded: N/A; 
Unfunded requests: $0.51.

Types of data used: Surveys for untrawlable waters: Fixed-gear survey 
(2004); 
Requested: $0.78; 
Funded: N/A; 
Unfunded $0.78.

Subtotal; 
Unfunded requests: $2.4.

Types of data used: Expand acoustic and recruitment surveys: 
Recruitment surveys (2003); 
Requested: $0.40; 
Funded: $0.20; 
Unfunded requests: $0.20.

Types of data used: Expand acoustic and recruitment surveys: Acoustic 
studies of Southern California bight rockfish[A] (2003); 
Requested: $0.20; 
Funded: $0; 
Unfunded requests: $0.20.

Types of data used: Expand acoustic and recruitment surveys: Coastwide 
augmentation of recruitment survey by Southwest Fisheries; 
Science Center (2004); 
Requested: $0.45; 
Funded: N/A; 
Unfunded requests: $0.45.

Types of data used: Expand acoustic and recruitment surveys: Acoustic 
surveys in inaccessible habitat (2005); 
Requested: $1.25; 
Funded: N/A; 
Unfunded requests: $1.25.

Subtotal; 
Unfunded requests: $2.1.

Types of data used: Ecosystem data: Coastwide observing system (2003); 
Requested: $0.80; 
Funded: $0.60; 
Unfunded requests: $0.20.

Types of data used: Ecosystem data: West coast observing system[A,B] 
(2003); 
Requested: $0.46; 
Funded: $0.20; 
Unfunded requests: $0.26.

Types of data used: Ecosystem data: Habitat-specific resource surveys 
and fishing gear impacts (2004); 
Requested: $1.45; 
Funded: N/A; 
Unfunded requests: $1.45.

Types of data used: Ecosystem data: Ecosystem studies of ocean 
productivity and climate impacts (2004); 
Requested: $1.28; 
Funded: N/A; 
Unfunded requests: $1.28.

Subtotal; 
Unfunded requests: $3.2.

Total; 
Unfunded requests: $7.7.

Quality of data used: Fishing survey vessel calibration (2003); 
Requested: $0.60; 
Funded: $0; 
Unfunded requests: $0.60.

Quality of data used: Trawl standardization (2003); 
Requested: $0.45; 
Funded: $0.38; 
Unfunded requests: $0.08.

Quality of data used: Stock identification, structure, and movement 
(2004); 
Requested: $0.53; 
Funded: N/A; 
Unfunded requests: $0.53.

Total; 
Unfunded requests: $1.2.

Other: Stock assessment training and coordination (2003); 
Requested: $0.35; 
Funded: $0.15; 
Unfunded requests: $0.20.

Other: Stock assessment training[A,B] (2003); 
Requested: $0.30; 
Funded: $0.50; 
Unfunded requests: $-.20.

Other: Stock assessment training (2003); 
Requested: $0.29; 
Funded: $0.29; 
Unfunded requests: $0.

Other: Stock assessment training (2002); 
Requested: $0.29; 
Funded: $0.29; 
Unfunded requests: $0.

Other: Stock assessment training (2001); 
Requested: $0.29; 
Funded: $0.29; 
Unfunded requests: $0.

Total; 
Unfunded requests: $0.

Total unfunded requests; 
Unfunded requests: $8.9.

Source: GAO analysis of NMFS data.

Notes: These figures exclude the cost of constructing a new fishing 
research vessel, which is currently designated to be home ported at and 
used primarily by the Northwest Center for surveys of fish species, 
such as Pacific groundfish. This vessel has not yet been funded and 
construction has not yet begun. 

The amounts requested by the Northwest Center in fiscal years 2004 and 
2005 are preliminary budget requests. 

[A] Includes funding for the Southwest Fisheries Science Center 
assistance to the Northwest Center's Pacific groundfish stock 
assessment improvement program. 

[B] Represents requests and funding for all marine species--not Pacific 
groundfish alone. NMFS was unable to identify what portion of these 
funds went solely to groundfish. 

[End of table]

The Northwest Center did not receive its full funding request, in part, 
because NMFS did not receive all the funding it had requested. Between 
fiscal years 2001 and 2003, NMFS received $20.6 million (80 percent of 
its request) in additional funding to implement improvements for all 
marine stock assessments. NMFS allocated $3.6 million (58 percent of 
funds the Northwest Center requested) to the Northwest Center for 
improving Pacific groundfish stock assessments, resulting in a $2.6 
million shortfall in the Center's request. This shortfall occurred in 
part because of NMFS' need to balance the requests of its six science 
centers against its program priorities and the available 
funds. [Footnote 18] According to NMFS officials, their goal is to 
achieve parity among the science centers in terms of their capability 
to conduct scientific work, such as stock assessments. 

Estimated Costs May Understate Actual Cost of Remaining Improvements: 

The $8.9 million needed to implement remaining recommended improvements 
is probably understated because the Northwest Center's budget requests 
primarily reflect the amount of money the Center officials believed 
they could realistically obtain, rather than the amount the 
improvements would actually cost, according to NMFS officials. The 
Northwest Center's budget requests for fiscal years 2004 and 2005 are 
preliminary requests submitted before the Northwest Center received 
its fiscal year 2003 funding. [Footnote 19] Consequently, the Northwest 
Center will likely submit revised budget requests for fiscal years 2004 
and 2005 that account for both its unfunded needs from fiscal years 
2001 through 2003 and items that were unexpectedly funded in fiscal 
year 2003. Moreover, the fiscal year 2004 and 2005 preliminary budget 
requests do not incorporate any unanticipated problems or data gaps 
that have developed since the Northwest Center submitted its 
preliminary requests. 

According to NMFS officials, NMFS' science centers, including the 
Northwest Center, primarily make and justify their funding requests in 
response to how much money Congress appropriates. After Congress passes 
NMFS' budget, NMFS asks its science centers to reassess and detail how 
much new money each needs to implement science center programs, such as 
marine stock assessment improvements. According to NMFS officials, it 
is unrealistic for a science center to request more funds than are 
available in its appropriation, even if it needs more. While NMFS' 
Northwest Center requested $6.2 million to implement improvements to 
Pacific groundfish stock assessments between fiscal years 2001 and 
2003, NMFS' 2001 West Coast Groundfish Research Plan estimated that 
almost twice as much money would be needed--approximately $11.7 million 
in new funding--to implement top-priority improvements to Pacific 
groundfish stock assessments. 

NMFS is now updating its plan and cost estimates for improving Pacific 
groundfish stock assessments. Using key findings from its December 2003 
review of the groundfish program, the Northwest Center plans to update 
its groundfish research plan, last published in 2001. According to 
NMFS, the updated groundfish research plan should be completed in late 
2004 and is designed to (1) provide a comprehensive framework for 
Pacific groundfish, (2) identify some of the greatest information gaps, 
and (3) provide guidance for setting priorities on work to fill these 
gaps. In addition, the updated plan will estimate how much such 
improvements will cost. 

Conclusions: 

Stock assessments are the key to effectively managing fisheries. They 
provide estimates of the species population, which NMFS uses to set 
harvest limits that allow for sustainability and/or recovery of the 
species. While stock assessment results often change from assessment 
report to assessment report, the more types of information used in the 
assessments, such as recruitment surveys and ecosystem studies, and the 
greater the accuracy and quality of the data, such as scientifically 
designed and collected data, the more reliable the assessment results. 
However, the Pacific groundfish assessments we reviewed did not (1) use 
scientifically designed and collected NMFS data of sufficient scope and 
accuracy, such as survey data on the abundance of groundfish residing 
in rocky, untrawlable habitats; (2) subject the non-NMFS data used to a 
standard process for assessing its reliability; and/or (3) identify the 
uncertainties of the assessments total biomass estimates. As a result, 
the reliability of the five assessments is questionable. Without 
reliable assessments, fishery managers may reach erroneous conclusions 
and take actions that could adversely affect the fishing industry 
economically or adversely affect the recovery and sustainability of the 
fishery resources. Moreover, without a comprehensive, integrated 
improvement plan, funding requests and planned actions to improve the 
stock assessments may not be coordinated, jeopardizing successful and 
timely implementation of assessment improvements. 

Recommendations for Executive Action: 

To improve the reliability of Pacific groundfish stock assessments, we 
recommend that the Secretary of Commerce require the Director of 
National Marine Fisheries Service to take the following four actions: 

* Continue efforts to collect more types of data, such as data obtained 
from surveys in rocky, untrawlable waters, recruitment surveys, and 
ecosystem studies, for groundfish assessments where reliable data are 
now lacking. 

* Establish a standard approach that requires that non-NMFS data used 
in stock assessments be evaluated for its reliability, and continue 
efforts to implement the task force's recommendations to improve data 
quality. 

* Require that stock assessment reports clearly present the 
uncertainties in the assessments, such as the margin of error 
associated with species biomass estimates. 

* Develop a comprehensive plan that integrates the NMFS stock 
assessment improvement plan with other NMFS plans to ensure that stock 
assessment improvement actions and budget requests are coordinated. 

Agency Comments and Our Evaluation: 

We provided the Department of Commerce with a draft of this report for 
review and comment. We received a written response from the Under 
Secretary of Commerce for Oceans and Atmosphere that included comments 
from the National Oceanic and Atmospheric Administration (NOAA). NOAA 
generally agreed with the report's accuracy and concurred with the 
reports recommendations. However, NOAA said it was concerned about the 
report's conclusion--that the reliability of stock assessments is 
questionable for the five species reviewed--because it could be 
misconstrued to infer that the assessments are unreliable for use in 
managing the west coast groundfish fishery. In this regard, NOAA 
provided additional comments to show the usefulness of the assessments, 
even if some of the input data used in the assessments contained 
errors. We stand by our conclusions that the five stock assessments we 
reviewed were questionable because the input data were insufficient 
and/or potentially inaccurate and that four of the assessment reports 
did not present the uncertainties associated with the biomass 
estimates. Nonetheless, we added language to the report to address 
NOAA's concern. Specifically, we expanded upon the fact that NMFS used 
the best information available at the time the stock assessments were 
conducted by adding information on the importance of the assessments to 
effectively manage the fisheries. Without these stock assessments, NMFS 
and fishery managers would have very limited information on which to 
base fishery management decisions. 

NOAA agreed with the report recommendation to continue collecting more 
types of data for groundfish assessments where reliable data are now 
lacking. NOAA said that the reliability of stock assessments will be 
improved if NMFS survey efforts are expanded and additional NMFS 
fishery data are collected. NOAA said NMFS places a priority on these 
improvements and will continue efforts to address this and other 
recommendations to improve the collection of fishery data as funding 
becomes available. 

NOAA also agreed with the report recommendation to establish a standard 
approach to evaluate the reliability of non-NMFS data used in stock 
assessments and continue efforts to improve data quality. NMFS said, 
through its west coast fishery science centers, it participates on 
interagency data committees, to develop quality assurance protocols and 
to assess the quality of non-NMFS data. NOAA agreed that it is 
important to ensure that these interagency data committees continue to 
highlight the need for standardized quality control procedures for the 
collection of data. 

NOAA agreed with the report recommendation to clearly present the 
uncertainties in the stock assessments. NOAA said that quantifying 
uncertainty of stock assessments is important to sound decision-making 
by providing more information about the assessment, although this 
quantification does not reduce the uncertainty in the assessment 
itself. While the methods used and the completeness of the uncertainty 
characterization varied from assessment to assessment, NOAA said it is 
desirable to have both a quantitative analysis of model uncertainty and 
an evaluation of the consequences of alternative model scenario. 

Finally, NOAA agreed with the report recommendation to develop a 
comprehensive plan that integrates the stock assessment plan with other 
NMFS plans to ensure that improvement actions and budget initiatives 
are coordinated. NOAA said that while much remains to be done, long-
term planning efforts and coordination among field and headquarters are 
ongoing, and NOAA is committed to these actions. 

NOAA's comments and our detailed responses are presented in appendix II 
of this report. NOAA also provided technical comments that we 
incorporated in this report as appropriate. 

As agreed with your offices, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 7 days 
from the report date. At that time, we will send copies of this report 
to the Secretary of Commerce and the Director of the National Marine 
Fisheries Service. We will also provide copies to others upon request. 
In addition, the report will be available at no charge on the GAO Web 
site at [Hyperlink, http://www.gao.gov].

If you or your staff have any questions about this report, please call 
me at (202) 512-3841 or Keith Oleson at (415) 904-2218. Key 
contributions to this report are listed in appendix III. 

Signed by: 

Anu K. Mittal, 
Director, Natural Resources and Environment: 

[End of section]

Appendixes: 

Appendix I: Objectives, Scope, and Methodology: 

We reviewed National Marine Fisheries Service (NMFS) stock assessments 
for five species of Pacific groundfish: Pacific hake (Pacific whiting) 
as well as four types of rockfish--bocaccio, canary, darkblotched, and 
yelloweye. Specifically, for these five species you asked us to (1) 
assess the reliability of NMFS' stock assessments, (2) identify which 
relevant recommendations from the stock assessment improvement plan 
have been implemented and which have not, and (3) identify the 
estimated costs associated with planned and ongoing improvements to 
groundfish stock assessments. We did not review the stock assessments 
of any of the other west coast Pacific groundfish species, thus the 
information contained in this report pertains to the five species we 
reviewed unless stated otherwise. 

For all three objectives, we reviewed key laws and agency reports and 
interviewed officials from NMFS, including officials from the Northwest 
Fisheries Science Center, which has lead responsibility for conducting 
Pacific groundfish stock assessments with assistance from other west 
coast science centers. 

To assess the reliability of the Pacific hake, bocaccio, canary, 
darkblotched, and yelloweye stock assessments, we examined 
methodological and administrative documents developed by NMFS and 
others to support the groundfish data collection, maintenance, and 
assessment process. We reviewed the controls over stock assessment 
data, the types of fish population surveys used, and recent Pacific 
groundfish stock assessment studies (2002 and 2004 studies for Pacific 
hake, 2002 and 2003 studies for bocaccio, 2002 study for canary, 2000 
and 2003 studies for darkblotched, and 2001 and 2002 studies for 
yelloweye). We examined whether and to what extent NMFS has processes 
and procedures in place to ensure the reliability of data used in the 
Pacific groundfish stock assessments. We reviewed the stock assessment 
reports and determined whether they articulated the level of 
uncertainty in the assessment model estimates. We interviewed an array 
of government officials and fisheries experts, including the Oregon 
Department of Fish and Wildlife, the Washington Department of Fish and 
Wildlife, the California Department of Fish and Game, the Pacific 
Fishery Management Council, the Pacific States Marine Fisheries 
Commission, environmental groups, and industry associations, as well as 
fishermen and academics. We did not simulate NMFS' stock assessment 
models nor evaluate the mathematical and statistical methodologies used 
in the models for Pacific hake, bocaccio, canary, darkblotched, and 
yelloweye. 

To identify the relevant recommendations to improve stock assessments 
that NMFS has implemented and has not implemented, we reviewed agency 
reports on marine fisheries stock assessments, strategic planning, and 
data collection. We also interviewed officials from the Oregon 
Department of Fish and Wildlife, the Washington Department of Fish and 
Wildlife, the California Department of Fish and Game, the Pacific 
Fishery Management Council, and the Pacific States Marine Fisheries 
Commission as well as environmental groups, industry associations, 
fishermen, and academics. 

To determine the estimated costs associated with NMFS' planned and 
ongoing improvements to Pacific groundfish stock assessments, we 
reviewed relevant budget requests and funding documents for fiscal 
years 2001 through 2005 and interviewed National Oceanic and 
Atmospheric Administration officials. We did not evaluate the accuracy 
of NMFS' budget requests for specific project items but rather used the 
amounts NMFS requested for these project items to estimate the total 
additional costs of implementing the planned and ongoing improvements 
to Pacific groundfish stock assessments. 

We conducted our review from May 2003 through April 2004 in accordance 
with generally accepted government auditing standards. 

[End of section]

Appendix II: Comments From the Department of Commerce: 

UNITED STATES DEPARTMENT OF COMMERCE 
The Under Secretary of Commerce for Oceans and Atmosphere 
Washington, D.C. 20230:

MAR 13 2004:

Ms. Anu K. Mittal: 
Director, Natural Resources and Environment:
United States General Accounting Office: 
441 G Street, NW:

Washington, D.C. 20548:

Dear Ms. Mittal:

Thank you for the opportunity to review and comment on the General 
Accounting Office's draft report entitled "Pacific Groundfish: 
Continued Efforts Needed to Improve Reliability of Stock Assessments," 
GAO-04-606. Enclosed is the National Oceanic and Atmospheric 
Administration's comments on the draft report. 

These comments were prepared in accordance with the Office of 
Management and Budget Circular A-50. 

Sincerely,

Signed by: 

Conrad C. Lautenbacher, Jr. Vice Admiral, U.S. Navy (Ret. ) 
Under Secretary of Commerce for Oceans and Atmosphere:

Enclosure:

NOAA Comments on the Draft GAO Report Entitled "Pacific Groundfish: 
Continued Efforts Needed to Improve Reliability of Stock Assessments" 
(GAO-04-606/June 2004):

Comment on the Scope of the Report:

The report title implies the scope of this report is Pacific Coast 
wide, but the focus is heavily weighted to survey and assess activities 
of the Northwest Fisheries Science Center (NWFSC). The report uses the 
acronyms NMFS and NWFSC interchangeably. Inclusion of information from 
the Southwest and the Alaska Fisheries Science Centers is highly 
inconsistent. Insufficient information was included to fully portray 
the scope, strengths and weaknesses of their past and present 
contributions to Pacific Coast groundfish work. However, inclusion of 
bocaccio in the set of GAO-reviewed stocks and references to NWFSC and 
AFSC work were frequent enough to preclude casting the report as 
exclusively pertaining to the NWFSC. We recommend the report be 
characterized as "A review of the NWFSC's groundfish program including 
aspects of West Coast groundfish research conducted by other West Coast 
Science Centers." This is more accurate than referring to it either as 
a review of the NWFSC or NOAA Fisheries' West Coast Groundfish program. 
It is more than the former, but less than the latter. This scope should 
be made clear in the introduction of the report. 

Recommended Changes for Factual Information:

GAO Highlights, par. 1, line 9: Replace 2002 with 2003. 

GAO Highlights, par. 1, lines 14-16: Change to read "The assessors who 
reviewed the quality of the non-NMFS data found errors that made "some 
of the data unusable or impaired the reliability of "certain" stock 
assessments." See our response to GAO recommendation 2. Also, this 
statement does not pertain to the hake assessment. 

GAO Highlights, par. 2, line 10: This statement is not correct. NMFS 
has increased a limited number of recruitment surveys and work on 
untrawlable habitat. 

Page 1, par. 2: NMFS also uses assessments conducted by independent 
scientists in state resource agencies. 

Page 1, par. 2, line 4: Should read, "Northwest Fisheries Science 
Center," as statement is not correct for NMFS generally. 

Page 1, par. 2, line 5: ...(1) the shelf and slope [insert "bottom"] 
trawl survey...

Page 2, par. 1, line 1: Delete "In response," replace with "In 1999," 
...review the stock assessment process "across the agency."

Page 2, par. 1, line 4: ...for conducting "some of the" Pacific ...

Page 2, par. 1, line S: Add citation for October 2001 Marine Fisheries 
Stock Assessment Improvement Plan NMFS. 2001. Marine Fisheries Stock 
Assessment Improvement Plan. Report of the National Marine Fisheries 
Service National Task Force for Improving Fish Stock Assessments, 
Second Ed. (Rev. ). U.S. Dep. Commerce, NOAA Tech. Memo. NMFS-F/SPO-56, 
69 p. , 25 appendices. 

Page 2, par. 3, line 2: The publication date for the canary assessment 
was 2002. 

Page 3, par. 1, line 4: Add "1998" National Research Report*, and 
footnote citation: "*NRC 1998. Improving Fish Stock Assessments. 
National Research Council. National Academy Press. Washington, D. C. 177 
p."

Page 3, par. 1, lines 7-11: These statements are incorrect because NMFS 
had and used fishery-independent data for all but the yelloweye 
rockfish assessment. Revision to this page should be made in line with 
the general comments we have made regarding use of surveys in 
assessments. 

Page 3, par. 1, line 13: Replace 2002 with 2003. 

Page 4, par. 2: There is no clarification of budget needs as NMFS-wide 
versus NWFSC-specific. 

Page 6, par. 2, bullets 3 and S, and footnote 2: The NWFSC and SWFSC 
have joint responsibility for salmon fishery and ESA responsibility in 
freshwater rivers and streams in Washington ...etc. and California. 

Page 8, bullet 1: Stock abundance ...stock's total "size or weight." 
Delete "population."

Page 8, bullet S: Recruitment research ...abundance of juvenile "and 
larval" fish...

Page 8, paragraph after bullets: The last phrase in the sentence is not 
correct. It should read, "For each stock assessment, a review panel, 
consisting of NMFS scientists and outside experts, independently 
reviews the methodology of the assessment and works with the assessor 
to ensure their comments are adequately addressed."

Page 8, last par. , line 3: Seven species of rockfish have been declared 
overfished. The other three are cowcod, widow rockfish, and Pacific 
Ocean perch. 

Page 9, bullet 1, line 2: Replace "11 other species of hake" with "15 
species of the genus Merluccius (hakes)*" [could include a new 
reference footnote - *Source: www. fishbase. org"]

Page 12, line 14; also page 14, full par. 1: Our general comments 
describe how the data situation for these species is not nearly as 
limited as described here. Certainly bocaccio has more fishery-
independent data than acknowledged here. Obtaining data from 
untrawlable habitat will improve the precision of the canary, bocaccio 
and darkblotched assessments and, perhaps more importantly, it will 
enable assessments for other rockfish that occur primarily on these 
habitats. 

Page 13, Table 1 and par. 1: Bocaccio clearly rates a "yes" for NMFS-
collected data. The Ca1COFI larval abundance data has been collected on 
a rigorous scientific basis by NMFS (SWFSC) for over 50 years. In 
addition, our general comments clarify the degree to which the survey 
data are not critically limited for darkblotched and canary rockfish. 

Page 13, Table 1: Highly standardized protocols are used for collecting 
the fishery-dependent data, including the non-NMFS data for rockfish. 
Our general comments show at least hake should be rated as a "yes" in 
this category. 

Page 13, par. 1, line 3: ...slope [insert "bottom trawl"] survey...

Page 13, par. 1, lines 4-S: The comma in the phrase "include, larval" 
should probably be deleted. The phrase "The non-NMFS data assessors use 
include...." is slightly awkward. Could it be replaced with "Assessors 
use non-NMFS data such as..."?

Page 14, full par. 1, line 10: Replace 2002 with 2003. 

Page 15, par. 1, line 1: Replace "lives in a different habitat than the 
other four species," with "live primarily in mid-water, above the sea 
floor, and so..."

Page 15, footnote 9: Reposition on page 14; replace citation as 
follows: Zimmerman, M. 2003. Calculation of untrawlable areas within 
the boundaries of a bottom trawl survey. Can. J. Fish. Aquat. Sci. 
60:657-669. 

Page 16, full par. 1, line 2: Introduces the NMFS 2001 Stock Assessment 
Improvement Plan in the document text. It would be helpful to also 
introduce the acronym "Marine Fisheries SAIP" and then use in the 
following pages. There are many references after this point using the 
phrase "improvement plan" and it seems likely all of these refer to the 
SAIP, but there also is the West Coast Groundfish Research Plan. 

Page 16, full par. 2, line 2: Replace "health" with "stock abundance."

Page 16, footnote 10: Reposition on page 15. 

Page 17, last par. , line 3: ...tiers of assessment [insert 
"improvements"] are as...

Page 19, par. 1, line 2: ...frequent acoustic [insert ","] and shelf 
and slope [insert "bottom trawl"] surveys. 

Page 19, par. 1: While this list of actions may accurately reflect what 
the NWFSC did to improve data quantity, it is not a comprehensive list 
of all that has been done to improve data quantity coast wide, and 
especially omits projects initiated by the SWFSC, some joint with the 
NWFSC. 

Page 21, par. 1: While this may represent the NWFSC's ecosystem science 
programs, it is not an accurate portrayal of the coast wide program. 
For example, in the SWFSC, both the Ca1COFI survey and the pelagic 
juvenile rockfish survey collect comprehensive environmental/ecosystem 
data, and the SWFSC's Pacific Fisheries Environmental Laboratory is 
wholly devoted to the conduct of environmental research in a fisheries 
context. 

Page 21, par. 2, line S: Replace "is" with "are."

Page 21, par. 2, line 6: Substitute ...dedicated research vessel for 
West Coast surveys. 

Page 24, Table 2: Footnote b only refers to requests and funding for 
the SWFSC. The NWFSC separated groundfish funding from non-groundfish 
funding. 

Page 26, last sentence of first full paragraph: This is somewhat 
misleading. The groundfish research plan was developed with all the 
NMFS Centers as a coastwide plan. So the term "its" in this sentence is 
misleading. In the next paragraph, the coast wide research plan will be 
updated (not just the NWFSC plan). 

Page 26, par. 4, line 3: Suggest replacing "survivability" with 
"sustainability."

Page 27, par. 1, line 4: (1)...etc. See comment for page 12 above. 

Page 27, par. 1, line I1: "...survivability of the fishery 
["resources"? or " industry" ?]

General Comments:

Reliability of Assessments:

The GAO report states, "The reliability of NMFS stock assessments is 
questionable for the Pacific hake and four rockfish species we 
reviewed, although the assessments were based on the best information 
available at the time they were conducted." This statement does not 
adequately convey the different degrees of precision associated with 
each of these assessments and could easily be misconstrued to mean all 
these assessments are an unreliable basis for management of the West 
Coast groundfish fishery. NOAA doubts GAO intended such an extreme 
conclusion from their findings, and NOAA would like to take this 
opportunity to provide additional information regarding these 
assessments: namely, that all have passed a scientific review and all 
are serving as the basis for formal status determination and management 
of these stocks. Our response to GAO recommendation 2 demonstrates 
quality assurance for non-NMFS data is not absent; thus, the 
assessments are not unreliable due to use of these data. Our response 
to GAO recommendation 3 indicates precision is quantified to the extent 
feasible today, is on par with state-of-the-science assessments 
elsewhere, and is rapidly improving. NOAA acknowledges quantifying 
precision of these assessment results would provide more information 
about the assessment, and expanded and improved data would improve the 
certainty of the assessment results and forecasts. Nevertheless, there 
is no reason to believe these assessments are inadequate for guiding 
this fishery. In addition to NOAA's responses to GAO recommendations 2 
and 3, NOAA provides the following information with regard to the 
reliability of these stock assessments:

Surveys as Stock Trend Indicators:

Stock. assessments require data on the relative trend in stock 
abundance and on the removals (catch). Both are needed in order to 
produce the target assessment results. Although catch data alone are 
not a substitute for survey trend data, in some circumstances with 
fishery effort information, the catch per unit of fishing effort can be 
standardized to provide an index of the trend in stock abundance. The 
National Research Council's report on "Improving Fish Stock 
Assessments" and NOAA's own studies indicate stock assessments should 
include a standardized, scientifically-designed survey of fish 
abundance so that changes observed in the survey confidently reflect 
true changes in the stock's abundance and not some confounding factor. 
These conditions are better met with the controlled conditions of a 
fishery-independent survey than with fishery catch per effort. 

NMFS and Non-NMFS Data:

The GAO report categorized the five reviewed assessments according to 
the availability of NMFS-collected survey data, often abbreviated to 
just "NMFS data." While NMFS collects nearly all the fishery-
independent survey data for West Coast groundfish and the states 
collect much of the fishery-dependent data on catch and fishing effort, 
it is more pertinent to focus on the degree of standardization of the 
survey data than on the source. NMFS is firmly committed to providing 
such surveys on a long-term basis, but useful surveys can be conducted 
by non-NMFS organizations. The desire for standardized, fishery-
independent surveys should not be equated with a concern about basing 
an assessment on non-NMFS data. Also, less precise assessments can be 
conducted with only non-NMFS fishery-dependent data as long as there is 
adequate standardization and quality assurance. 

Assessments:

The Pacific hake assessment is one of the most data-rich and 
sophisticated assessments in the nation. It is based upon a highly 
calibrated acoustic survey of hake abundance and a very complete system 
for monitoring the hake catch. Even if the portion of the hake catch 
data provided by the states had some major shortcomings (which is 
unlikely), this would have little impact on the overall assessment's 
precision and certainly would not make it unreliable. More frequent 
Pacific hake acoustic surveys and a more geographically extensive 
recruitment survey will improve the timeliness and precision of short-
term forecasts, but the overall assessment of the status of the stock 
certainly is highly reliable according to all scientific reviews. 

The bocaccio, canary, and darkblotched rockfish assessments all obtain 
adequate abundance trend information from the NMFS bottom trawl surveys 
because these species commonly occur in these surveys. The bocaccio 
assessment includes several other fishery-independent sources of 
abundance trends, particularly the NMFS-SWFSC juvenile rockfish survey 
providing an index of recruitment, and the SWFSC led CaICOFI survey 
provides a larval index of spawning biomass. The fact that the bottom 
trawl survey cannot access the roughest habitat does not invalidate its 
usefulness as an index of relative changes in the overall abundance of 
these three stocks, even if the density of fish in untrawlable habitat 
is typically different than the density on trawlable habitat assuming a 
random or consistent bias. New surveys, some based on new technology, 
providing information from untrawlable habitat will allow more direct 
evaluation of the total 
stock and calibration of the bias assumptions, but the current fishery-
independent survey data are likely to be sufficiently complete to 
determine these stocks have declined substantially into an overfished 
state. With improved and more frequent surveys and more timely 
assessments using these surveys, recovery of these stocks and possible 
declines in other stocks will be more accurate and precise and detected 
sooner. 

The yelloweye rockfish assessment cannot obtain relevant information 
from the NMFS bottom trawl survey because this species primarily occurs 
on rough habitat; instead, the yelloweye assessment relies upon the 
trend observed in recreational fishery catch per unit effort. Certainly 
fishery catch per effort is less standardized a data source than the 
NMFS bottom trawl survey so this is logically the least precise of the 
assessments reviewed by GAO, but use of fishery catch per effort is a 
standard practice for many important assessments where fishery-
independent surveys are impractical (such as highly migratory tunas). 
The yelloweye assessment contains a caveat regarding uncertainty in the 
calibration of the fishery catch per effort and it acknowledges the new 
effort to collect new kinds of survey data from the untrawlable 
habitat. The yelloweye assessment represents a rational use of 
available fishery-dependent data until sufficient new fishery-
independent data become available. 

On page 16, the GAO report references the doubling of estimated 
bocaccio biomass when the assessment was redone in 2003 and reiterates 
GAO's call for uncertainty ranges. Note, most of the doubling was due 
to factors (a new abundance index and a change in the level of natural 
mortality used in the assessment model) that would not be addressed in 
a standard statistical analysis of uncertainty. 

NOAA Response to GAO Recommendations:

Recommendation 1: "Continue efforts to collect more types of data, such 
as data obtained from surveys in rocky, untrawlable waters, recruitment 
surveys, and ecosystem studies, for groundfish assessments where 
reliable data are now lacking."

NOAA Response: NOAA agrees with this recommendation. The reliability of 
stock assessments will be improved if NMFS survey efforts are expanded 
and additional fishery-independent data are collected. NMFS places a 
priority on these improvements and through the Expand Stock Assessment 
- Improve Data Collection budget line has already started new budget 
initiatives and programmatic expansions in this area. The agency will 
continue to address the recommendations of the Marine Fisheries Stock 
Assessment Improvement Plan (and others) as funding becomes available. 

Recommendation 2: "Establish a standard approach that requires that 
non-NMFS data used in stock assessments be evaluated for its 
reliability, and continue efforts to implement the task force's 
recommendations to improve data quality."

NOAA Response: NOAA agrees with this recommendation. NMFS, through the 
West Coast Fisheries Science Centers, participates on the interagency 
data committees for the Pacific States Marine Fisheries Commission's 
Pacific Fisheries Information Network (PacFIN) for commercial West 
Coast fishery data, and the Recreational Fisheries Information Network 
(RecFIN), to:

develop quality assurance protocols and to assess the quality of 
fishery-dependent data. Quality assurance protocols were last outlined 
and specified in a NOAA Technical Memorandum (Commercial Fisheries Data 
Collection Procedures for the U.S. Pacific Coast Groundfish. 1997. 
Sampson and Crone (eds)). 

Each assessment relies upon these procedures and protocols to produce 
sound fishery-dependent data with good quality assurance. In cases 
where novel applications of these data are attempted, the assessment 
scientists will be more involved in preparation of the raw data and may 
find the data are inadequate for the' new application, such as catch 
per unit effort for a particular fishery sector. Rejection of certain 
data for such a new purpose does not invalidate the entire fishery 
monitoring program. Therefore, NOAH agrees it is important to assure 
the terms of reference of these interagency data committees continue to 
highlight the need for standardized quality control procedures for the 
collection of the data. NMFS membership on the PacFIN and RecFIN 
committees assures such protocols are monitored and implemented. 

Recommendation 3: "Require that stock assessment reports clearly 
present the uncertainties in the assessments, such as the margin of 
error associated with population estimates."

NOAA Response: NOAA agrees with this recommendation. Quantifying 
uncertainty of assessments is important for sound decision-making by 
providing more information about the assessment, although this 
quantification does not reduce the uncertainty in the assessment 
itself. Currently, the Scientific and Statistical Committee of the 
Pacific Fishery Management Council has terms of reference for stock 
assessments including requiring that the uncertainty be characterized. 
However, the methods used and the completeness of the uncertainty 
characterization has varied from assessment to assessment. It is 
desirable to have both a quantitative analysis of model uncertainty due 
to statistical goodness-of-fit within a selected model scenario, and an 
evaluation of the consequences of alternative model scenarios to 
investigate potential biases. Doing both for complex fishery stock 
assessment models is technically challenging and the West Coast 
Goundfish assessments are near the state-of-the-science. The GAO report 
focused on the statistical goodness-of-fit, which was available for the 
Pacific hake assessment, but this cannot be done routinely with the 
older generation models necessary for the types of data available for 
the four rockfish assessments reviewed by GAO. NMFS has only been able 
to work slowly on new model development at its current level of 
funding, but in 2004, it will roll out and review a next generation 
model suitable for rockfish assessments that will be capable of 
producing the desired uncertainty calculations. In addition, NMFS' 
cooperative research with the University of Washington is exploring 
methods to quantitatively combine uncertainty analyses across multiple 
model scenarios, thus achieving a new level of overall uncertainty 
characterization. NMFS will assist in developing new terms of reference 
more clearly outlining the requirement for uncertainty characterization 
and giving clear guidance on the methods to be used to calculate and 
report this uncertainty. 

Recommendation 4: "Develop a comprehensive plan that integrates the 
NMFS stock assessment improvement plan with other NMFS plans to ensure 
that stock assessment improvement actions and budget requests are 
coordinated."

NOAA Response: NOAA agrees with this recommendation. The NMFS Office of 
Science and Technology, together with all six regional Fisheries 
Science Centers, is responsible for developing budget initiatives for 
major program areas focusing on improving the scientific basis for 
fishery management. Long-term program development and field-
headquarters' coordination activities are continuing for the following: 
Expand Stock Assessments-Improve Data Collections, Fisheries 
Statistics (National Fisheries Information System), Economics and 
Social Sciences Research, Fisheries Observers, Fishery Oceanography, 
National Cooperative Research, and new NOAA Fishery Survey Vessel fleet 
construction. These multi-year program activities have been favorably 
reviewed by our external partners, constituents, and the Congress. 
Initial new funding has been received, but much remains to be done. 
NOAA is committed to continuing on this course. 

The following are GAO comments on NOAA's letter dated May 13, 2004. 

GAO Comments: 

1. We added clarifying language to the scope and methodology section of 
the report to clearly identify the species and activities covered by 
the review. 

2. We revised the report to show the publication date. 

3. We revised the report accordingly. 

4. We revised the report to clearly show that NMFS has not collected 
enough ecosystem data and that the frequency and range of recruitment 
surveys are limited. The statement does not address untrawlable 
habitat. 

5. We revised the report to clarify that NMFS "generally uses" NMFS's 
staff or contracts with outside experts. 

6. We revised the report to more clearly differentiate between NMFS as 
a whole and NMFS' Northwest Center in particular. We made similar 
revisions, as appropriate, throughout the report. 

7. We revised the report to specify "bottom" trawl survey. We made 
similar changes, as appropriate, throughout the report. 

8. We revised the report to include the year and scope of the task 
force review. 

9. We revised the report to indicate that the Northwest Center is 
responsible for coordinating groundfish stock assessments. 

10. We revised the report to include the citation. 

11. We revised the report to include the date and citation of the 
National Research Council report. 

12. The NMFS data used in the bocaccio, canary, and darkblotched 
assessments were limited because NMFS conducted its surveys in 
trawlable waters only. NMFS data were not available for untrawlable 
waters, which these species also inhabit. For this reason, we did not 
revise the report. 

13. We revised the report to clarify the shared responsibilities of the 
Northwest and Southwest Centers. 

14. We revised the report to include NOAA's recommended definition of 
stock abundance. 

15. We revised the report to include larval fish. 

16. We revised the report to clarify the role of the review panel. 

17. We revised the report to more explicitly distinguish the five 
species related to our report from other overfished Pacific groundfish. 

18. We revised the report to more clearly describe the distribution of 
Pacific hake. 

19. Bocaccio survey data for untrawlable habitats, as stated in comment 
12, was unavailable. For this reason, we did not change the report. 

20. NOAA commented that highly standardized protocols are used for 
collecting non-NMFS data (fishery dependent data) for rockfish. We 
found that although NMFS does have collection and quality assurance 
procedures for state-collected non-NMFS data, NMFS does not check or 
have a standard process to verify that these data have been reviewed 
for reliability. As discussed in our report, some assessors chose to 
review the raw data, while others did not. Assessors who voluntarily 
reviewed raw non-NMFS data found mistakes that either made some of the 
data unusable or could have impaired the accuracy of the stock 
assessments. For these reasons, we did not change the report. 

21. We revised the punctuation accordingly. 

22. We revised the report to clarify that Pacific hake live in mid-
water habitat. 

23. The footnote placement and citation are in accordance with GAO 
guidelines. For this reason, we did not change the report. 

24. We believe that our report has addressed this issue. By referring 
to the West Coast Groundfish Research Plan by its complete title, we 
adequately distinguish between the two reports. For this reason, we did 
not change the report. 

25. We changed "health" to "stock abundance.": 

26. The footnote placement is repositioned in report. 

27. We revised the report to include assessment "improvements.": 

28. As indicated in our report, we illustrate some of the actions that 
the Northwest Center took to improve data quantity and did not intend 
to provide a comprehensive list of all actions conducted to improve 
data quantity coast wide. However, we added footnote 16 to clarify the 
actions taken by the Southwest Center. 

29. The example we provided is not intended to be a comprehensive list 
of all ecosystem research conducted on the west coast. Instead, it 
illustrates the type of work the Northwest Center is conducting and the 
opportunities for improving ecosystem research. For this reason, we did 
not change the report. 

30. After reviewing the report we believe no change is required because 
of subject-verb agreement. 

31. We added clarifying language. 

32. We believe that table 2 notes "a" and "b" in our report already 
adequately address this issue. Annotations for projects that do not 
separate out groundfish funds occur only in items that are annotated as 
Southwest Center projects. For this reason, we did not change the 
report. 

33. We changed "survivability" to "sustainability.": 

34. We changed "survivability" to "sustainability" and added fishery 
"resources" for clarification. 

35. NOAA commented that GAO does not adequately convey the different 
degrees of precision associated with the stock assessments and GAO's 
conclusion that the reliability of the five assessments we reviewed is 
questionable and could easily be misconstrued to mean all these 
assessments are an unreliable basis for management of the west coast 
groundfish fishery. NOAA also commented that the five assessments GAO 
reviewed all passed scientific review and are serving as the basis for 
formal status determination and fishery management. Our report 
acknowledges that stock assessments are scientifically reviewed and are 
a key tool for managing fisheries. However, we found the reliability of 
the five assessments questionable for the three reasons we highlighted 
in our report, and we recommended actions on how to improve the 
reliability of the stock assessments. We added clarification to the 
report to show that stock assessments are a key tool for managing 
fisheries and are important in making decisions about setting harvest 
levels and developing plans to rebuild overfished stocks. 

NOAA also commented that quality assurance for non-NMFS data is not 
absent. As stated in our response number 20, we found that although 
NMFS does have collection and quality assurance procedures for state-
collected non-NMFS data, NMFS does not check or have a standard process 
to verify that these data have been reviewed for reliability. As 
discussed in our report, some assessors chose to review the raw data, 
while others did not. Assessors who voluntarily reviewed raw non-NMFS 
data found mistakes that either made some of the data unusable or could 
have impaired the accuracy of the stock assessments. For these reason, 
we did not change the report. 

36. NOAA commented that it is more pertinent to focus on the degree of 
standardization of the survey data than the source. By categorizing 
data as NMFS data and non-NMFS data, we were not implying that non-NMFS 
organizations could not conduct useful fishery-independent surveys. We 
categorized the data in this manner because NMFS currently conducts 
nearly all of the fishery-independent surveys and non-NMFS 
organizations collect most of the fishery-dependent data. Footnote 7 in 
the report states that NMFS generally refers to its data as fishery-
independent data and to non-NMFS data as fishery-dependent data. For 
these reasons, we did not change the report. 

37. We believe the Pacific hake biomass estimates are questionable 
because the assessment used non-NMFS data that NMFS did not check or 
subject to standard data reliability testing. Assessors who reviewed 
raw non-NMFS data for other stock assessments found mistakes that 
either made some of the data unusable or could have impaired the 
accuracy of the stock assessments. For this reason, we did not change 
the report. 

38. NOAA commented that bocaccio, canary, and darkblotched assessments 
all obtain adequate abundance trend information from the NMFS bottom 
trawl surveys. NOAA also commented that although bottom trawl survey 
cannot access the roughest habitat, it is useful as an index of 
relative changes in the overall abundance. As stated in our report, we 
found that the NMFS survey data used in these assessments were limited 
in scope because the surveys were conducted only in trawlable areas. 
Assessors estimated overall biomass using the NMFS data collected from 
the trawlable area, which has a different abundance rate than the 
untrawlable area. Stock assessors commented that relying on survey data 
from trawlable waters only increases the uncertainty of stock 
assessments. For these reasons, we did not change the report. 

39. As noted in our report, the National Research Council found that 
the inclusion of NMFS survey data was the best option for a reliable 
estimate of abundance because such surveys use an unbiased statistical 
design, control sampling locations, and provide for quality assurance. 
Northwest Center officials said that to obtain reliable results, each 
stock assessment should include at least one source of NMFS-collected 
data of sufficient scope and accuracy because such surveys are unbiased 
and scientifically designed. NMFS data were unavailable for the 
yelloweye assessment. Northwest Center officials also raised concerns 
about basing assessments solely on non-NMFS data such as commercial and 
recreational catch data. Catch data do not provide the species' 
relative or absolute biomass, according to NMFS officials. Catch data 
alone are insufficient because fishermen are not randomly sampling the 
ocean, but are fishing areas that they are allowed to fish and they 
believe have the most fish; fishing restrictions, such as a total 
allowable catch, can limit the amount of fish being caught; and catch 
data have often been inaccurate for a variety of reasons, such as 
imprecise accounting for dead fish tossed back into the ocean. For 
these reasons, we did not change the report. 

40. NOAA commented that the doubling of estimated bocaccio biomass in 
2003 was due to factors that would not be addressed in a standard 
statistical analysis. Although a standard statistical analysis may not 
fully address the doubling of an estimate, an assessment without an 
uncertainty range does not quantify and communicate any of the 
uncertainty. For this reason, we did not change the report. 

[End of section]

Appendix III GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Keith W. Oleson, (415) 904-2218: 

Staff Acknowledgments: 

In addition to the person named above, Leo G. Acosta, Kristine N. 
Braaten, Allen T. Chan, David Dornisch, Alan Kasdan, Robert Marek, 
Cynthia C. Norris, Carol Herrnstadt Shulman, and Tama R. Weinberg made 
key contributions to this report. 

(360345): 

FOOTNOTES

[1] Marine Fisheries Stock Assessment Improvement Plan, U.S. Department 
of Commerce, NOAA Tech. Memo. NMFS-F/SPO-56, 69, 2001. 

[2] Pub. L. No. 94-265, 90 Stat. 331 (codified as amended at 16 U.S. C. 
§§ 1801-1883). 

[3] In addition, the Northwest Center and the Southwest Center share 
some responsibilities in freshwater rivers and streams in Idaho, 
Oregon, Montana, and Washington. 

[4] Ibid. 

[5] Pub. L. No. 104-297, 110 Stat. 3559 (1996). 

[6] Under the Magnuson-Stevens Act, "bycatch" means fish that are 
harvested in a fishery, but which are not sold or kept for personal 
use. Bycatch includes fish discarded for regulatory or economic 
reasons. 

[7] The 2003 darkblotched update was not a full stock assessment, but 
rather updated the 2000 darkblotched stock assessment by adding some 
additional data in order to estimate new biomass figures. The most 
recent estimate from this update is for 2001. 

[8] The most recent estimate of biomass for yelloweye from this stock 
assessment is for 2001. 

[9] NMFS generally refers to its data as fishery-independent data and 
to non-NMFS data as fishery-dependent data. 

[10] Science and Its Role in the National Marine Fisheries Service, 
National Research Council, National Academy Press, July 2002. 

[11] Zimmermann, Calculations of Untrawlable Areas within the 
Boundaries of a Bottom Trawl Survey, NRC Research Press, July 2003. 

[12] U.S. General Accounting Office, Standards for Internal Control in 
the Federal Government, GAO/AIMD-00-21.3.1, (Washington, D.C.: 
November 1999). 

[13] Improving Fish Stock Assessments, National Research Council, 
National Academy Press, 1998. 

[14] Canary Rockfish STAR Panel Meeting Report, Northwest Fisheries 
Science Center, May 7, 2002. 

[15] U.S. General Accounting Office, Quantitative Data Analysis: An 
Introduction, GAO/PEMD-10.1.11, (Washington, D.C.: May 1992). 

[16] The Southwest Center also took some independent and joint actions 
with the Northwest Center to improve data quantity. 

[17] Pacific hake has a separate observer program. Discards are the 
amount of fish unintentionally caught and not retained on the fishing 
vessel. Discard information is collected for use in assessing the 
mortality for a number of groundfish species. 

[18] The sixth science center, the Pacific Islands Fisheries Science 
Center, opened in 2003. 

[19] In light of the delayed congressional appropriation of fiscal year 
2004 funds, NMFS has not yet determined the precise amount of money 
available to its science centers. As a result, as of April 2004, NMFS 
had not yet asked the Northwest Center to update its preliminary 
requests for fiscal year 2004 funds. 

GAO's Mission: 

The General Accounting Office, the investigative arm of Congress, 
exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www. gao. gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www. gao. gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. General Accounting Office

441 G Street NW,

Room LM Washington,

D. C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www. gao. gov/fraudnet/fraudnet. htm E-mail: fraudnet@gao. gov

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, NelliganJ@gao. gov (202) 512-4800 U.S. 

General Accounting Office, 441 G Street NW, Room 7149 Washington, D. C. 

20548: