|
January 15, 2009
See Report Documents
In early August, 2008, FDA sent two draft documents, now entitled “Report of Quantitative Risk and Benefit Assessment of Consumption of Commercial Fish, Focusing on Fetal Neurodevelopment (Measured by Verbal Development in Children) and on Coronary Heart Disease and Stroke,” and “Summary of Published Research on the Beneficial Effects of Fish Consumption and Omega-3 Fatty Acids for Certain Neurodevelopmental and Cardiovascular Endpoints,” to seven individuals who are expert in a range of scientific disciplines, identified below. The peer reviewers were asked to provide individual, written comments in response to a specific set of questions and to provide other comments on the draft. At the time of review, the two documents identified above were contained in a single document.
We greatly appreciate the peer reviewers’ comments and suggestions, as well as their willingness to provide them relatively quickly despite the length and complexity of the document. The charge questions are set forth below. The reviewers’ full responses to the specific charge questions, and other comments, are provided below without attribution to the specific reviewer.
In the last section of this document we have provided a table that identifies issues raised by the peer reviewers and sets forth whether FDA agrees or disagrees with the comments and describes the actions the agency has taken in response. While we have carefully considered and made several changes in response to the peer reviewers’ comments, this peer review report is being posted in draft form. Further edits to the document in response to these peer reviewers’ comments and to other review comments are likely.
Elaine M. Faustman, Ph.D.
School of Public Health and Community Medicine
University of Washington
Seattle, WA 89105
Dr. Faustman graduated cum laude with a dual major in Chemistry and Zoology from Hope College, Holland, MI and received a Doctorate in Pharmacology and Toxicology from Michigan State University. In addition to serving as a Professor in the Department of Environmental and Occupational Health Sciences at the University of Washington, Dr. Faustman is also the Director of the Institute for Risk Analysis and Risk Communication in the School of Public Health and Community Medicine, Director of the Center for Child Environmental Health Risks Research, Director of the Pacific Northwest Center for Human Health and Ocean Studies, and Director of the Reproductive and Developmental Toxicology Core, NIEHS Center for Ecogenetics and Environmental Health, and chairs the University of Washington Chemical Hazards Advisory Committee. She also serves as an Adjunct Professor in the Evans School of Public Affairs at the University of Washington and as an Adjust Professor in the Department of Engineering and Public Policy at Carnegie-Mellon University in Pittsburgh, PA. Dr. Faustman is a Fellow of the American Association for the Advancement of Science, a Diplomate of the American Board of Toxicology and serves as a panel member on the National Advisory Panel for the National Oceanic and Atmospheric Administration's (NOAA) Oceans and Human Health Initiative. Dr. Faustman's primary areas of study are the mechanistic investigation of reproductive and developmental toxicants; molecular mechanisms of action of metals and pesticides; quantitative risk assessment; development of biologically based dose response models for non-cancer risk assessment; in vitro toxicology; molecular epidemiology; toxicogenomics; and public policy. She has served as a presenter and/or chair at several international seminars and symposia, including the International Society of Exposure Analysis (ISEA), International Society for Environmental Epidemiology, the 5th Congress of Toxicology in Developing Countries, and the South Africa Toxicology Society. Dr. Faustman serves as a reviewer for a variety of scientific journals, including: Applied Occupational and Environmental Hygiene Journal; Teratogenesis, Carcinogensis and Mutagenesis; the American Journal of Epidemiology; and Aquatic Toxicology. She has published extensively on toxicity and teratogenicity of metals.
Herman J. Gibb, Ph.D.
Sciences International
Alexandria, VA 22314
Dr. Gibb received a Masters in Public Health (Environmental Health) from the University of Pittsburg and a Doctorate in Epidemiology from Johns Hopkins University. He is President of Sciences International. Dr. Gibb is an Adjunct Professor at the George Washington University School of Public Health and belongs to the International Society of Environmental Epidemiology. Dr. Gibb chairs the World Health Organization's Foodborne Epidemiology Reference Group (FERG) task force on foodborne chemicals and is a member of the FERG source attribution task force to evaluate methods to determine the disease risks from food versus those from water. Before joining Sciences, Dr. Gibb held positions as the Associate Director for Health and Assistant Center Director at the National Center for Environmental Assessment of the U.S. Environmental Protection Agency (EPA). As the Associate Director for Health, Dr. Gibb was responsible for the Integrated Risk Information System, EPA's on-line system of health risk assessments. He was the Project Officer for EPA's cooperative agreements with the World Health Organization. He directed EPA's assessment of inhalation exposures and potential health risks to the general population that resulted from the collapse of the World Trade Center towers. He was the recipient of the EPA's Scientific and Technological Achievement Award for his study of lung cancer mortality and clinical irritation among chromate production workers and the recipient of the EPA's Gold Medal for Exceptional Service for his work on the drinking water standard for arsenic. Dr. Gibb was a member of the White House Interagency Committees on Mercury and Risk Assessment. He was the lead author of EPA's Mercury Research Strategy.
Dariush Mozaffarian, MD, Dr.PH, FACC, FAHA
Harvard School of Public Health
Boston, MA 02115
Dr. Mozaffarian received a medical degree from Columbia University, a Masters in Public Health (Epidemiology) from the University of Washington, and a Doctorate in Public Health (Epidemiology) from Harvard University. He is an Assistant Professor, Department of Medicine, Harvard Medical School and Assistant Professor, Department of Epidemiology, Harvard School of Public Health. He is founder and co-director of the Program in Cardiovascular Epidemiology at the Harvard School of Public Health. Dr. Mozaffarian's teaching has included Cardiovascular Epidemiology at the Harvard School of Public Health and Tufts School of Medicine. Dr. Mozaffarian's primary area of study has been the effects of lifestyle factors on multiple endpoints, including coronary disease, sudden death, stroke, heart failure, and atrial fibrillation. Examples of specific research projects include investigation of relationships of different fish meals with arrhythmic and non-arrhythmic coronary events; relationships of dietary fiber from fruit, vegetable, and cereal sources with stroke and coronary event; relationship of trans fatty acid intake with systemic inflammation; relationship of fish intake with heart failure and atrial fibrillation; effects of fish oil on heart rate in randomized trials, and mercury, selenium, and risk of cardiovascular disease in women and men. The aim of the latter research is to investigate prospectively the relationships of mercury and selenium levels and fish and omega-3 fatty acid intake with incidence of coronary heart disease and stroke. Dr. Mozaffarian has participated on a FAO/WHO Expert Consultation on Fats and Fatty Acids in Human Nutrition and on a United States Department of Agriculture Seafood Education Project Advisory Group. He has published extensively on the effect of fish and omega-3 fatty acid consumption on risk of coronary heart disease and stroke.
Gregory M. Paoli, M.A.Sc.
Risk Sciences International, Inc.
Ottawa, ON, Canada, K1N 6Z4
Mr. Paoli has a Master of Applied Science in Systems Design Engineering from the University of Waterloo. He is President of Decisionanalysis Risk Consultants, Inc., specializing in risk assessment and risk management in the field of public heath and safety. Within Canada, Mr. Paoli has served on Expert Committees of the National Roundtable on the Environment and the Economy and is a member of Health Canada's Expert Advisory Committee on Antimicrobial Resistance Risk Assessment. He has provided guest lectures at the Queen's University's Public Sector Executive Programme and School of Public Policy, the University of Calgary's Faculty of Management and the University of Ottawa's Institute of Population Health. In the United States, Mr. Paoli has served on an Institute of Medicine Committee tasked to Review the United States Department of Agriculture's E. coli 0157:H7 Farm-to-Table Process Risk Assessment. He was appointed to a NRC Committee entitled "Improving Risk Analysis Approaches Used by the Environmental Protection Agency." Mr. Paoli served for several years on an Expert Panel to develop a risk ranking framework for the FDA and was on the peer review panel for the Harvard BSE risk assessment. He has served on several international expert panels including Expert Consultations as part of the Joint Food and Agriculture Organization and World Health Organization (FAO/WHO) Activities on Microbial Risk Assessment. Mr. Paoli has provided training in risk assessment approaches in North America, Japan, and South America. He also provides lectures as part of the Harvard School of Public Health continuing education course in Probabilistic Risk Assessment.
Barbara Petersen, Ph.D., M.P.H.
Exponent
Washington, DC 20036
Dr. Peterson received a Masters in Public Health in Nutrition from the University of California at Los Angeles and a Doctorate in Biochemistry from George Washington University. She is employed by Exponent, an engineering and scientific consulting firm, where she serves as Principle Scientist in Exponent's Health Sciences Center for Chemical Regulation and Food Safety. Dr. Peterson's areas of expertise include exposure assessment methodology, functional food safety and efficacy evaluations, food consumption profile modeling, and applications of Mote Carlo techniques to risk assessments for chemicals, including contaminants, pesticides and nutrients. Dr. Peterson has directed the design and conduct of numerous statistically based market basket studies, including acute and chronic assessments for pesticides, compliance assessments under proposition 65, and market research. Dr. Peterson served on the EPA Science Advisory Board's Integrated Exposure Committee and as an Expert Advisor to the FAO/WHO for several sessions of its Joint Expert Committee on Food Additives and Contaminants and for numerous consultations on risk assessment. She also served as Principal Investigator for the National Cancer Institute's International FOODBASE project, a major effort to collect and computerize descriptive and summary information on food consumption surveys conducted in more than 40 countries. Dr. Peterson has provided statistical support to FDA's Center for Food Safety and Applied Nutrition, including developing criteria for evaluating nutrition databases, and to EPA's Office of Research and Development. She has been a faculty member in risk assessment training programs for government scientists in the European Union, Thailand, the United States, and China. She has published extensively on methods for estimating dietary exposure.
Kimberly M. Thompson, Sc.D.
Harvard School of Public Health
Boston, MA 02115
Dr. Thompson has a Master of Science degree in Chemical Engineering Practice from the Massachusetts Institute of Technology and a Doctor of Science degree in Environmental Health from Harvard University. She is an Associate Professor of Risk Analysis and Decision Science at the Harvard School of Public Health Department of Health Policy and Management and the Department of Society, Human Development, and Health. She is the creator and Director of the Kids Risk Project. Dr. Thompson is also Associate Professor of Risk Analysis and Decision Science (Pediatrics), Children's Hospital, Harvard Medical School, where she is co-founder of the Center on Media and Child Health. Prior academic appointments include Visiting Associate Professor, MIT Sloan School of Management. Dr. Thompson's research interests and teaching focus on issues related to developing and applying quantitative methods for risk assessment and risk management, and consideration of the public policy implications associated with including uncertainty and variability in risk characterization. She is particularly interested in issues related to variability in risk for sensitive sub-populations, particularly children, and the potential risk tradeoffs associated with policies designed to protect them. The work includes research on a range of children's risks including injury, environmental, medical, and product-related risks, as well as perception of children's risks and the portrayal of risky behaviors in popular entertainment media. Her publications includes the book entitled "Risk in Perspective: Insight and Humor in the Age of Risk Management," a guide to help consumers take charge of health information.
Renee C. Wachtel, MD
Department of Developmental and Behavioral Pediatrics
Children's Hospital and Research Center at Oakland
Oakland, CA 94609
Dr. Wachtel received a medical degree from the State University of New York, Downstate Medical Center. She is Director of the Division of Developmental and Behavioral Pediatrics, Children's Hospital and Research Center at Oakland, Oakland, California. Previously, she was Professor of Pediatrics, University of Maryland School of Medicine, and the Director of the Division of Behavioral and Developmental Pediatrics, University of Maryland School of Medicine. She was also an Associate Professor of Pediatrics at The Johns Hopkins School of Medicine. Dr. Peterson serves as Co-Chairperson, Committee on Developmental and Behavioral Pediatrics, Northern California Chapter, American Academy of Pediatrics; Co-Chairperson, Committee on School Health, Northern California Chapter, American Academy of Pediatrics; and Chair, Autism Task Force, Children's First Medical Group. She has conducted research and published extensively on a range of neurodevelopmental issues.
Each expert peer reviewer was provided with a written “charge” concerning the document, as follows:
When the FDA generates a scientific assessment, it is presenting its scientific evaluation about the accumulated evidence. The peer review should provide input on the reasonableness of judgments made from the scientific evidence. The result should be an independent determination by each peer reviewer as to the appropriateness of (a) the assumptions made and hypotheses postulated, (b) the methodology utilized, (c) the quality and relevance of the data and information, (d) the accuracy of the analytic results, and (e) whether the conclusions reached are supported.
This well written and well researched report seeks to provide updated scientific evidence regarding the health risks and benefits of eating commercial fish products. The purpose is to assist the FDA in developing consumer advisory information about the health consequences of exposure to methylmercury (MeHg) from fish products in the diet of American consumers. Three "health endpoints" are considered: effects upon neurodevelopment, fatal coronary heart disease and fatal stroke. In addition to reviewing the available scientific literature, quantitative risk modeling is included, to provide a quantitative assessment of potential health effects of different potential health advisories. Using statistical methods, the document seeks to separate the potential health consequences of the unintentional ingestion of MeHg found in some fish products from the potential health consequences of ingesting commercial fish products, some of which contain negligible amounts of MeHg.
The results of the report are well supported and indicate that for the general public, there are substantial advantages to fish consumption, and small, if any, risk, if low MeHg fish are consumed. Neither the thumbnail summary nor the executive summary has a section at the end that provides these conclusions.
This topic is complicated by a number of factors:
All of these potential confounding issues are discussed in the report, including potential explanations for why individual studies have different results regarding health effects. The extensive bibliography is helpful in providing documentation about the scientific basis of the report's discussions and findings. A few suggested additions are noted below. Adding the two references about the documented beneficial effects of fish oil would add to the premise of the report that there are clinical studies documenting the beneficial effects of fish components upon neurodevelopment.
As a medical professional with specialized expertise in neurodevelopment, and as an advocate in the developmental disability community, I have several additional concerns about what is missing from the document, rather than what is in the document:
The report is very well written, well organized, and well supported. While the neurodevelopmental effects research was well characterized from a MeHg exposure point of view, there was limited discussion of the limitations/strengths of the different developmental outcomes measures used and their generalization ability. Some of the studies were more credible than others, based upon outcome measures used and ages of testing. But in the discussion and analysis, all neurodevelopmental outcome measures were considered equally credible. For example, in the Hibbeln 2007a study, early developmental outcome was measured by the DDST, a screening test administered by the parent, with a subset actually tested using a standardized measure (the Griffith) at 18 months of age. The accuracy of developmental measures is well known to be much better within the normal range at later ages. Much more compelling data in this paper is the standardized IQ data from age 8 years, administered to more than 5000 children, with generally better p values as well. An advantage of this study is the effort to control for a number of environmental variables. Unfortunately, this study did not differentiate species of fish consumption, portions or assessment of MeHg levels. Similar results were found by the US study by Oken et al., which was able to correlate MeHg levels, maternal pregnancy fish consumption, and standardized child outcome measures (but at age 3) in a prospective cohort of 341 maternal-child pairs.
A table comparing the different studies would be very helpful to allow the reader to compare the studies and their results more easily.
In general, the modeling presented data variables and assumptions in a comprehensive fashion, although at times hard to follow for a non-statistician. A clearer summary would be helpful.
Some modeling assumptions seem questionable, such as subtracting 3 months from the Iraq data since the ages of walking and talking were recorded in 6 month increments, which seems arbitrary and probably not supportable.
However, the "what if" modeling of neurodevelopmental effects was not sufficiently explained. For example, what is the definition of "age of talking?" Was it first word including names (mama or dada), first word not including names (not including mama and dada), age of having a specific number of words in his/her vocabulary, age of speaking in phrases or sentences? These normal milestones range from 10 months to three years. Standardized developmental tests are much more predictive, especially if "age of talking" is obtained retrospectively. The modeling based upon amounts and types of fish consumption were much more clearly delineated.
Some additional references are:
Jacobson JL and Jacobson SW. Risks to Child Health from Methylmercury Exposure in Immigrant Populations. Journal of Pediatrics 2006; 148:716-718.
Innis et al. Increased levels of mercury associated the high fish intakes among children from Vancouver Canada. Journal of Pediatrics 2006; 148:759-763
Clifton JC. Mercury Exposure and Public Health. Pediatric Clinics of North America 2007; 54: 237-269.
Sato et al. Antepartum seafood consumption and mercury levels in newborn cord blood. Am. J. Ob Gyn. 2006; 194:1683-8.
Holland et al. Maternal supplementation with very long chain n-3 fatty acids during pregnancy and lactation augments children's IQ at 4 years of age. Pediatrics 2003:111, e39-44.
Dunstan et al. Cognitive assessment of children at age 2.5 year after maternal fish oil supplementation Arch. Dis. Child. Fetal. Neonatal Ed. 2008; 93:F45-50.
The conclusions presented in the report are well supported by the various methods of analysis and literature review.
See above section.
See my comments in Section I. I am not sure that enough data exists to perform modeling about these issues, but the areas need to be addressed.
Yes, but the intervention strategies will need to be crafted better than current strategies to be effective. We do not want to have a complex message that will confuse consumers, or precipitate changes in dietary behaviors that are the opposite of the message.
In the thumbnail summary, it would be more understandable if the literature review was cited first, and that it included both prenatal and postnatal exposure upon neurodevelopment studies. The risk assessment then performed as part of this report would have a clearer context when discussed thereafter.
The executive summary is extremely well written and clear. The discussion of the different approaches to neurotoxicity and risk/safety assessments is helpful background information for the reader of the report. Dividing the literature review to different levels of Average US Exposure is a very useful method to help the reader keep the level of exposure in different studies in context.
The document takes on a difficult question – a quantitative risk-benefit analysis of fish consumption contaminated by methylmercury. The document does a good job of laying out the assumptions, methods, limitations, and conclusions in its evaluation of risks. In general, it provides sufficient detail for the reader to understand how the document's conclusions were reached. Additional detail is needed in some areas, however (see discussion below). The document considered several models and "what if" scenarios in its evaluation which allowed the authors to explore various approaches and ways to look at the issue and made the evaluation more enlightening. The document describes some of the limitations in knowledge (e.g., the constituents of fish that could contribute to neurological development, the relationship between omega-3 fatty acids in fish and methylmercury, whether a diet relatively high in selenium can neutralize methylmercury toxicity in humans) as important areas for further research. The document also points out the differences in the levels of methylmercury in the studies where effects have been observed with the levels consumed in the U.S. (e.g., Iraq and Minamata 100X; Seychelles, Faroe Islands, and New Zealand 10X, etc.). This is helpful to the reader in gaining a perspective of the studies where effects have been observed vis-á-vis the U.S. experience. While the overall impression for the reader is that consumption of fish is beneficial, methylmercury could present a problem for women of child-bearing age under some scenarios. The document is written for a U.S. audience, but the document will be used worldwide and that consideration should be made in developing the final message, particularly with respect to what the document refers to as the methylmercury-to-fish ratio. For example, for localized populations in the Amazon living near gold mining operations or populations in the Orient where shark or tuna may represent a significant food source, unrestricted consumption of fish could present a problem. The same may be true for some populations in the U.S. Overall, however, unrestricted fish consumption would appear to be a benefit in most populations.
In general the document was logical, and the presentation was clear.
The executive summary, however, is too detailed and needs to make the summary points more concisely. Particularly confusing in the executive summary is the discussion on Quantitative Risk Assessment for Fetal Neurodevelopment. Perhaps the Thumbnail Summary or a slightly expanded Thumbnail Summary could be made the Executive Summary?
My understanding of Table IV-11 is that the first percentile of the population could experience negative effects, presumably because they did not eat enough fish. It took me several readings to understand that, however. I think the confusion is caused by the title of the first column, "Population Percentile." (Population percentile of what?) The text which discusses the table could also be made more explicit.
The first time that the word, "baseline" is used in the document is on page 19. Discussion on baseline continues throughout the text, but it is not until the bottom of page 132 that there is a good definition of what is meant by "baseline."
One assumption that seems questionable is the assumption that the combination of the Iraq (Marsh et al. 1987) and Seychelles (Myers et al. 1995) data "was sufficiently close to a methylmercury effect minus fish to give us a reasonable approximation of it." (page 107, middle of page). There were 680 mother-infant pairs from the Seychelles (where fish were consumed) and 81 mother-infant pairs in Iraq. It is difficult to understand how fish could not have had an effect in the model at the lower dose (Seychelles) given that there were so many more mother-infant pairs in the Seychelles study than in the Iraq study yet the document states in several places that the models were dominated by the Iraq data (e.g., page 107, second bullet; page 114, first para). It is also unclear how the data can be combined to estimate age of first talking or age of first walking when the ages of the Iraqi children are unknown.
A second assumption that is questionable is the combining of the UK data (Daniels et al.) study with the Iraq and Seychelles data in the combination net effect model given that the Daniels et al. study did not examine age of talking. The authors argue (page 108) that the Daniels et al. study did include tests of verbal comprehension at young ages and therefore the results are comparable to the Iraq and Seychelles data, but verbal comprehension occurs at an earlier age than talking. Since the Daniels et al. study did not observe an adverse effect of methylmercury, the inclusion of the study with the Iraq and Seychelles data could therefore bias the model (underestimate the effect of the methylmercury) since age of comprehension is younger than the age of talking.
Finally, it is not intuitively obvious how one combines data from a study where there was no fish consumption (Iraq) with two studies that examine methylmercury exposure via fish consumption to arrive at a net evaluation of fish and methylmercury consumption.
The appropriate literature has been cited.
The conclusions appear reasonable given the results of the quantitative risk assessment; however, some of the assumptions used to develop the assessment may be questionable.
There was sufficient information on how the data were identified for the quantitative risk assessment, and the criteria used to determine the suitability of the data for the quantitative assessment were adequately described (pp 110-111, 141, 152-153). A primary criterion for choosing the data for the assessment was the access to individual, as opposed to group data. This resulted in the selection of three studies – the Iraq study, the Seychelles study, and the UK study (Daniels et al.). The selection of these three studies may open the analysis to criticism in that it identified two large studies (Daniels et al. and Myers et al.), both of which found no effect of methylmercury, to combine with a much smaller study (Iraq) where effects from methylmercury were observed. Individual data provide a much better basis for model development, however, and thus provide a defensible position for the selection of these three studies. Interpretation of the data from the three studies in an overall statement on risk-benefit may be complicated by the assumptions which are discussed above, however.
It is unclear what is meant by "two-dimensional population models" (page 132). There is some discussion of it in an appendix, but more description of what is meant by the term should be provided in the body of the text.
The document states that in the "absence of methylmercury," the "model" was able to predict the age of first walking and the age of first talking. The "model" predicted a "central estimate" for first walking of 10.4 months (page 117) and a "central estimate" for first talking of 15.1 months (page 114). These "central estimates" are early in comparison with the literature on child development. Most textbooks suggest a central estimate for walking of 12 months. It depends what is meant by "talking," but the first word other than mama/dada is usually spoken by 9-14 months, but 4-6 words are not said until 11-20 months and two word combinations until 14-21 months. An explanation needs to be provided for the discrepancy between the model estimates and the data in the literature on ages of walking and talking.
See the discussion of assumptions in my response to charge question 2 above.
I am not aware of any.
Yes, given the current FDA/EPA guidance to limit their consumption of fish to 12 ounces per week during pregnancy.
p. 54, paragraph beginning "Two years later," - Change "Gandjean" to "Grandjean."
p. 120, paragraph beginning "Cohen et al…" - Change "Harvand" to "Harvard."
p. 126, first paragraph - Change 63 grams to 28.6 grams.
p. 132, paragraph beginning "The age-of-talking …"- Change "Given the evidence in support a beneficial…" to "Given the evidence in support of a beneficial…"
p. 142, paragraph beginning "Of the thirteen…"- Delete "of" in the first line.
From an analytical perspective, the overall approach to risk assessment for methylmercury represents a relatively thorough treatment of information from the literature (e.g., gaining access to raw data for intensive meta-analysis with respect to dose-response relationships). While the scientific-level reasoning is thoroughly described, the nature of the computational approach taken and the details of the implementation are not adequately described. As a result, it is not possible to determine the validity of the computational implementation.
The overall computational structure of the model is poorly described, with a vague description spanning only a few pages in Appendix A. A crude representation of the flow of information is provided in two high-level diagrams (Figures AA-18 and AA-19). While these figures are appropriate and simple enough to be understood within a general summary, they are not sufficient to describe the computational approach. Relative to the details of the computation they are superficial and do not provide nearly the amount of information that would be required to fully understand what is being computed, and how all of the many computational model elements work together. The relative amount of documentation given to interpretation of the scientific literature as compared to the computational implementation of the model is quite imbalanced given that the computational implementation is just as much part of the results as the many scientific assumptions and the raw data.
This level of report documentation may have been more tolerable if the Excel implementation had been adequately documented separately, within the Excel worksheets or within the VBA code itself. As a simple example, the document (Appendix A) indicates that the model is implemented in Microsoft Excel. While true, the full implementation spans multiple worksheets in multiple workbooks in multiple folders, with a Word document entitled "What's on this disc?" providing less than a quarter of one page as the documentation of the computational structure of the system and the relationships between the various files. As a result, when reviewing the document, it is not clear in which of the many files (and within each file, which macro or worksheet) the corresponding technical implementation is performed. The inter-relationship between these many files is not possible to construe (except as can be inferred in a time-consuming and quite uncertain process from scrutiny of the raw VBA code). The VBA implementation appears to use a custom VBA library called MC2D but it appears to have no separate source of documentation (it is not referenced in the report). The principal VBA macros are not identified, requiring scrutiny of the raw VBA code before determining which macro should be run first.
The macros themselves are generally not documented. Even when some documentation is provided, it tends to be minimally helpful or unhelpful. At one point, for example, there is the following line:
"GammaTrap = alpha * beta ' use mean value if gamma function craps out"
The purpose of this function must be inferred from the VBA help documentation on the built-in function gammainv, which indicates that this is an algorithm which will not always produce a numerical solution. This code proceeds to replace the intended value with the mean of the distribution when Excel produces an error. It is not clear if this function will be used as a frequent part of the simulation, or if it is a piece of code that captures an extremely rare exception in the application of the gammainv function that can be safely ignored. It is also not clear whether the replacement of error-handled estimates with the mean of the distribution will provide a bias toward reduced or elevated exposure estimates, or will simply narrow the apparent variability. It would be a simple matter to determine how often this 'Trap' was required in the development of the estimates.
In the same module, there are the following lines:
GSD = 0.37 + 1.4 * CurrentModel(SpeciesIndex, 2)
beta = 0.11 + 0.34 * (1 - CurrentModel(SpeciesIndex, 2))
These lines come with no documentation. Through scrutiny of the code and by inference, the reviewer might guess that these lines in the code represent the use of the uniform distribution referred to in the paragraph below (p. 174 in electronic copy provided). However, it is not clear where the constants come from. It is only by determining (from elsewhere in the VBA code) that CurrentModel(SpeciesIndex,2) is a random number between 0 and 1 that it is possible to make this inference. This type of guesswork must be repeated every few minutes by a reviewer of this model.
"Since raw data were unavailable for some species, distributions were generated with modeled distributions that reflected reported arithmetic mean values published from a National Marine Fisheries Service survey (NMFS 1978) for each group and a range analogous to those obtained from tuna, shark, and swordfish. Lognormal and Gamma distributions were used to represent the data, with each model assigned a probability of 0.5 to represent model uncertainty. The magnitude of the shape parameters (the geometric standard deviation of the lognormal distribution and the beta parameter of the gamma distribution) were represented as uniform distributions that encompassed the range of values resulting from fitting the shark, swordfish, and tuna data. The scale parameters (the geometric mean of the lognormal distribution and the alpha parameter of the gamma distribution) were calculated from the arithmetic mean in the NMFS survey and the shape parameter (Evans et al., 2000)."
This situation is unfortunate given the minimal effort that would be required to facilitate understanding: a simple line of documentation within the code stating that this was the implementation of a uniform distribution and that CurrentModel was a random number distributed as Uniform [0,1] would have been immediately apparent and would save considerable effort in scrutinizing the code. Similarly, a simple footnote in the report indicating that the paragraph above was implemented in a particular macro and/or a particular spreadsheet would avoid having to match the report description with the code implementation in a haphazard, time-consuming fashion.
As another example of the guesswork required, the document does not indicate how long it might take to run the model (and, for example, if it takes a long time as one might assume, whether it could be run in stages). There is also no indication as to what uncertainty and variability iterations would be minimally required to produce reliable results. The number of iterations that are applied in providing the report's results is never justified as providing a reasonably stable estimate of the mean, median or of any percentile of the resulting distributions. This may be particularly relevant given that there is a delicate balancing of risks and benefits which may be prone to being affected by numerical stability of statistical estimates. If this sensitivity is known not to be critical, it would be a simple matter to provide a summary of the stability of estimates as a function of simulation size (both uncertainty and variability iterations). Given the number of uncertainties applied in the model and the repeated use of probability trees, it is not possible to hazard a guess as to what level of simulation might be required to reach a given level of stability. In addition, there appear to be multiple places in the code where the actual 2-D simulation might be governed. There is an uncertainty loop within the MeHg intake model, as well as the apparent role of the MC2D model in conducting a 2-D simulation model for the risk estimates. It appears as though the MC2D model may provide some form of resampling from the uncertainty iterations in the MeHg intake model, but this again is largely a guess pieced together from a minimal description provided in Appendix A and the VBA code.
The repeated requirement for 'guesswork' both at the micro level (e.g., what individual lines of code are intended to achieve) as well as at the macro level (e.g., the general lack of documentation as to how the many spreadsheet files interact to form a complete simulation) forbids a formal characterization of the model as being sound. In the end, while no concrete 'fatal' errors were found in the computational implementation, it is simply not adequately documented to allow sufficient scrutiny to support or refute an opinion that the model and corresponding results are technically sound. Any opinion would simply be an opinion on what the reviewer has come to believe is being computed rather than being based in knowledge of precisely what is being computed. Determining the soundness of the model would require either considerably more documentation both within and outside of the model software, or extensive interaction with the author(s) of the code to determine exactly what is being computed, where, and in what order.
The inability to render an opinion given this level of documentation is unfortunate, particularly given this reviewer's general impression of the approach is that it is generally sound and might even be considered exceptional in the extent of the attempt to generate a faithful representation of the uncertainty that manifests from the raw data in both exposure assessment and dose-response assessment. This effort has been undermined, at least to this point in its development, by the inadequacy of the computational documentation. Had the computational implementation been better described it would have been possible to provide more substantive comment on the approach taken.
The reviewer recommends that the computational implementation be thoroughly described both within the VBA code and in a separate formal appendix and then subject to further review, particularly given a favorable overall impression of the underlying theory being applied in structuring the model. The overall structure of the simulation should be described as a separate matter from the treatment of any specific piece of evidence in the model. This level of documentation would allow for a peer review of the model and its results without the distraction of uncertainty as to how the results are ultimately computed.
The FDA assessment is comprehensive. The assessment addresses both the risks (due to methylmercury in fish) and some of the benefits (of fish consumption). In conducting this assessment FDA has explicitly acknowledged the body of literature demonstrating that substantial health benefits accrue from eating fish while at the same time acknowledging and equally substantial literature demonstrating the toxic effects of methylmercury. The assessment was designed to quantify the dose response for several effects including neurological developmental effects, coronary heart disease and fatal stroke in a way that allows a determination of a net benefit (or if appropriate a net benefit).
FDA has done a thorough and balanced job of assessing both the risks and benefits.
Key attributes of the assessment include the detailed in-depth analysis of all of the available literature using techniques that weigh the benefits versus the risks (and on the same scale). Because my personal expertise is in dietary exposure assessment, I have focused especially on the analyses of exposure – the methods used to estimate short and long term consumption and to conduct the assessment.
The study acknowledges the potential benefits of eating fish (due to factors other than methylmercury) and conducts an in-depth assessment of the "net benefits."…Essentially the benefits are measured with the same endpoints as "adverse" effects but without any attempt to identify what the factors are or to measure the presence or absence of the factors. For example, it may be that benefits are conveyed due to consumption of fish that contain omega 3 fatty acids and/or selenium and these nutrients offset some or all of the adverse effects of methylmercury. The FDA assessment does not attempt to identify these factors or to estimate their presence in the diets (other than a brief analysis of fish vs. marine mammals).
The FDA model recognizes that there many amounts of fish consumption that provide a net benefits while other amounts of fish (either very low or very high) that provide net risks. Therefore, the models incorporate very sophisticated dose response modeling. In doing so, it appears that all of the available data have been used – in one analysis or another.
The study notes that very high intakes of methylmercury have been demonstrated to cause net negative effects and as a result of contamination events in Japan and Iraq have convincingly demonstrated the adverse effects of methylmercury at high doses. Subsequent studies in which populations were exposed to much lower levels have looked for the same effects – with somewhat conflicting results. FDA has conducted in-depth evaluations using the individual data from many of these studies. The results are presented in a clear, understandable format and the order of those presentations is helpful. In particular, FDA has made it clear that there are very likely thresholds for different effects and simulated those thresholds by identifying the studies where consumers were eating similar amounts to Americans, and those where consumers were eating 10 times versus 100 times as much fish (and by extrapolation methylmercury) as Americans.
Given the available data, I cannot suggest any further analyses that would be of value to the users of this assessment. I have made a few suggestions below that would assist the reader in the understanding the implications of the findings.
The complexity of these analyses makes it imperative that there be a rigorous QA/QC of each set of data and analyses. I attempted to independently derive some of the estimates. I did not find any significant differences but could not begin to conduct a true QA/QC of the components of the assessments. It would be useful to include summaries of the QA/QC in an appendix to the report as a way of documenting the conclusions.
In summary, I cannot suggest any additional analyses that I think would make a meaningful difference in the report. I think additional discussion of some of the analyses would help most readers understand the significance. The uncertainty analyses and "what if" scenarios provide useful measures for understanding the impact of potential policy guidance. The difficulty in identifying a "threshold" type value where the adverse effects exceed benefits is clearly demonstrated in this assessment. In fact, I am persuaded by these analyses that such a threshold must be well above the levels consumed by even the high fish consuming populations that have been studied to date except where in the 2 studies where there was extremely high levels of methylmercury in the food (Iraq and Japan).
Yes. As I noted above, the FDA document is thoughtfully organized and provides a clear description of the methods, assumptions and even the ways in which each of the datasets were utilized. In my opinion, the evaluations that the FDA assessment considered are as comprehensive as the data would allow.
On p. 180 there is a discussion of diet-blood relationships and results that could be noted in more detail in the text. This is particularly important given the ongoing biomonitoring form NHANES and other sources.
It does not appear to me that there is any realistic scenario (e.g. based on what is known about the American diet) where American consumers would consume so much commercial fish that they would have a net risk from eating fish. In other assessments I have seen discussion of subsistence fisherman consuming large amounts of fish from a single source that is highly contaminated and it is possible to assume that a pregnant female consumes large amounts of high methylmercury fish on an on-going basis. These diets aren't reported in the national surveys – nonetheless FDA has shown prudence is identifying "cut off" points of the amount of such fish that might result in net negative effects. It appears to me that these cut-off points are more restrictive than necessary based on the risk assessment (in other words I didn't see a clear threshold around 12 ounces of fish).
Yes. I reviewed the literature on fish consumption and cardiovascular disease and am convinced that there is clear evidence of a positive benefit in consuming fish and that this is the consensus of a wide array of scientists. The components of the fish that provide these benefits have not been conclusively determined although omega 3 fatty acids and selenium have been suggested. The lack of knowledge concerning the factors and the almost certain variability in the levels in different types of fish makes the analyses attempted by FDA difficult. Nonetheless, the results are in agreement with those of other scientists and should be highlighted. In several places the possibility of a net adverse effect was noted in the report but it was not clear to be whether this was a remote possibility (e.g. the extreme bounds of a confidence interval based on conservative assumptions) or a plausible possibility. This was highlighted in the thumbnail on p. 8 where there is a statement "this combination net effect" model estimates results that are beneficial most of the time but that can occasionally be adverse…" However, I could not determine a realistic scenario where the net effect would be adverse …on even a short-term basis (and this appears to be confirmed in the "what if" analyses). If the "adverse" scenarios occur only rarely I think this sentence should be deleted or further qualified. At a minimum there should be additional discussion in the text.
I have assumed that QA/QC have been conducted to ensure that the modeling is as described as it was impossible in the time frame and with my level of expertise to repeat most of the analyses.
The Finnish study of fish consumers appears to suggest adverse effects due to fish consumption but given other potential differences (genetic differences, alcohol and smoking, etc.), it is possible that other conclusions could be drawn based on these findings. Differences in test subjects should be noted in the text. This study highlights many of the difficulties in looking at differences in different populations. There are probably population differences that are larger than any likely adverse impact of methylmercury in the fish of these populations and it would be helpful to make some note of this in the report.
I believe all of the appropriate literature has been cited (and more importantly considered in designing the risk assessment).
The analyses incorporate all of the important variables. The methods used to integrate the consumption data and the methylmercury concentrations in food are appropriate and are widely used. The approach to correcting for long term consumption is the best available methodology (although new (2003-2006) NHANES data will allow this approach to be validated since short term (2 day) and longer term (30 day) are now available for the same individuals. (Based on preliminary work with the new day, I believe the findings will be essentially the same as though obtained through the methodology in this report).
More specifically, I considered the variables that should be included are listed below along with my assessment of FDA's handling of these variables:
For the exposure modeling exercises I considered the following questions:
Yes. I have a few suggestions that I think would improve the report (in terms of providing those of us with less familiarity of the risk assessment models more background information so that we can better understand the significance of the results).
187 – top of page, "the estimate is employed in our analysis as a normal distribution..." It would be helpful to know why this was assumed to be a normal distribution and whether a different assumption would make any difference (if so how much). I believe this could be done without actually doing additional modeling based on the statisticians understanding of the impact of this assumption versus a different assumption.
Comments on the assessment of the role of methylmercury/fish consumption on neuro-development:
Yes. The results seems to suggest that the threshold for the delays in the age of talking is above any of the exposures that were seen in the long terms epidemiological studies (e.g. Saychelles, Faroe Islands, NZ) except for the poisoning episodes in Iraq/Japan. That is, the delay in the age of talking: consumption over 12 oz doesn't seem to delay talking (in a clinically significant way unless the fish are highly contaminated)…I don't believe that it is possible to model a situation where you could eat enough fish with typical levels of methylmercury to cause a delay beyond 1-2 days (the investigators in the Iraq study used 6 month intervals).. I reached similar conclusions regarding IQ decrements. It would be helpful to the reader is the actual changes (in numerical terms) were inserted in the bullets on p. 165. On p. 193 the table (Net IQ decreases) has both IQ and delayed walking – are the units the same? In my opinion the results of the UK study (Daniels et al 2004) should be further discussed in the text. The levels of exposure to methylmercury are likely to relevant to US populations and the findings show net positive benefits.
Comments on the assessment of the role of methylmercury/fish consumption on CHD death rates:
Tables IV-16-20 needs additional explanation of the assumptions and findings (and even the units).
The findings of these analyses highlight the importance of more people consuming fish on the incidence of CHD death. I don't believe the results apply to fish consuming individuals who simply consume more/less fish. Am I correct? This section would benefit for an expanded discussion of the assumptions and findings.
Comments on the assessment of the role of methylmercury/fish consumption in fatal stroke:
On p.158 the difference in the size of the confidence intervals between the Carrington and Bouzan based model is highlighted. I believe these differences are likely due to the differences in data that were included in the two models. If that is correct, it would be helpful to mention that.
At the top of p 159 the documents notes "...47 additional stroke deaths may be caused by fish consumption." I think it should be clear that the model attributes these deaths to fish consumption…and the net benefit should be expressly calculated an included.
Yes. It is important to note that the risks are from a different component than the benefits (My understanding as result of my review of this report and of the published literature is that there is no benefit of methylmercury itself so the benefits are due to other components of the fish (ALA and other long chain fatty acids most likely).
For most audiences of the report, the assessments would benefit from some additional general background information about the range of variability in the types of endpoints that are measured. For example, changes in onset of talking and walking are highly variable endpoints that are likely to be affected by many factors. Although the FDA analysis provides indications of the variability, there is little discussion about the precision of these types of measurements (inter and intra individual variability in measurements) and effects of IQ, socioeconomic status, etc. on the measurements. Background information will be particularly helpful since the studies were done in different populations (with different baselines) and different researchers used different surrogate measures for the endpoint. Also the large battery of tests that were run in some studies make it even more likely that one or more of the endpoints would show statistical significance just by chance (again, comparison to clinically important value would assist the reader). The end points involving behavioral and clinical assessments may not be able to be detected or to be clinically meaningful. For example, can changes in IQ of less than one point be meaningfully measured or would not be judged to be different? If not, then predictions of less than 1 point should be defined as a "no effect?" Similarly is a delay of < 1 day in talking/walking of any clinical significance?
There are probably other endpoints that could be assessed but based on the epidemiological studies they would be very minor and of no public health consequence. FDA has identified those with the largest effects.
The "what if" scenarios raise one of the most important regulatory questions and the discussion of the assumptions that went into these analyses should be expanded along with additional discussion of the impact of assumptions on the outcome and the likely significance to those making policy decisions, e.g. what guidance is appropriate?
Based on actual poisoning episodes (Iraq and Japanese populations), very high levels of methylmercury intake must be avoided. The impact of the intake of lower levels of methylmercury does not appear to cause any significant adverse effects in any of the populations studied (despite decades long studies). Thus the challenge is to determine the "threshold" where the net benefits of fish consumption cross to net adverse effects due to methylmercury (and most likely other substances).
Additional comments on some of the scenarios:
p. 19 "what if scenarios" that exclude benefits – should be clearly identified and not used for policy decisions. Fish is not consumed without benefits.
p. 20 "…other sources of nutrients found in fish" – based on a review of foods – this is not a likely scenario in my opinion
p. 21 – "…could produce a small neurodevelopmental decrement (up to 4 %) in some people." My review of the data does not suggest that this arise in any realistically meaningful situation. Additional discussion should further highlight the assumptions and their impact on the outcome.
The discussion of the results of the "what if" guidelines should specifically note that methylmercury exposures are a function of the amount of the food consumed and the level of methylmercury in the fish. Therefore, the "what if" scenarios must address both variables as should any resulting "guidance," e.g. should guidance focus on amounts of food consumed, types of food (e.g. high methylmercury containing fish) or both?
These results indicate that the difference in risk of adverse effects above and below 12 oz per week is not large (and would not be clinically meaningful unless the concentrations in the fish are as high as those seen in frank poisoning episodes or are at the highest levels seen in the TDC and those species are consumed all of the time by high fish consumers). While it is possible to imagine such a scenario, none of the available data identify individuals in the US population who are consuming such species on a long-term basis and that should be noted in the discussion. On p. 164 in the first paragraph under (g) Summary and Interpretation, the statement is made "the risk assessment predicts, that on a population basis, capping maternal fish consumption at 12 ounces would reduce the national average level of neurodevelopment slightly even when pregnant women eat only 'low methylmercury' fish." This is an important finding and should be further explained since current guidance includes the 12-ounce figure. Would this same effect be seen if it were capped at 16 ounces/week? Or is it the type of fish that should be capped? Or is it the need for more variability in the type of fish?
On page 65:the report states "the risk assessment indicates that encouraging consumption up to, but not beyond 12 ounces per week essentially accepts some risks in order to obtain nutritional benefits from fish consumption, while also avoiding slightly higher risks and benefits associated with higher consumption..." In my opinion, this is a premise which, when tested in the risk assessment, wasn't confirmed. The risk assessment didn't show that limiting consumption to12 ounces produced any clinically meaningful difference in outcomes. There was a net benefit at much higher levels of consumption – even considering the high methylmercury fish.
Editorial note: The figures in the various chapters are incorrectly referenced in the text in several places including in Appendix A…Figure AA-4 on are incorrectly referenced in the text.
First, the U.S. FDA should be lauded, in no uncertain terms, for undertaking this crucial and critically important analysis. Public health recommendations must have – must – as their principal intent the improvement of health. The traditional principles of toxicology (risk)-based assessments were developed to consider toxin-containing exposures that were otherwise generally health-neutral, e.g., paint, plastics, water (and most foods), etc. Such principles can lead to bizarrely illogical results when applied to exposures that otherwise have important health benefits. This problem is particularly patent for fish consumption, for which the health outcomes that may be worsened by toxins in fish are the same as those that may be benefited by nutrients, especially omega-3 fatty acids (but also selenium and vitamin D), in fish. Thus, risk-assessment of both risks and benefits is essential to enable sound conclusions, and consequently appropriate recommendations, for health effects of fish consumption.
Many great strengths are evident. The report focuses on the endpoints of greatest relevance: neurodevelopment and cardiovascular disease. Studies of both risk and benefit are considered. Both qualitative and quantitative analyses are included. Generally, the conclusions are sound and reasonable. However, careful attention to revisions to address a few crucial limitations would greatly improve the accuracy of the information provided and, especially, the clarity of presentation and soundness of conclusions. In summary, these include:
Performing this analysis as an estimate of the effect of MeHg "if ingested in food other than fish" is clearer, although not very meaningful – where such levels of MeHg would come from, absent another grain disaster, is unknown. Such theoretical exposure is not meaningful for the U.S. public. Further, this is not a Report to investigate the potential effects of MeHg in isolation (just as it does not quantify the potential effects of EPA+DHA alone, selenium alone, or vitamin D alone). This modeling should be deleted altogether, or at the very least removed from the Summaries and placed in an appendix, as it is not based in any real scientific scenario and presents highly misleading and confusing "findings."
The Report has many more strengths than limitations, and once again the panel must be lauded for their efforts. My comments below focus largely on the few – but very important – limitations that are present in the Report.
Generally yes, with three important exceptions:
Performing this analysis as an estimate of the effect of MeHg "if ingested in fish that are lacking in nutrients that could have a beneficial effect" is wholly invalid. This description and interpretation have little scientific justification and defeat the purpose of the Report – to evaluate the risk to U.S. consumers from MeHg in commercial fish products. There are certainly no commercial fish products in which MeHg is present without any other beneficial nutrients, and indeed there are no fish products of any kind in the world in which MeHg is present without any other beneficial nutrients. In no uncertain terms, this interpretation of this analysis, and any and all references to "fish consumption" in the descriptions of the modeling results, must be removed. Descriptions could remain that discuss "maternal consumption of MeHg up to xx amounts per day in foods other than fish" etc., but not fish.
Importantly, even the first interpretation (MeHg from foods other than fish) is not meaningful for exposures in the U.S. public, and further this is not a document to investigate the potential effects of MeHg in isolation. Thus, this modeling should be deleted altogether as it is not based in any real scientific scenario and presents highly misleading and confusing "findings" in a Report of quantitative risk-benefit assessment of fish consumption. Neither EPA+DHA-only, selenium-only, nor vitamin D-only analyses are performed – why should MeHg-only analyses be performed and highlighted so strongly? The MeHg-only analysis could be included in an appendix (but certainly not in the Executive or other Summary!), but with clear definitions that this is an analysis of MeHg exposure not from fish, and with removals of all references to fish consumption as a metric, as described above.
A good example of clear and quantitative presentation of results is seen on page 4, paragraphs 2-3, e.g., that includes specific percentiles of consumption, gains in IQ points, and oz's per week of fish.
A converse example is seen on page 4, paragraph 4, in which vague and imprecise wording –"relatively high," "could," "reduce the benefits," "unusual cases," and "a fraction" – leaves interpretation to any reader's imagination of what these words mean. This imprecision should be replaced with quantitative language: i.e., at what levels of mercury in fish and what levels of consumption such effects could be seen; the CI or some other measure of the word "could;" by what extent, quantitatively, the benefits could be reduced (which should also be re-emphasized as being very different from net harm); what percentile of the population is likely represented by "unusual;" what precise fraction of an IQ point could be lost; etc. Development of clear and quantitative answers to these questions is obviously challenging, but this indeed is the charge of this Report, and this single paragraph in many ways is the core of the neurodevelopment conclusions.
Similar vague and confusing imprecisions are seen in the summary of the effects of the models (page 19, page 21). The description of the Combination Net Effect model is unclear and very difficult to follow in these instances. For each of the major predicted effects, the description of the findings should be explicitly quantitative, and a summary Table or preferably Figure should be included. Page 21, Combination Net Effect model, what is the likely quantitative benefit? To what metric does "four percent probability" refer? Who are "some people," and how large is this group? How big is the "larger net benefit?" etc.
Another important example: Page 20, paragraph 3, last sentence. "In this case…" How plausible is this scenario? What proportion of commercial fish, if any, contain high MeHg and low nutrients? Do what proportion of fish consumers does it apply? How likely, quantitatively, is a "small adverse effect," and what does "small" mean? Again, sentences like these are the heart of the Report, and should be clear and quantitative.
Similar quantitative details should replace the indistinct wording describing findings for CVD (page 5, paragraphs 2-3; page 26, paragraph 2). For example, page 5, paragraph 3, states "a small possibility of increased risk." Several points should be made clear. First, two estimates were performed, and the similarity of the central estimates should be emphasized (and the explicit numbers of prevented CHD deaths included). Second, only one estimate had a CI that included zero, and the percentile at which this was seen should be specified (e.g., was it at the 92% upper limit?). Also, if the Summary mentions results for the outer bound in one direction, it should include the results for the outer bound in the other direction. Thus, rather than the vague statement "a small possibility of increased risk," a clear and quantitative statement should be provided, e.g., "Using two techniques, both central estimates predicted ~35,000-40,000 fewer CHD deaths due to current fish consumption. The 95% CI's of one estimate only included benefits of ~5500 to 60,000 fewer deaths. The 95% CI's of the second estimate, which were much broader due to differences in assumptions relating to uncertainty, included an ~8% probability of no benefit and, at the upper limit, an ~5% probability of up to 50,000 greater CHD deaths, but also included at the other extreme the possibility of up to 250,000 fewer deaths due to current fish consumption." [Much of the confusion surrounding these impossibly broad CI's will be mitigated by re-assessment of the fish and CHD effects using more reasonable, appropriate, and comparable assumptions – see comment 5bA].
The above are only examples – the Thumbnail and Executive Summaries (and the body of the report) should be carefully reviewed to minimize imprecise statements and maximize quantitative conclusions.
The quantitative results for the "what if" scenarios for CHD should also be included in both Summaries.
The body of the report also gives short thrift to CVD. Following Section II-A's 30 pages of careful discussion and evaluation of neurodevelopment literature, including substantial (and appropriate) attention to studies of benefits of fish consumption, Section II-B gives 8 pages to the MeHg papers (a remarkably porous and limited set of data, if judged using an objective set of hierarchies of evidence quality [see 2A, below]), and then nothing at all on the numerous large prospective cohort studies of CHD death totaling >300,000 individuals, the many biomarker studies of CHD death, the many large prospective cohort studies of nonfatal CHD or total CHD outcomes, the many large prospective cohort studies of stroke, and the several large RCTs (>35,000 subjects) of fish or fish oil for both primary and secondary prevention of CHD. Placing the full account of this embarrassingly rich data in the Appendix is okay, but at least several pages of summary of this rich data must be added to Section II-B, explicitly detailing the relative sizes, strengths, and robustness of these studies compared with the MeHg and CHD data (and even compared with all the neurodevelopment data).
Generally yes, with two important exceptions:
For example, factors affecting evidence quality include study design (RCT of disease outcomes > prospective cohort of disease outcomes >> RCT of intermediate risk factor >> retrospective case-control study of disease outcomes >>> observational study of intermediate risk factor >>> case-series); power (driven largely by number of events); accuracy of assessment of exposure, outcome, and covariates; avoidance of multiple hypothesis testing (often too frequent in neurodevelopment studies); control for confounding; statistical power; generalizability; etc. Some of these issues are occasionally mentioned for specific studies, but a clear description of hierarchies of evidence quality is needed.
Based on emphasis in the report (e.g., page content alone), one would be led to believe that the best evidence comes from small retrospective case-control studies and poorly controlled case series; then somewhat less robust evidence from modestly sized retrospective case-control studies and small prospective cohorts; then somewhat less robust evidence from short-term randomized controlled trials of intermediate phenotypes, and then the least robust evidence from large prospective cohort studies and randomized trials of disease outcomes. Clearly, this order should be exactly reversed, together with reversal of the report's greater attention to relatively small studies with many limitations vs. little attention to large studies with greater strengths.
A good place to formally address the issues of hierarchies of evidence is page 12, paragraph 3, by adding 2-3 paragraphs of explicit details for strengths and limitations of different study designs, especially within the category of observational studies. A summary figure or table of evidence hierarchies would be particularly useful. In the detailed descriptions and interpretations of individual studies in the body of the report, the panel appears to have taken some of these issues into consideration, but the Report lacks an explicit summary of criteria for how these different studies were judged. Of particular relevance are (a) the RCTs of DHA for neurodevelopment (largely postnatal, but some prenatal), that demonstrate biologic benefits of increasing DHA above current background intakes; (b) the RCTs of fish or fish oil for CHD, performed in both primary prevention populations (JELIS) and patients with established heart disease (DART, DART2, GISSI-P, GISSI-HF), that demonstrate reductions in CHD for increasing EPA+DHA above background intakes; (c) meta-analyses of fish oil RCTs for total mortality, arguably the most important outcome; and (d) the numerous large prospective cohort studies of fish intake and CHD death. Indeed, the RCTs of fish oil (both for neurodevelopment and CHD) are relatively underemphasized in the report (certainly in the Summaries), whereas they provide some of the strongest evidence and should be especially emphasized. The idea that RCTS of fish oil somehow do not inform us about effects of fish consumption is disingenuous – the corollary for such a fallacious argument would be that the Iraq poisoning tells us nothing about health effects of MeHg in fish simply because the exposure was not fish.
One of the recurrent tenets of the Report is that nutrient-specific benefits of fish could not be assessed. This is incorrect. First, many RCTs have addressed effects of EPA+DHA for both neurodevelopment and CHD, with results largely quite congruent with observational studies of fish consumption. That these RCTs often tested a variation of the hypothesis compared with the observational studies should be highlighted as a strength, not a limitation, of the overall evidence. For example, neurodevelopment RCTs of DHA were often (though not exclusively) postnatal, whereas observational studies of fish consumption were often prenatal; and CHD RCTs of fish or fish oil were often (though not exclusively) in patients with established heart disease, whereas observational studies of fish consumption were often in patients without known CHD. The correspondence of the results of these related but not identical hypotheses tested is powerful. As described above, it is disingenuous to state that the similar results of several large prospective cohort studies of maternal (prenatal) fish consumption and neurodevelopment in many countries and several RCTs of DHA supplementation (mostly postnatal, one prenatal) and neurodevelopment are not strongly co-confirmatory simply because one group of studies were observational and assessed prenatal fish while the other group were RCTs and assessed postnatal DHA – rather, the similar results using different populations, study designs, and exposures (fish vs. fish oil) provides even stronger confirmatory evidence than if the studies were more similar. It is also disingenuous to state that the similar results of many large prospective cohort studies of fish consumption and CHD death in many countries totaling >300,000 individuals and now several RCTs of fish or fish oil and CHD death in many countries totaling >35,000 individuals are not strongly co-confirmatory simply because one group of studies were assessing fish intake in individuals without known disease and the other largely (but not exclusively) fish oil intake in individuals with known disease – rather, the similar results using different populations, study designs, and exposures (fish vs. fish oil) provides even stronger confirmatory evidence than if the studies were more similar. Based on these large bodies of evidence, EPA+DHA are clearly major active ingredients for both neurodevelopmental and CHD benefits of fish intake. This does not exclude the possibility that other factors in fish, particularly selenium and vitamin D, may also have benefits, but EPA+DHA are clearly the major players.
Second, the Report explicitly uses a biomarker of dietary exposure for one constituent in fish, i.e., MeHg levels, to derive quantitative models and conclusions about the specific effects of this factor on health. This is done even though other factors are present in fish; it is possible that effects of this constituent are confounded by other constituents (both beneficial and harmful); and the levels may not perfectly reflect diet due to metabolism and changes in exposure over time. Indeed, this type of assessment is the rule rather than the exception in toxicologic analyses.
Given this treatment of MeHg, there is no justifiable reason for not performing comparable and objective analyses for a different biomarker of dietary exposure to another important constituent in fish, i.e., long-chain omega-3 (EPA+DHA) levels. In all respects, use of tissue levels of MeHg as a biomarker for assessing health effects is entirely analogous to use of tissue levels of EPA+DHA as a biomarker for assessing health effects.
Paucity of data is certainly not the limitation; the extent of observational literature on tissue EPA+DHA and CHD is at least comparable (if not superior) to that for MeHg and neurodevelopment, and certainly far exceeds the literature for MeHg and CHD. Arguments for inability to extricate effects of EPA+DHA vs. other nutrients cannot be made. First, if so, identical arguments could be made for MeHg, that is certainly present with other toxins and nutrients yet is evaluated for its independent effects. Second, EPA+DHA is clearly a major active nutrient for both neurodevelopmental and CHD benefits, with RCT data supporting, in consistent doses, directions, and magnitudes, the observational benefits of EPA+DHA for both neurodevelopment and CHD benefits (whereas no such trial data exist for MeHg – the closest corollary is the accidental MeHg exposures that are clearly limited in dose-comparability).
Thus, the Report's assumptions in assessing effects of MeHg (evaluated as a biomarker, concluded to be causal based on observational studies only) are radically different from those for EPA+DHA (evaluated only indirectly as fish intake in dietary questionnaires, always discussed with major caveats as to causality given the observational nature of the evidence). This cannot be justified on any biologic, epidemiologic, or scientific grounds – the only reasons for these discrepancies are the historical traditions of toxicology assessments vs. nutritional assessments.
To provide an objective and comparable assessment of benefits and risks, and to communicate these appropriately, several issues must be addressed. First, the biomarker studies of health effects of EPA+DHA (many for CHD, a few for neurodevelopment) should be discussed, including in the Summaries. Second, the difference in exposure (mis)classification between biomarkers and estimated dietary intake, and the implications of these differences, should be considered and discussed. Exposure misclassification that is largely random (e.g., in prospective studies, although not always so in retrospective studies) can strongly bias results toward the null, and such misclassification is much larger for estimated dietary intake than for measured tissue biomarker levels. Consequently, estimates of health benefits based on estimated fish consumption will be much more biased toward the null, compared with estimates of potential harms based on tissue MeHg levels. This bias toward the null can be quantified by comparing studies of estimated fish intake and sudden death vs. tissue EPA+DHA and sudden death: in the former, the RR estimates indicate a 3-fold (~33%) lowering of risk, whereas in the latter, the RR estimates indicate a 10-fold (90%) lowering of risk. Consideration of these critical issues, and clear description of them, would lead to appropriately stronger conclusions about benefits vs. risks.
Finally, a quantitative assessment, or review of prior assessments, of quantitative benefits of EPA+DHA should be included: this should be done for both neurodevelopment and CHD, that each have both observational and RCT evidence on effects of EPA+DHA. It is understandable (although unfortunate) that the panel may not wish to carry out such new quantitative assessments for this Report, that would require substantial additional time and work. However, in this case, this omission should be made explicit and emphasized, not because "we were not able to assess these" (page 5) or "we lack the data necessary" (page 144), but because "we chose not to do so, even though such analyses are possible, scientifically justified, and have been done by other groups."
Generally an excellent job of selection and inclusion of papers. Two RCTS should be added: JELIS, a large RCT demonstrating benefit of EPA for CHD in a predominantly primary prevention population, GISSI-HF, a large RCT demonstrating benefit of EPA+DHA for total mortality in a patients with established heart disease.
Yes, with the exception of the limitations detailed above. Attention to these limitations would improve the appropriateness and clarity of the conclusions about relative risks and benefits, as well as relative strengths of evidence for these different effects.
Somewhat. Important limitations that require attention:
The Report must either re-model the effects for MeHg and neurodevelopment and fish and neurodevelopment using similar methods as for fish and CHD (i.e., using all available estimates from all reports, fitting data from individual studies rather than pooling, adding sampling error by not assuming a common variance, and adding further tremendous uncertainty by not modeling a common effect across studies), or re-model the fish and CHD effects using more similar methods as for MeHg and neurodevelopment and fish and neurodevelopment.
The Carrington model for CHD death gives outrageously large CI's for an effect based on such enormous amounts of data, compared with the modeling and CIs for MeHg and neurodevelopment and fish and neurodevelopment that are based, in this Report, on such relatively little data. The resulting uncertainties in the Carrington CHD estimate are so large – ranging from -208,000 to +29,000 CHD deaths in men 46+ due to current fish consumption – as to render the estimate nearly meaningless from a practical standpoint. In particular, as modeled, the assumption regarding an independent (different) effect in each individual study of fish and CHD is unwarranted, and such a drastic assumption is not made for MeHg and neurodevelopment or fish and neurodevelopment. Removal of this drastic assumption would provide much more valid estimates for fish and CHD and importantly, render the methods for this estimate at least partly more similar to methods used for other endpoints.
Finally, data from biomarker studies of EPA+DHA should also be included in the modeling of fish and CHD effects – the data for MeHg is entirely, after all, from biomarkers. (Data from RCTs of fish and CHD could also be included, although this is less essential.)
Similarly, for fish and neurodevelopment, the additional data from other studies should be included in the quantitative assessment, at least in sensitivity analyses. The absence of availability of individual-level data does not mean that this other data should be ignored – sensitivity analyses should be done to include and account for these several other studies (e.g., using pooling or other similar methods as were used to combine the studies of MeHg and neurodevelopment).
See comments elsewhere.
No – these endpoints are the only ones for which sufficient evidence exists to make such calculations.
Yes, they are reasonable.
Page 4, paragraph 4: The concept "reduce the benefits" should be emphasized as being quite different than "net harm."
Page 11: A nice summary of safety assessment issues for MeHg. Discussion here is crucial regarding the traditional lack of "safety" assessments for low consumption of fish, that could be as least as harmful (and likely more, based on the conclusions of this Report and other studies) for both neurodevelopment and CHD as MeHg. Toxicologists often communicate in terms of safety and risks, whereas nutritionists often communicate in terms of benefits – these are of course identical concepts, requiring only a change of the reference group. If the reference group were made to be adequate fish consumption, what are the safety limits and risks for the population of having lower consumption?
Page 13, paragraph 4: "The researchers anticipated that effects would appear as subtle differences…" The subtlety of the neurodevelopment effects in question, throughout this Report, should be expanded and emphasized further here and elsewhere.
Page 21, last paragraph: As differences in risk of adverse effects above and below 12 oz/week are not large, and consumption >12 oz/week includes a possibility of larger net benefit, what the do the final conclusions indicate about the utility of the 12 oz/week limit? This is also at the heart of the purpose of the Report. The data appear to suggest that, at the current state of knowledge, the 12 oz/week limit should be discarded and that women should eat a variety of fish and (since commercial fish are at the low end of the spectrum on average) more so than 12 oz/week. An explicit conclusion should be drawn here about this.
Page 22: Summary of RCTs of DHA should be included here.
Page 35: It is a shame that "the primary practical function of the consumer advisory is to encourage 4.65% of women to increase the margin of safety for their fetuses." The primary practical function should be to improve the net health of U.S. women and neurodevelopment of their babies.
Page 37: Title of this Section should be "Scientific Basis For Risk and Benefit Assessment." Also, this section did not review "studies that looked for associations between the consumption of fish and health endpoints." Many health endpoints were not evaluated, including fish and cognitive decline, fish and atrial fibrillation, fish and depression, etc.
Page 40: Two other major categories of studies should be added and reviewed:
Page 73: The much higher levels of MeHg and the absence of selenium in the Finland study, compared with the U.S., should also be discussed in the Executive Summary.
page 74: The Kuopio authors' own conclusions that mercury in fish "could attenuate the protective effects" of fish consumption, and the very important difference between (a) attenuation of protection and (b) net harm, should be included and emphasized in the Executive Summary. (This is also the case for neurodevelopment outcomes).
Page 75: Guallar et al. The retrospective case-control design (subject to strong selection bias) and the inclusion of only nonfatal CHD (not fatal CHD, the focus of this report that is much more strongly benefited by fish consumption than nonfatal CHD) should be emphasized. This design is very weak to investigate the effects of exposure on fatal CHD, given the strong possibility of selection bias of controls in retrospective studies and the inclusion of only survivors. These issues should also be discussed in detail when defining the general "hierarchies of evidence" that form the background for this report (see comments in 2A). Results for DHA in this study – showing net benefit with higher consumption, and only an attenuation of benefit, but not net harm, before adjusting for mercury – should be included.
Page 77: Limited power to detect associations is not a strong argument here, as the number of cases in this prospective nested case-control study was much larger than in the Finland study.
Page 139: "The collective size of these studies… compares favorably [emphasis added] to the… studies that measured mercury exposure." The understatement of the Report!! The numbers should be included here: the numerous large prospective cohort studies of CHD death totaling >300,000 individuals, the many biomarker studies of CHD death, the many large prospective cohort studies of nonfatal CHD or total CHD outcomes, the many large prospective cohort studies of stroke, and the several large RCTs (>35,000 subjects) of fish or fish oil for both primary and secondary prevention of CHD – probably in total nearly 1,000,000 individuals studied.
Page 144, 147: Table IV-14, fourth row heading, and Table IV-16, each row heading, clarify that these estimates are for current fish consumed, e.g., "Median change in CHD death rate due to current fish consumption."
Page 163: Cannot make similar assumptions for nonfatal CHD, as effects are likely smaller – see comments elsewhere. No evidence, however, that fish consumption has different proportional effects for fatal vs. nonfatal stroke.
Page 241: Line 11: "Interestingly," should be replaced with "As would be expected." "Substitution" arguments hold little water in assessing potential mechanisms for benefits of fish consumption. First, the average amounts of "harmful" macronutrients (saturated fat, trans fat, and dietary cholesterol) in, for example, meat that would be replaced by 8 oz per week of fish is remarkably small and would have little impact on overall daily dietary intakes of these dietary factors. Second, for "substitution" effects to have any meaningful mathematical impact on risk, predominantly one type of food or foods would have to be replaced nearly uniformly by the entire population consuming fish, not only in the U.S., but in all countries in which similar effects of fish consumption have been seen. The plausibility of this is very close to zero. Third, the similar effects of small amounts of fish oil (1 g/d = 9 calories/day!) on CHD in RCTs, compared with observational studies, indicates that the benefit is due to what is present in fish, not what is not present. A discussion of these issues, including in the Executive Summary, would be helpful to put this "substitution" notion to rest.
Overall, the report presents a very thorough analysis of the risks and benefits of fish consumption in the U.S., and I commend the authors on this important work. The report provides both updated reviews of critical literature and helpful links back to prior risk assessments on this topic that I found very helpful in sifting through the weight of the evidence. I particularly appreciated that the authors were very careful to highlight critical data, including both old and new studies, and that they made their assumptions very explicit throughout the document. The information appears accurate, although I did not investigate the underlying models exhaustively. I had access to some of the spreadsheets used to support the analyses in the report and I found no errors in my review of them, but these were not the primary focus of my review. If they are released publicly, the spreadsheets would benefit from much better documentation for purposes of review, particularly given the relative complexity of two-dimensional Monte Carlo analyses.
With respect to the report, I think that it would benefit from substantial editing and that this will significantly improve the clarity of the presentation. The report reads as if it was written by multiple authors, with writing in passive third-person in some places and active first-person in others. (I strongly encourage use of active first-person throughout, which provides a much friendlier and accessible tone to the reader). I found it striking that the main body of the report contains almost no figures. Although the report includes many tables, I found numerous places in which I would have personally preferred to see a figure. For example, many of the data and results are presented as percentiles in tables, but I would prefer to see the cumulative distribution functions presented with the means clearly noted on them (and then being able to look up the actual numbers associated with various percentiles in an appendix). I appreciated very much that the authors used appendices, which kept the text in the main chapters contained. However, I very much missed having a roadmap to what I would find and where I would find it within the document (beyond looking at the list in the Table of Contents). I would suggest changing the title of the "Thumbnail Summary" to "Abstract" and adding a section at the end of the "Abstract" or "Executive Summary" that is a "Roadmap" for the report (brief descriptions of the subsequent sections that would then lead to the Table of Contents and then the Executive Summary). I am not clear about the anticipated audience for the Executive Summary, but I would suggest that while the content of the Executive Summary is very good, the writing itself probably needs some work with respect to writing it for the appropriate audience. In the attached I suggested heavy edits to the "Thumbnail Summary" and the beginning of the "Executive Summary," but I suggest that a professional editor be hired to edit the entire document.
Overall, the conclusions of the report appear sound. I commend the authors for conducting a rigorous analysis and I found the versions of the model that they developed (the "Carrington" models for CHD and stroke) to be very interesting and persuasive with respect to the robustness of the results. If anything, the authors appear to be understating the significance of their findings. The message that increasing fish consumption by 50% would decrease risks of CHD and stroke and that in general most Americans need to eat more fish to improve their health could easily be lost, but this is the most critical insight and message from all of the analysis in my opinion. The fact that women of childbearing age and their children would most likely benefit from eating more fish low in MeHg is also very important, particularly given the common perception created at least in part by the prior EPA/FDA advisory that pregnant mothers who eat fish are endangering their babies. Similar to the prior advisory, the message from this report is nuanced, but it is clear that eating fish provides significant health benefits to nearly all (if not all) members of the population, with fish low in MeHg and high in n-3 fatty acids appearing to offer the greatest benefits. I think that the conclusions are correct, but as noted above they may not be clear enough for all audiences.
The report is generally well-written, but the organization could use some improvement and it would help if the report specified the target audience(s) (at least making clear who should understand the Abstract, Executive Summary, parts of the main report, and appendices). As noted above, I believe that the document would be significantly improved if it had a roadmap at the beginning that made clear what the reader should expect in each section. Some sections also feel repetitive to some degree (e.g., going through the analysis for CHD and then similarly for stroke), but this is a necessity. However, I think that the authors should do as much as they can to make it feel less repetitive. I also think it might make sense to reorganize the report to put the CHD and stroke results before the fetal neurodevelopment results since readers might find it easier to understand these. This would mean that large sections of the report and subsections within would need to be reorganized, but I think it would help the report to better emphasize the results that are relevant for most US consumers.
I think the text needs a lot of tightening up and that the authors should make the report more engaging by adding some figures and graphics that can supplement or replace some text or tables in places (see my suggestions in the track changes version). I also think that the authors will run into challenges with some of the concepts that they use. As a scientist with some training in statistics I found their use of the Z-scores helpful, but I do not think that this will work with a broader audience. I hate to say it, but I think that they should keep the Z-scores in the background and technical parts of the document and convert all of the results to the natural interpretation (e.g., increased by X%). In most (if not all) places that mention the Z-score the interpretation is also provided, so I am mainly suggesting that the Z-score aspects be relegated to an appendix or reported in parentheses instead of as the main result.
I was very impressed by the thorough explanations that the authors provided for their assumptions throughout the document. The authors took the time to review the critical evidence, and to explain why they made particular choices in their modeling (e.g., in the descriptions of the "Carrington" CHD and stroke model choices). I did not find any assumptions that seemed inadequately justified or inappropriate, and in fact, I commend the authors for their resourcefulness in obtaining, analyzing, and combining data using rigorous statistical models. The authors also carefully specified appropriate input distributions for their models with consideration of both uncertainty and variability.
The authors have done an excellent job summarizing the existing literature. Given that studies continue to come out related to this study topic, it is not surprising that the report might be missing some recent relevant papers (e.g., Oken et al., "Associations of maternal fish intake during pregnancy and breastfeeding duration with attainment of developmental milestones in early childhood: A study from the Danish National Birth Cohort." American Journal of Clinical Nutrition, 2008; 88(3):789-796).
Another paper that I think would be useful to include is: Hicks, Doris, Pivarnik, L, McDermott, R. "Consumer perceptions about seafood - an Internet survey." Journal of Foodservice 2008; 19(4):213-226. (DOI: 10.1111/j.1748-0159.2008.00107.x), which surveyed U.S. consumers about "seafood consumption frequency, sources of information about seafood and preferred formats, knowledge of key seafood issues, and barriers to seafood consumption," and "if they had heard positive or negative information about seafood and where they heard this information." I believe that this study and prior literature that it cites might provide some additional useful context about seafood consumption (and therefore exposure), consumer perceptions of risks and benefits, and the potential impacts of advisories.
The conclusions are appropriate and they follow very logically from the quantitative risk assessment. With the multiple what-if scenarios that the authors considered and their careful and thorough discussion of all of the assumptions, I think that the main challenge that readers will face will be sorting through the numerous results to get the major messages.
The report provides sufficient information and explanations throughout. I was particularly impressed with the detailed discussion of the specific inclusion criteria for the data used, and that the authors developed their own models (i.e., the "Carrington" versions) based on their best judgment. I believe that the methodology they used was appropriate and that it represents the state-of-the-art in the field. The other data that the authors might include (mentioned above) relates to analysis of consumers' perceptions of risks from eating seafood and how these might impact food choices and exposure. I think that this might be a useful addition because it may suggest that one message from the report should directly address the impact of the prior EPA/FDA advisory on health.
One issue that the authors need to make sure that they check carefully throughout the report is that they always refer to methylmercury when that is what they mean. In some cases they just say mercury, and this leaves the reader wondering if they mean total mercury or methylmercury. The authors also assume that methylmercury and organic mercury are completely synonymous, and while methylmercury most likely represents the main type of organic mercury, other forms are also possible. I think that the assumptions made are appropriate, but the existence of other forms should probably be mentioned somewhere.
The uncertainty analysis represents a state-of-the-art analysis for which I commend the authors. I believe that the uncertainty analysis adds a large degree of confidence that the main insights are robust, in spite of wide ranges for some of the confidence intervals. The ranges presented encompass the range of possibilities, but more importantly they provide insights about the likely probabilities. The authors are cautious to note that their main conclusions apply to most people, but may not apply to all, and they provide analyses that explain why prior studies provided inadequate information. For example, I found the report's description of the most recent Faroe Island results that separate the fish and marine mammal (pilot whale) data particularly helpful in explaining what previously has been puzzling to me with respect to why the Faroe Island and Seychelles Island studies appeared to yield different results.
The scientific assumptions are explained in detail and they are appropriate. The appendices provide a wealth of useful details about what the authors did and they provide the support needed to understand the science underlying the assumptions.
The authors have included the significant endpoints and I do not believe that they should be modeling any others. The report is very dense as it is, and I would not suggest that the key findings be diluted by discussion of other possible endpoints. The one main omission that the authors explicitly state is beyond the scope of their report relates to the substitution of non-fish foods that might pose larger health risks if people reduce their fish consumption and/or additional benefits that might result from consumers substituting fish for other foods in their diets that might lead to benefits beyond the CHD and stroke benefits. I believe that some studies exist that explore these potential risk-risk tradeoffs in some depth and that the authors should probably cite that literature and further discuss the challenges of characterizing any additional benefits that might exist.
The intervention scenarios support a very useful set of analyses that appropriately represent the possibilities. I found it particularly helpful that the authors explored the impacts of both increases and decreases in fish consumption and discussed the uncertainty around the estimates. The what-if scenarios for fetal neurodevelopment results are more difficult to understand, even if one understands Z-scores, and they are also complicated by the different results obtained with the different models used (i.e., Combined Net Effect, Age-of-Talking, and Age-of-Walking). This is really the main reason for why I suggested above that it might make sense to put the CHD and stroke results before the fetal neurodevelopment results.
A mark-up of the document with track changes has been provided to the FDA/CFSAN authors.
This reviewer was very interested in reading this report and this reviewer applauds the authors for delving into the extensive literature on methylmercury and fish consumption to accomplish this analysis. The ideas and approaches for modeling are interesting and exciting. However, this reviewer has major areas of concern and the focus of this reviewer's extensive comments are present in several areas including: the choice of the data to model and implications of what was not modeled, interpretation of the modeling results and how this interpretation is placed in FDA's role in establishing allowable levels of MeHg in commercial fish. Many assumptions are made and these are clearly delineated in almost all cases, however many of the initial assumptions, in this reviewer's opinion, would severely limit the interpretation and application of the results. These limitations are not designated as clearly as needed and the recommendations on how to use the model findings imply that these would not limit application. In that regard, this reviewer's conclusions frequently differed from the authors. In regards to the question of whether major revisions are needed, yes, lots of clarification and discussion and suggested changes will be needed. This is because two of the key areas noted above on which this reviewer's comments will focus, are ones that have or could have significant impact on the final conclusions and extrapolation of FDA's modeling for risk assessment. This reviewer feels that these questions and requests for additional considerations are significant as the ideas proposed in this report would alter considerably how we currently are conducting risk assessments on this topic.
The document is logical and clear but very detailed. In order to fully understand the document it is this reviewer's opinion that the appendixes must also be read. Unfortunately this reviewer's comments are in most cases making suggested modifications and additions rather than providing ways to shorten the document. In fact, in several places, this reviewer suggested moving information from the footnotes back into the report. As mentioned in my summary it is when assumptions are made in order to accomplish modeling comparisons that this reviewer had most difficulties with the report as the limitations in the assumptions were not always carried forward to limit the conclusions or applications. It is in this regard that this reviewer feels that the report is not always "logical."
Please also refer to the numerous specific comments below that specify where clarifications are needed.
Many scientific assumptions were made in the document and this reviewer feels that most assumptions were explained. In several very important places this reviewer feels that the assumptions made are not then used to qualify the results and at least one of these situations is with a major conclusion of the report. Please see the detailed comments below as well as reviewer comments for question (5c).
Specific response number 49 also lists another example of assumptions that are made and which are not well analyzed and are in conflict with other (NAS) reviews.
The document does an excellent job of citing highly relevant and current literature. In only a few places did this reviewer have some additional suggestions. First, please refer to specific comments 46 and 47 below regarding fish consumers and non-consumers and appropriate literature. See also references that are listed as part of comment number 35 on regional and cultural differences in fish consumption and relevant reports. There is also a new, in press report evaluating national fish advisories that would be good to cite in this report. It is Scherer, et al. 2008, Environmental Health Perspectives 116: 1-9.
Not always and these inconsistencies are the most significant of this reviewer's comments. In many cases the initial assumptions made in order to allow for modeling of the varied endpoints would result in significant qualifications of the model output. Please refer to specific review comments that are listed below. For example, in specific comment number 80 for page 159, second paragraph: Authors make important points about model estimates. Yet this reviewer was left wondering what to do with these "problems" in model estimates. This paragraph states that "For this same subpopulation, the "Carrington" stroke model's central estimate is that 127 stroke deaths are being averted for this subpopulation from fish consumption but at the fifth percentile confidence interval it also estimates that fish consumption could be averting up to 6,476 deaths." For this latter estimate to be correct, fish consumption would have to be averting over four times more stroke deaths than occur in this subpopulation on an annual basis. At the other end of the spectrum – the 95th percentile of the confidence interval – the "Carrington" stroke model predicts that fish consumption may be causing 434 stroke deaths, i.e., roughly one-third of all stroke deaths for women aged 16-45. Such an outcome seems implausible given the nature of the underlying data (see previous footnote). The results from the fifth and 95th percentile confidence intervals raise similar issues for other subpopulations." This reviewer was left wondering if such "implausible" observations at both ends of the dose response model outputs negate these models. Authors need to provide more in terms of what these observations mean for their overall risk assessment.
No, not always. For example, this reviewer asks the FDA modelers to re-examine how they define consumption rates in their models. Please see specific comments number 31 through 37. Also this reviewer does not feel that variability in cultural and regional differences in fish consumption was adequately modeled. There are just two examples of numerous examples detailed in (5c).
No, again, not always. For example in specific comment number 54, very important neurodevelopmental studies were excluded from the model. See also reviewer response number 65 for CHD.
No, not always. An example of where inappropriate assumptions were made in order to facilitate modeling and which was done with minimal analysis following is seen in the example given for specific comment number 54 regarding what was modeled and what studies (in particular studies with positive findings) were not modeled. This is only one example but an important one which has important implications for neurodevelopmental endpoints. Another example is found in specific response number 78 where this reviewer is asking for explanations for the biological assumptions in the CHD versus fatal stroke models. Exactly what studies are relevant and which are not is an important decision that was not clear in the text or in the application.
Because of the limitations in the literature, the report correctly notes that some endpoints could not easily be modeled and when they were, they required assumptions to be made. Hence the answer to this question is yes, but the human literature is limited and cannot be used to fully answer many of the relevant endpoints. As I note in my specific comment number 54, these limitations resulted in "leaving out" some studies because they were not in a form that could be modeled in the current state of the models. In specific comment numbers 30 and 50, this reviewer asked whether the human data could be supplemented by available primate data. This would be the largest area of available research that was not modeled but which could still be relevant.
My reviewer comments also document many limitations in modeling the benefits of fish consumption where fish intake was measured but that methyl mercury levels had to be calculated based on that intake and not on measured values. This highlights another area where additional data is needed but not everything is available in every study so the report does a heroic effort towards trying to model what is available.
See specific comments numbers 44 and 45 that suggest that true separation of individual versus population modeling was not achieved. See also specific comments number 41 through 47 that discuss the need to ensure that modeling is done for consumers not consumers and non-consumers as an average of the population.
In specific comment number 44, this reviewer also noted that peak exposures during critical time points during development should not be forgotten nor averaged over total developmental period without regard for modeling peak exposure which could occur during these critical time points of development.
An important aspect of these risk questions from an overall public health view that was not modeled was the "big picture cumulative methyl mercury model." For example, the report largely dismissed the importance of modeling the 20 percent commercial fish consumption that the FDA regulatory jurisdiction would be concerned with within the broader context of potential contribution of methyl mercury exposure for the other 80 percent fish consumption that was non-commercial and not within FDA control. For many other pollutants, an approach is used that is based on a relative source contribution model where each source is modeled within the total contribution. In this case when recommendations are made for 12 ounces of fish per week for pregnant women and that these calculations are very close to a fine balance between benefits and risks without considering the variability and quantitative contribution of the other 80 percent, individuals making these decisions and public health officials outside of the FDA regulations find it very difficult to provide the "broader context" view. This lack of broader model was a serious deficiency in this report and limits its applicability for overall risk assessment applications. The lead IEUBK models have shown us how important these source contribution evaluations can be for overall risk evaluations. A better kinetics model for total methylmercury is needed to match the sophistication of the endpoint and benefits modeling.
This reviewer was a bit confused by this question and asks: Appropriate for what? The document specifically lists "Intervention Scenarios" from pages 224-230, however throughout the report various advisory scenarios are described and modeled in great detail. This reviewer will respond to the broader discussion. Yes, the advisory scenarios are ones that hinge upon the FDA advice for commercial fish and they are discussed in appropriate detail; however, the public and consumer is trying to answer a broader question. In fact, the consumer's question is: Is it safe to eat this fish? And how much can I eat? And they expect a synthesized response that deals with risks from multiple pollutants and toxins that may affect neurodevelopment and that may be present as a mixture in fish tissue. However, because of jurisdictional responsibilities, as risk assessors, we have to answer from our appropriate contexts. Thus, are the intervention scenarios appropriate, yes, looking at what happens above and below the 12 ounce guidance makes sense for this report. The larger question that this reviewer is struggling with is whether these are the appropriate ones for the consumer who needs to make informed decisions about eating fish or red meat, or soy or fish, etc. In these cases, the scenarios developed only for methylmercury and for only commercial sources are limited and do not match consumer needs.
References to add include:
| Parameter | Group | n | GM | Mean | Percentiles | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| 10th | 25th | 50th | 75th | 90th | 95th | |||||
| Estimated Hg intake (µg/kg/d) |
Japanese Cohort | 106 | 0.09 | 0.14 | 0.02 | 0.05 | 0.09 | 0.18 | 0.25 | 0.37 |
| Korean Cohort | 108 | 0.05 | 0.07 | 0.01 | 0.02 | 0.05 | 0.09 | 0.15 | 0.19 | |
| NHANES 1999-2000a | 1,727 | 0.02 | NA | NA | NA | NA | 0 | 0.04 | 0.13 | |
| Hair Hg (ppm) | Japanese Cohort | 106 | 1.23 | 1.57 | 0.5 | 0.78 | 1.37 | 1.96 | 2.68 | 3.52 |
| Korean Cohort | 108 | 0.61 | 0.75 | 0.29 | 0.39 | 0.67 | 1.02 | 1.29 | 1.52 | |
| NHANES 1999-2000b | 17256 | 0.2 | 0.47 | 0.04 | 0.09 | 0.19 | 0.42 | 1.11 | 1.73 | |
This reviewer is greatly concerned by several points in the exposure calculations. First that average fish consumption estimates dominate the exposure scenario. As a person in public health we are interested in protecting fish consumers not an average of fish and non fish consumers. In developmental toxicity we assume that both peak exposures as well as area under the curve (integrated exposure) are important for outcome. If average exposure is estimated by averaging fish consumers and non consumers the dynamics of peak exposures on critically important life stage sensitive days during development is lost. Authors must address these concerns in their modeling context.
This reviewer asks that for the individual calculations that this view be examined and incorporated in a way so that the individual consumers are protected. Is FDA's regulation designed to protect consumers or only consumers averaged across consuming and non-consuming populations?
Of particular concern is the fact that the NAS, 2000 report specifically states after reviewing the neurobiology of the outcomes that "The committee concluded that it would be inappropriate to pick the Seychelles study as the basis for risk assessment, given the available evidence for positive effects in New Zealand and Faroe Islands studies as well as in the Seychelles pilot study" (page 286, NAS, 2000) The committee went on to conclude that there was a "good argument" for choosing the Faroe Islands study because the Faroe Islands was large (over 900 children), had been extensively analyzed and reanalyzed, and had two biomarkers of exposure. This same study does a detailed analysis of multiple outcomes across the studies even joining two types of analyses – that of specific outcomes as well as suggestions and analyses as an integrated model. Given such a robust bases start it is surprising that the FDA study chose to not use so many studies and outcomes in their analysis. It was as if the need to model benefits outweighed the scientific judgments on outcomes – throwing out the impacts of studies and outcomes as shown in Table 10-4. Choosing the SeyChelles study which although it had over 700 children, had such a low power to detect effects (only approximately 50%) for some of the outcomes from the Faroe Island and New Zealand (see page 278, NAS, 2000) seems to counter the desire to understand potential risks and ultimately a true understanding of potential benefits of eating fish. It is important to model these potential benefits but not at the "expense" of throwing out significant findings of positive adverse health outcome.
Authors should add some acknowledgement in this section that because most women of child bearing ages do not know that they are pregnant until later in first trimester the assumptions that women of child bearing age would extend their lifestage where they would be consuming 12 oz a week is highly acceptable. Tie modeling observations back to the biology.
Likewise for fatal stroke it appears that 7 out of 10 studies are for males only. How was this handled? Why are these limitations in data not described? Rather emphasis seems to be on stroke death rate by age for gender and the fact that so few studies relevant for women are available. Again the excitement of the modeling is diminished by the lack of a thorough biological discussion of the limitations. This reviewer was disappointed as this group of authors is uniquely poised to conduct these assessments but in their excitement about the modeling forgot their own earlier caveats.
Page 164, (g) Summary of Interpretation. Fetal Neurodevelopmental risk assessment. This author looked forward to this section yet the bottom line from the modeling exercises was rather disappointing (see page 165, paragraph 4 and 5. Page 166 paragraph 1). If one combines these statements with this reviewers earlier concerns about what models are modeled then the conclusions that at 12oz fish benefits outweigh adverse healthy benefits for unborn offspring are significantly weakened from the summary statements found earlier in this document (see thumbnail summary and executive summary). Based on these observations this reviewer feels that the author should more closely reflect the limitations (qualifiers) stated in this section in the earlier conclusions on pages 4-5, 18-19 and 20-21. For example statements on page 20, second paragraph that says on line 2 "The size of an adverse effect is typically smaller, and can be much smaller, than the size of a beneficial effect." VS page 165, 3rd and 4th paragraph. "The risk assessment indicates that encouraging consumption up to, but not beyond, 12 ounces per week essentially accepts some risks in order to obtain nutritional benefits from fish consumption, while also avoiding slightly higher risks and benefits associated with higher consumption, as described below:" "Consumption of up to 12 ounces of fish per week is most likely to produce a small neurodevelopmental benefit over no fish consumption for most people but it could also produce a small neurodevelopmental decrement (up to four percent possibility) for some people. Consumption over 12 ounces per week includes a small possibility of a larger net benefit but it also includes a slightly increased possibility (five percent rather than four percent) of a net adverse effect." This section continues to say that for "age of talking," "maternal consumption of up to 12oz of fish per week would likely result in delays in age of talking of less than a day for most offspring. Consumptions over 12 ounces of fish per week could indicate the possibility of a somewhat longer delay, although the likelihood of larger delay is low." This section continues and says that "IQ: Based on the results from this model, maternal consumption up to 12 ounces of fish per week would likely result in IQ decrements for most people of less than one IQ point while consumption over 12 ounces a week would most likely result in slightly larger IQ decrements – although still less than one IQ point. For both the age-of-talking and IQ models, avoiding shark, swordfish, king mackerel and tilefish could help reduce the decrement for the offspring of women who eat these species." However the authors state two things earlier that 1) they where unable to model all of the epidemiology results and thus used a selection of studies (see this reviewers comments at #49) and 2) the authors also state that they were unable to attribute benefits to specific fish types and that methylmercury was modeled as an estimate of fish ingestions. Several of these specific factors makes this author question the wording and implied certainty both of the magnitude of adverse effect as well as attribution to specific fish beneficial factors in the overall report conclusions.
The sophistication of the modeling is overshadowed by inconsistent assumptions across endpoints. For example for one endpoint only US studies are taken (CHD) for another (fatal stroke) a mixture are taken. But these inconsistencies more significant as although there are inconsistencies in the databases available, arguments are given for each of the end points that are counter to the choices made in the next assessment. This reviewer would suggest less ambitious modeling of everything and instead use consistent assumptions for key end points and include other less robust or less relevant data points for appendix discussions. A good example of the shift in emphasis that is needed is the request by this reviewer to move footnote 55 into the main text. This is just one example of the misplaced emphasis.
The following responses are to peer review comments that requested changes or clarifications. The responses are provided in either or both of the two right side columns in the table, below. It should be noted that the peer reviewers received a single document to review that contained (1) the risk and benefit assessment with accompanying and explanatory materials; and (2) two appendices devoted that inventoried research and analyses on beneficial health effects from fish and omega-3 fatty acids. Partly in response to peer review comments FDA has divided the document into two documents that track (1) and (2) above. The first table, below, is devoted to comments and agency responses involving the paper containing the risk and benefit assessment and accompanying and explanatory materials ("Report of Quantitative Risk and Benefit Assessment of Consumption of Commercial Fish, Focusing on Fetal Neurodevelopment (Measured by Verbal Development in Children) and on Coronary Heart Disease and Stroke"). The second table addresses comments and agency responses involving the paper containing the inventory of research and analyses on potential beneficial effects from fish and omega-3 fatty acids ("Summary of Published Research on the Beneficial Effects of Fish Consumption and Omega-3 Fatty Acids for Certain Neurodevelopmental and Cardiovascular Endpoints.")
We did modify the text, however, to clarify that the modeling of individual nutrients was beyond the scope of this assessment without speculating on whether we could or could not have done so.
