HHS Log--links to Department of Health and Human Services website

Concept Paper: Quality in FDA-Regulated Clinical Research
Background to HSP/BIMO Workshop
5/10-5/11/07 4/26/07

Issues:

Can the research and regulatory communities develop consensus definitions of "quality":

Of clinical data?
For conduct of clinical research?

Can clinical research achieve both high quality and efficiency?
Can the research and regulatory communities develop a "desired state" for achieving the above?
Can the research and regulatory communities agree on concrete actionable steps to move towards the desired state?

Introduction

Quality in research is comprised of a wide range of elements. Such elements include a scientifically valid protocol, meaningful informed consent, appropriate attention to patient safety, complete and accurate recording of results, proper performance of tests and evaluations, and appropriate record verification and retention. Once a suitable study is designed and thoroughly reviewed, assurance of quality is dependent on the behavior of the clinical investigator, which is affected by training and integrity. Additionally, quality is supported by appropriate monitoring. Monitoring provides a direct assessment of certain aspects of quality, and is a means of highlighting the activities of those responsible for executing the clinical trial.
.
It is of interest that two very different approaches to study monitoring have long co-existed with the industry model. The cancer cooperative groups have utilized what might be described as periodic site evaluations rather than study specific inspections. The cooperative groups also have regularly used sites with established and experienced investigators, offering a joint venture or "team" approach that may have its own quality assurance component. For example, large studies at the National Heart, Lung, and Blood Institute have data reviewed centrally with site inspections largely "for cause," an approach that apparently is considered satisfactory and has supported major findings.

All of the different modalities that are part of "quality assurance" have strengths and limitations that could make them more important in some situations than others. They could also be focused differently depending on the stage of the clinical trial, the particular study endpoints, whether investigators are primarily collectors and transmitters of data (to sponsors or endpoint evaluation committees), or whether investigators must make critical judgments as well.

For commercial studies done to develop pharmaceutical products, on site monitoring of performance has become the rule, with sponsors visiting essentially all sites every 4 to 8 weeks to assure performance. This is costly, of course, and while it contributes to aspects of study quality, it is not really practicable for large outcome trials, and has failed to detect real fraud.¹ As interest in large trials grows, it is becoming apparent that the industry model is not feasible, without modification, for those trials.

Background

FDA inspections usually occur after the completion of a study, and cover a few sites of multi-center studies, at most. This oversight of has the dual purpose of ensuring the safety and welfare of trial subjects, and ensuring the reliability and integrity of the trials and the data generated from such trials. For trials deemed to involve more than a minimal risk to subjects, regulatory filings are required prior to trial initiation (21 CFR 312 and 812). Sponsors, clinical investigators, IRBs, and FDA share the job of human subject protection for these trials. In particular, FDA scientific and medical reviewers evaluate subject safety (both trial design and supporting data), and, for hypothesis-based trials, the adequacy of the design. FDA investigators rarely inspect ongoing trials except for cause. Sponsors, IRBs, clinical investigators and other parties, for example, contract research organizations (CROs) must comply with FDA regulations pertaining to trial conduct (21 CFR 50, 56, 312, and 812). In some cases, there is applicable guidance some of which were last substantially revised in 1987.²

Since then, clinical trials have evolved dramatically. In particular, clinical trials are no longer primarily conducted at a single center. More often, they are not only multi-centered but multinational. In addition, the use of electronic record-keeping in the studies has increased dramatically and will continue to do so; and trial conduct has become much more complex, both administratively and scientifically. It is now common for multiple entities to be involved in aspects of trial conduct, for example, site management organizations, subcontractors who perform various clinical tests or procedures, and CROs. In addition, the scientific complexity of trials—the number of procedures performed, the number of observations, and design complexity—have increased significantly. These changes are creating challenges for the clinical research community, the industry, and the FDA.

When research was largely the product of academic physician investigators, it was generally assumed that their behavior would reflect both their responsibilities to patients and high scientific standards. As research became less the province of individual investigators and moved toward multi-center efforts, which increased the distance between those who initiated the study and those who carried it out, this assumption became less supportable. Also, most modern investigators have more of a commercial and financial stake than before. This is perhaps most obvious in commercially sponsored studies but seems also true of most government-funded studies, especially when investigators are not the protocol developers. In these settings it seems apparent that intensive training on general responsibilities and the specifics of the study are critical.

A critical destruction affecting the quality of the data is random errors (sloppiness), which are not directed or introduced for a purpose, and bias the data. As a general matter, sloppiness obscures differences between treatments and tends to undermine a showing of effectiveness or, more generally, of a difference between treatments. It is, therefore, very much in the interest of a sponsor to minimize sloppiness, unless the study is a non-inferiority trial, where sloppiness can sometimes support a favorable finding. In contrast, biased data, or fraudulent data, is a major concern of regulators, as it is designed to lead to an incorrect conclusion.

Clinical trials are needed to answer an increasing number of questions about the benefits and risks of therapies, about the best intervention to use in a given medical condition, and about the best choice of therapy in a particular individual. The increasing costs and difficulties of conducting clinical trials limit the number of these questions that can be answered, thus maintaining a great deal of uncertainty in medical practice. It is highly desirable that clinical trials be conducted in the most efficient and effective manner possible while continuing to achieve ensure subject protection and trial quality.

In response to these issues, FDA has undertaken the Human Subject Protection/Bioresearch Monitoring (HSP/BIMO) Initiative. As part of this effort, we are undertaking a comprehensive evaluation of the policy and regulatory framework informing FDA’s oversight of clinical trials. We intend to modernize the regulation of clinical trials in a manner that enhances the efficiency of clinical trials while maintaining high standards of human subject protection, trial quality, and data integrity. The HSP/BIMO initiative contributes to the goals of FDA’s effort under the Critical Path Initiative (http://www.fda.gov/oc/initiatives/criticalpath/) to modernize the scientific process used in medical product development.

Over the last twenty years, important experience with achieving high quality and efficiency has been gained, first in the industrial sector and more recently in other settings including healthcare. The Malcolm Baldrige National Quality Award was established in 1987 to reward quality awareness in business, health care, education, and nonprofit sectors and was inspired by the ideas of Total Quality Management. The first award for health care was given in 2002 to SSM Health Care in St. Louis, Missouri. Since then, the healthcare award has been given every year to those health care organizations demonstrating the integration of quality throughout the delivery of healthcare to patients, setting a new paradigm. It is the purpose of this workshop to set up such a paradigm in clinical investigations. This experience recognizes that quality is a systems property that quality must be "built in" to an enterprise and cannot be achieved by testing or oversight alone. FDA has relied on these principles in its "Pharmaceutical Quality for the Twenty-first Century" Initiative, which has changed the way FDA approaches the regulation of pharmaceutical manufacturing. In September 2006, FDA issued guidance for industry entitled "Quality Systems Approach to Pharmaceutical Current Good Manufacturing Practice Regulations." This document provides FDA’s thinking on how manufacturers can implement quality systems and risk management approaches while continuing to comply with FDA’s cGMP regulations. FDA believes that similar principles of modernization can be used in the clinical trial arena.

Discussion Points

Data Quality Challenge

Over time there has been an escalation in the amount and type of data collected during a clinical trial, and a marked increase in the complexity and diversity of the databases in which this information is stored. However, the ability to ensure the quality of the collected data that supports the scientific goals has created an increasing burden on existing methods and has not kept pace. Specifically, the retrospective detection and elimination of errors through the reconciliation of case report forms with source data, comparison of case report forms with protocol specifications, and the resemblance of clinical database entries to case report forms has become an increasingly resource intensive process for assuring data quality. Ultimately the primary reason for assuring data quality is the assurance of reliable data upon which to base clinical decisions of safety and efficacy.

In addition, the current system has evolved from a single-site/single-investigator profile to a globalized and highly decentralized venture, straining an already antiquated system of data storage, monitoring and auditing. Consequently, there has been a relative diminution in the contribution of a single site to the overall trial. Furthermore, new arrangements by sponsors in the implementation of the trial have resulted in the delegation of some of the clinical trial responsibilities to entities not directly subject to FDA regulation for example subcontractors, sub-investigators, study coordinators, testing facilities, and data managers.

Clinical trial designs have become increasingly more sophisticated and reflect the study of complex biological, device, and combination products. The intricacies of modern clinical trials extend to the inclusion of vulnerable populations (e.g., children) which may warrant sub-stratification and analyses of results, as well as super-imposed additional human subject protections on the overall trial design, one of which is increased transparency. Much discussion has occurred regarding data-sharing in the pediatric arena as well as the setting of rare diseases. Data-sharing would require assurance of data-integrity such that quality medical care could stem from research activities.

Many factors, including the misinterpretation of current federal regulations, have resulted in costly and inefficient validation procedures with little certainty that human subject protections are enhanced as a result. The impact of these methods to assure quality data and human subject protection is uncertain. To rectify this uncertainty, a modern research enterprise demands a refined and creative approach that:

Establishes a definition of data quality,
Ensures that the design of protocols is linked to the handling of data (data management, data clean-up, data standards, data monitoring and auditing) implements quality, and
Establishes metrics for measuring quality.

Useful Attributes of Quality in Clinical Research

The following descriptions are being posed to better understand quality. This is not an all-inclusive list; it is meant to stimulate discussions about the definitions of quality.

DATA QUALITY

High-quality data - The IOM Report entitled "Assuring Data Quality and Validity in Clinical Trials for Regulatory Decision Making," 1999: High-quality data may be defined as data strong enough to support conclusions and interpretations equivalent to those derived from error-free data."
Fitness for use - Regulatory decisions regarding safety and efficacy would be predicated on reliable data such that they could support marketing applications, and would ultimately provide high-quality products for clinical use by patients and prescribers.
Pragmatic - Set limits on variability by determining acceptable levels of variation (requires some scientific basis for determining "acceptability"). By virtue of the inherent nature of data, it is understandable that some level of variability has to be accepted.

TRIAL AND DATA QUALITY

Risk-management approach – Certain characteristics of a clinical trial may trigger additional scrutiny, e.g., no preceding experience with a country or sponsor, a sponsor-investigator (individual investigator) clinical trial, or a clinical trial with a vulnerable subject population.

the probability that "x" level of variation could affect sensitivity of the analysis and the conclusions derived from it, and
the setting of specifications and process parameters as well as assessments to mitigate the risk of a change to a process or specification (see Guidance for Industry: Quality Systems Approach to Pharmaceutical CGMP Regulations, p5).

Characteristics to satisfy a need - Applied Clinical Trials/International Organization for Standardization 9000: the total set of characteristics of a product or service that affect its ability to satisfy a customer’s stated or implied needs (customers meaning the society, research subjects, sponsors, regulatory authorities, hospitals, institutions, IRB/REC). Translated into the setting of clinical research this would suggest reliability and credibility of information coupled with compliance of the trial process with specified requirements.
Safe, effective, patient-centered, timely, efficient, and equitable - AHRQ contracted with the IOM to assist in the design of the new national health care quality report. The IOM Committee on the National Quality Report on Health Care Delivery was established in 1999 and was charged with laying out a vision of the National Health Care Quality to include identifying the major aspects of quality that should be reflected, and providing specific measures of quality. It defined health care quality as "the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge." It operationalized this definition by ascribing six attributes of a quality health care system: safe, effective, patient-centered, timely, efficient, and equitable. Selected measures for the individual quality criteria were grouped: 1) the overall importance of the aspects of quality being measured; 2) the scientific importance of the aspects of quality being measured, and; 3) the feasibility of the measures. Although the National Quality Report focuses on the health care delivery system with regard to personal health care, the Agency should evaluate whether these attributes and their measurements can be adapted to the clinical research setting.

Possible Steps towards Better Quality in the Clinical Research Enterprise

Adopt uniform definitions of quality, i.e., all research constituencies commit to a national, and possibly international, statement of quality compatible with a modernized research system as a whole and with a predisposition to measurable outcomes, reflecting the best current approaches to research
- Identify aspects of the system of clinical research that impede quality and/or neither promote quality nor human subject protections

Develop methods and procedures that build in quality in protocol design and real-time process monitoring rather than relying solely on retrospective inspections of protocol compliance.

Design of protocol and investigator’s brochure with clear definitions of entry and diagnostic criteria and methodology
During protocol design phase, give careful planning to the types of data warranting collection. Collect only data essential for interpretation of trial results, and consider implementing risk-based approach to data collection, analysis, and monitoring. Included in this should be a discussion of how the study design impacts the degree of monitoring needed and endpoints warranting active monitoring and review. For example, large simple trials with mortality endpoints may require less monitoring and auditing than a randomized clinical trial with surrogate endpoints or subjective endpoints.
Investigator training development, and consideration of investigator certification
Proactively plan risk assessment into the drug development process. As is discussed in the Agency’s "Guidance to Industry: Pre-marketing Risk Assessment (March 2005),"³ the adequacy of the assessment of risk is a matter of both quantity (ensuring that enough patients are studied) and quality (the appropriateness of the assessments performed, the appropriateness and breadth of the patient populations studied, and how results are analyzed). Good clinical risk assessment in the later stages of drug development should be guided by the results of a comprehensive preclinical safety assessments and a rigorous, thoughtful pharmacology program (including elucidation of metabolic pathways, identification of possible drug-drug interactions, and determination of any effects from hepatic and/or renal impairment), as well as the observed safety profile in phase 1 and early phase 2 trials. Since many safety signals in a development program will be heralded by adverse events that occur between study visits, proactively plan an approach to systematic collection and presentation of "unscheduled" visit data (adverse events, laboratory, vital sign, electrocardiograms, etc.).
Implement a risk-based approach to trial inspections (e.g., site selection process) as well as a modernized approach to inspections such that only data affecting/supporting the safety profile or efficacy claims will be monitored and inspected in real-time
- Consider iterative monitoring processes that enhance reliability without duplication
- Consider, when appropriate for sites and trials based on appropriate risk factors, alternatives to traditional on-site monitoring (e.g., remote or electronic monitoring, statistical process control). Also, consider whether there are unique issues raised by sponsor-investigator, and how should these be addressed
- Standardize data collection (e.g., case report forms, adverse event report forms, electronic medical records, nomenclature). Borrow from work already accomplished in information technologies such as software development, data management, data transfer formats
  - Training
  - Pre-testing of forms and procedures
- Standardize data format (e.g., CDISC) to facilitate data auditing
- Standardize data storage to facilitate real-time trial audits, transparency of data and data sharing
Expand utility of collected data beyond a single trial such that the databases can be used over longer term. Potential sharing promotes collective involvement of research community, and respects the ethical underpinnings of human subject involvement in trials (maximize utility of accumulated data with reduced exposure to research risks)
Educate all stakeholders in this effort so that the research enterprise is approached as a totality (IRB/REC members, clinical investigators, clinical research coordinators, academia, journal editors, compliance and enforcement staff, data managers, inspectors, medical reviewers, HHS and other government agencies, sponsors, hospitals)

Footnotes

¹In 1997, Richard Borison, M.D. and Bruce I. Diamond, Ph.D. pleaded guilty to felony charges. Borison and Diamond were charged with developing and executing a scheme that involved the conduct of over 100 research projects for more than 20 pharmaceutical companies. They systematically stole in excess of $10 million from the Medical College of Georgia by using the school’s resources and pocketing the proceeds; routinely lied to conceal their crimes; and endangered the safety of the patients and study participants.

² http://www.nihs.go.jp/dig/ich/ichindexe.html

³ http://www.fda.gov