Evaluating Our Web Presence: Challenges, Metrics, Results
A CENDI Sponsored Symposium Co-Sponsored by the National Library of Medicine, Tuesday, April 17, 2001
Presentation Abstracts In Order of Presentation
Fred B. Wood, "A Strategic Approach to Web Evaluation" (PowerPoint File, 860KB)
The World Wide Web offers many new opportunities for the National Library of Medicine and other government agencies to more effectively carry out statutory missions. But the Web has made it much more difficult to determine if we are succeeding in our efforts. The web-based information and user environment is more complex, and traditional evaluation methods do not work or work as well. NLM and CENDI have recognized the need for a more comprehensive, innovative, and multi-dimensional strategy for web evaluation. This includes the integrated use of web log usage data, usability studies, online surveys, and online user panels, combined with traditional methods such as telephone surveys, outreach feedback sessions, and focus groups. We also include Internet connectivity performance evaluation (e.g., web download times) in our overall framework. And we are emphasizing, to the extent possible, the use and impact of web-based information from the user or customer perspective. This includes new efforts to understand why users are seeking information, how easily users are able to access and download information, and what users are doing with the information they find on the web. Over the last year, NLM and CENDI have collaborated to inventory our use of web evaluation methods, and share experiences. NLM has experimented with a range of web metrics and evaluation methods. We believe that a strategic approach to web evaluation is emerging. We hope that this approach will advance the knowledge base in this field, and prove useful to the many government agencies and others who are faced with the challenge of more rigorously evaluating web-based information services. Previous Page
Kevin Mabley, "Online Surveys for Web Evaluation: Research Methods for Understanding Your Audience" (PowerPoint File, 1,078KB)
The World Wide Web provides a new vehicle for companies and organizations to reach their audiences. Understanding a site’s visitors is essential to provide a valuable experience for users and to quantify customer value. Measuring this value requires using data collection and analysis techniques in order to turn anonymous hits and clicks into actual customers and to answer critical marketing questions. Whether marketing objectives involve outreach initiatives targeted to specific segments of the public, or building a profitable customer base, data analytics provides a solid basis for decision-making. This presentation will demonstrate quantitative and advanced analytic research methods used to aid organizations better understand their role in the online space. Previous Page
Kevin Garton, "Collecting and Analyzing Web Usage Data from User Panels" (PowerPoint File, 1,016KB)
This presentation will provide an overview of web-based research panels, including panel recruiting methodologies, weighting and projection techniques as well as data collection and analysis procedures. An industry overview will be provided, highlighting the pros and cons of various methods. Specific data examples will be discussed, including the use of panel data in evaluating web site effectiveness, marketing programs and strategic planning. Finally, the presentation will highlight uses of panel data in both the public and private sectors. Previous Page
Bill Silberg, "Web Evaluation from a Corporate Perspective" (PowerPoint File, 304KB)
The World Wide Web has been characterized as a revolutionary technology that promises to eventually transform numerous fields of intellectual, professional and commercial endeavor, including medicine. Experimentation with the Web as a clinical tool, particularly in the areas of decision support, information dissemination, education, and publishing, has been under way since the mid-1990s. Initial efforts within academic and other not-for-profit settings were quickly augmented by commercial "e-health" initiatives, many of them supported by venture capital and the seemingly boundless enthusiasm of the public markets. The resulting "dot-com boom" of the late 1990s, however, quickly gave way to a "dot-com bust" that continues today. The result has been significant market consolidation within the commercial "e-health" arena as well as a new sense of sobriety within the not-for-profit community as to the true opportunities, costs, and challenges of an "e-strategy."
As part of this natural evolution, many commercial "e-health" companies -- at least those still in business -- are reviewing and reevaluating their business models and corporate strategies. The focus is on the public markets' clear message that "old-economy" standards of measurability, profitability, sound business models, and sustainable growth, all still apply. Within this environment, the issue of metrics -- measures of performance and, thus, success or failure -- has achieved particular prominence. Just a few years ago, within the "e-health" arena as well as more broadly, gross metrics were the rule on the Web. Reliability, validation and the true value of such metrics in describing visitor behavior and needs were often ignored or dealt with in an inexact or cursory fashion. There were any number of reasons for this, ranging from the nascent nature of the technology to the fact that commercial interests weren't very aggressive at first in asking for them. Those days, however, are over.
Metrics and their importance within the e-health arena are now being viewed critically from two key perspectives:
- Internal, as "e-health" companies seek to better understand still ill-defined user and market behavior with an eye toward improving efficiency, responsiveness, value, utility and commercial viability;
- External, as investors, advertisers and other commercial supporters seek to maximize return on investment within their own "e-strategies" and look for accepted measures of effectiveness and performance comparable to those expected of traditional media.
There is growing consensus as to the kinds of metrics that health and medical Web sites find most useful and valuable for strategic and operational business purposes. Standardization of such metrics, whether gathered internally or through third parties, remains somewhat elusive, however. And some of the measures that commercial entities would find most compelling must be viewed within the context of legal and ethical concerns that, if not peculiar to the use of the Web in a medical context, have been amplified by this technology's enormous clinical promise. Previous Page
Carlynn Thompson, "Web Sites That Work: The Role of Evaluation at DTIC" (PowerPoint File, 1,990KB)
The Defense Technical Information Center has been providing Information Technology support to the Office of the Secretary of Defense components since 1994 to include development, implementation, and maintenance of public access Web sites. DTIC’s experience in publishing Defense information on the Web has created a center of excellence for meeting the complex information delivery needs of DoD organizations which are required to present a wide range of information to a variety of audiences in the most rapid, effective, and secure manner possible.
Over 400,000 worldwide users visit DTIC sites 8 million times a week, delivering 458,000 megabytes of data. Government Executive Magazine has recognized several DTIC designed sites as a "Best Feds on the Web. " The criteria for this recognition were to: provide excellent customer service to the public by having a well-designed site that includes a large amount of useful information; use the Web to improve business practices in their agencies or across government; and make use of new technologies that other federal sites should consider emulating.
The DTIC staff has been directly involved in developing administrative policies and procedures including a disaster plan, content publishing guidelines and software development guidelines. These policies have been used to educate content providers on the critical elements of content delivery.
Key to the success of Web Services at DTIC has been objective assessment of Web sites and networks. Areas that are evaluated include: network response times, network bandwidth, systems availability, site design, usability, direct customer feedback, web server statistics, privacy of users, law and policy, and security.
We have learned many lessons over the past six years in providing public access web services. As a web service provider, there are few clear-cut rules that everyone should follow. Each environment is different. It is very important to balance services provided with sound security measures and conscientious site evaluation. Previous Page
Lisa Zolly, "The Role of Usability Testing in Web Site Design: The National Biological Information Infrastructure's Experience" (PowerPoint File, 1,870KB)
In January 2000, the National Biological Information Infrastructure (NBII) Program, managed by the USGS Biological Informatics Office, initiated a five-month deconstruction and redesign of its Web site. The centerpiece of the Program, the site provides a gateway to biological and natural resources information maintained by federal, state, international, and non-profit agencies; academic institutions; and private industry. A distributed system maintained by multiple developers, and encompassing diverse formats, site was re-engineered to encompass expanding thematic and regional content, resolve fundamental design flaws, and provide easier access to the data and information. Using heuristic analysis, the NBII design team implemented changes to the architecture and interface of the entire site. To examine the effectiveness of these elemental changes for actual users of the NBII Web site, the design team initiated usability testing in October 2000, with the assistance of a professional usability consultant. Test results confirmed the overall soundness of the site redesign, and provided concrete data to improve and enhance the user’s experience in locating needed natural resources information. Previous Page
Susan Elster, "Consumer Health Information in Allegheny County, Pittsburgh, PA: An Environmental Scan" (PowerPoint File, 164KB)
Recent advances in consumer health information technologies have the potential to significantly change the kinds of health information to which consumers have access, making it possible for them to participate in decision making about their own health care in new ways. Because information technologies have the potential to play such a critical role in improving consumer access to better health information, the Jewish Healthcare Foundation (JHF) sought to learn to what extent information technologies are being used in Allegheny County (Pittsburgh, PA). JHF commissioned an environmental scan from the University of Pittsburgh’s Graduate School of Public and International Affairs to identify what precipitates a consumer’s search for health-related information, what barriers are encountered, and what role new information technologies play in providing information. The study relies on information collected from focus groups, telephone and written surveys with more than 1,000 Allegheny County residents.
The study documented the following major findings:
- Health Care Providers (HCP) play a central role in providing and evaluating health information for most of those surveyed (regardless of health status); however, a quarter of those surveyed also looked for additional sources of information – primarily from the Internet. The relative youth of these "New Information Seekers" indicate that they will make up an increasing proportion of health care consumers, challenging both existing information resources and traditional HCP-patient relationships to meet their needs.
- Although consumers reported consulting a wide variety of sources beyond their HCP for information, none were ranked as "very helpful" by even a third of respondents.
- HCPs and consumers confront a number of communication challenges: Those managing chronic illnesses are less likely to report satisfaction with their HCP than others. In addition, while consumers seeking additional information are most likely to get that information from the Internet, AND would be most likely to trust the information on an Internet site recommended to them by their HCP, HCPs overwhelmingly distribute print material and brochures to their patients. Respondents reported that fewer than 4% of HCPs recommended a web site to them.
- Beyond Internet searching, there is limited use of health information technologies by HCPs or health care consumers.
- African Americans, those with lower income and education, and those over 65 have lower Internet access. The public library, however, appears to provide a significant point of access for such households.
