1997 Partnerships for Networked Consumer Health Information Conference

Transcripts of Plenary Sessions and Breakout Sessions

Does It Work? 2: "Quality: What Is It? How Do We Know? Who Is Responsible?"

Wednesday, April 16, 1997
9:30- 11:00 AM

Moderator: Helga E. Rippen, M.D., Ph.D., Director, Health Information Technology Institute, Mitretek, McLean, VA

John H. Renner, M.D., President, Consumer Health Information Research Institute, Independence, MO

Susan Alpert, M.D., Ph.D., Director, Office of Device Evaluation, Center for Devices and Radiological Health, Food and Drug Administration, Rockville, MD

Kiyoshi Kuromiya, Founder, Critical Path AIDS Project, Philadelphia, PA

George D. Lundberg, M.D., Editor, Journal of the American Medical Association

Rippen: This session addresses questions regarding a networked information system, developed through partnerships, such as does it work? How do we know? Who is responsible? The questions are addressed in a formal presentation and through audience participation.

Several standards for quality will be introduced. They include general standards for health information [these will appear in the Journal of the American Medical Association (JAMA)]; a set of criteria to evaluate site quality; and Food and Drug Administration (FDA) input on design and implementation of Internet standards.

How many present here today provide information on the Internet? That's some. How many use information on the Internet? That certainly is a lot. Well, we have a wonderful panel in place to discuss the issues of quality from a variety of perspectives.

What I'd like to do is set the stage and state that the Internet provides a wealth of information. Some of us have very deep concerns about the Internet. Quality is a real issue. In other words (pointing to a slide of a great white shark) the shark bites and we need to determine if we want to control it before it bites us. Should we be teaching people about usage of the Internet and about certification of sites? Do we want to go further into censorship? So again, there are a lot of great issues to discuss.

These panelists will discuss a variety of topics regarding delivery of quality information over the Internet. John Renner will discuss the necessary criteria to be used in evaluating consumer information on the Internet. Susan Alpert will discuss the pertinent issues faced by the regulatory agents as well as their role in quality assessment. Kiyoshi Kuromiya will examine patients’ and providers’ concerns as they investigate the numerous resources available on the Internet. George Lundberg will discuss how to best implement mechanisms on the Internet as well as issues of concern for the public and practitioners.

Lundberg: I consider this a privilege to kick off this session.

How do you know who's responsible? Work, Finish, Publish. These three key players are depended on to act with responsibility, and they are in the reach of the editor. On the web, a person can be an editor, a writer, or a consumer peer reviewer, which is someone who submits one's work in a voluntary fashion to be screened and analyzed by others with similar expertise.

An editor asks someone, "Is it new? Is it true?" And he then asks, "So what?" and "Who cares?" These questions are pertinent in determining if the material is published in a medical journal. For guidelines for acceptance of materials for publication, see the International Committee of Medical Journal Editors.

I suggest that voluntary standards be applied on the Internet. One of my favorite quotes comes from a hall at Florida State University. "Half of knowledge is knowing where to find knowledge." This appears on Dodd Hall; the author's name has been removed.

On the Internet there are a variety of places where one can search for knowledge: online services and CD-ROMs -- which never really made it. They are not being utilized that much even though they are a good medium. There are journals, but you have to mail them and use up space and resources to produce them.

The Internet has several advantages. As a system with worldwide connectivity, it is easily accessed, and it is voluminous in capacity. Our experience on the Internet at JAMA has been somewhat recent, but we are pursuing new avenues to present our data to the public. At the onset of the installation of our web site, the Presidential Board set up a strategic plan, and the usage has been wonderful. To view our site, see www.ama-assn.org. We couldn't use "ama" because it was being used by the American Manufacturers, and we couldn't use JAMA because of a Japanese site.

At our last reading, we were receiving enormous amounts of hits per month, and they have been climbing to an average of 400,000 per week. There is regular solid information available, but there will be new journals available from JAMA that will only be accessible on the web and not distributed in print.

Standard expectations for a quality product are that it posses totality of characteristics, features, and services. It must also have the ability to satisfy the needs of the recipient. Yet quality, in essence, is in the eye of the beholder.

I would like to say a few words about assessing, controlling, and assuring the quality of medical information on the Internet. How do you decide what to use on the Net? [Mr. Lundberg hands out a copy of a JAMA Internet journal page entitled, "Assessing, Controlling, and Assuring the Quality of Medical Information on the Internet: Caveat Lector et Viewor--Let the Reader and Viewer Beware," an editorial by Bill Silberg, (e-mail: Bill_Silberg@ama-assn.org) and by Dr. Musacchio, (e-mail: Robert_Musacchio@ama-assn.org) See JAMA, April 16, 1997--Vol. 277, No. 15].

In traditional editorial forums, the rules of publishing have had standards in place for over 5 centuries. There should be concern as well as excitement for the medium of the Internet as it becomes more and more used to bring medical information to patients and medical practitioners. Our concerns at JAMA are the possibility of dissemination of inaccurate, misleading, or incomplete information. The Internet is a vast source of communication, but it can be abused. Consumers need to be cautious about what they accept as fact on the ever-expanding number of web sites dedicated to the dissemination of medical advice, diagnosis, and treatments. The issue of quality for the communication of medical information becomes extremely important.

Basic standards of quality must be adhered to for credibility of the medical information.

(Outline follows)

1. Authorship: We must determine who are the authors, the contributors, and with whom are their affiliations. Do they have relevant credentials, and should these individuals or groups be providers of medical information? The pertinent question to be asked is whether these individuals are real authors, guest authors, or ghost authors.
2. Attributions: Should be listed clearly, and all relevant sources should be available for review.
3. Disclosures: It is important to show who bears the ownership or sponsorship or who provides advertising for the sites. From where does their funding support originate? Other issues to consider are whether there are conflicts of interest among information providers and those who provide financial backing for those sites. If you get a grant to put up stuff, make it clear if there is an advertisement attached. If it is an editorial, that should be stated. The location to which you are sending the information should also be as credible as the location from which it originally came. You must be able to warrant that author's standard.
4. Currency: Dates must be included so that the public knows when the content was posted. Updates should also be indicated.

It is my hope that a set of core standards can be applied without imposition from the Government. I don't believe that regulation is necessary nor that punitive actions should be taken against those who do not abide by the voluntary standards. I don't want to display arrogance on behalf of JAMA or the standards we uphold. However, in the proper forum we want a quality assessment of credentials expected of those putting medical information on the web. It is a medium that provides so much information, and much of it can be misleading -- we want people to be able to distinguish the flowers from the weeds.

Renner: How many of you are on the Internet one hour a day? Two hours? Three hours? Four hours? All day? Does anyone present here today have what they dubbed on the news last night as Internet Fatigue Syndrome? How many of you would believe anything in the National Inquirer? How many would believe what is in Prevention Magazine?

I'm on the board of Prevention Magazine, so I guess I had better disclose my bias from the start. No, actually, they have come a long way in the type and substance of the medical information they provide.

I learned 30 years ago patients behave differently when they are in the office by themselves than when they are with others.

Be careful not to stereotype the Internet with one set of evaluation and criteria -- the components of Internet are varied. We must be very specific when trying to understand each of these parts. Who's on a LISTSERV will determine if you believe in the source -- even on a physician LISTSERV.

Five Criteria Areas:

1. Technical
2. Content
3. Credibility
4. Usefulness
5. Linkage

Each criteria has 12 subparts, which I cannot cover in the time allowed, but I will mention a few items. For instance, linkages with 23 pages of disclosures may provide validation to integrity and accuracy.

We must be careful when looking at credentials -- people have phony Ph.D.s. We have physicians who have lost their practices and still give out advice on the Internet. Credibility: for a few years I ran an FDA database that collected information on rascals on the Internet. There are scalawags on the Internet, and there are web site developers selling web site expertise to them.

We have web sites for people who have been accused of federal crimes. Turn up your crap detectors. There is a lot of hostility on the Internet. I am concerned about how to develop relationships with people who are distrustful, with people who do not trust consumer-based groups. But when you don't know someone, it is going to be difficult to develop those relationships.

The volume of data on the Internet continues to grow. How can you control the quality of new sites? Many sites have irrelevant information on the Internet. Many hits are fluff -- some editors are beginning to think that there are no good sites. Look up osteoporosis and a psychic healer site will come up because people have learned how to put the keywords in.

I brought some examples of available products: something for Downs syndrome; extracts from human autopsies in zoos by a dead doctor. How many of you have ever heard of someone doing autopsies on humans in zoos? And now there is someone on a Canadian AIDS site who is trying to sell an AIDS cure over the Internet. It is absurd. I respond to many of these ridiculous sites just to see the crap these charlatans are trying to sell to the public.

I really think we need people who are willing to raise red flags, and we need voluntary participation to minimize "weird web sites." Thank you.

Kuromiya: Social service and health information are essential needs. I will present what the consumer needs. I have AIDS; I am a content provider for persons with AIDS; and I provide sites (476,000 hits per month) for AIDS-related medical information. I am also an Internet provider.

What are people with AIDS looking for on the Internet? At first it was information in the English language. In the early 90’s, the therapeutic guidelines were issued. Sixteen years into the AIDS epidemic, there has been an expansion of information services.

Most of the social care service came from the family or community. The quality of services may have been frowned on by governmental organizations or other medical groups, but it served those who had been traditionally underserved -- the incarcerated, minorities, and of course, the gay community.

Advocacy groups have made a difference. Now AIDS patients sit on community council boards and on AIDS research panels -- it is a matter of life or death. We could not wait for the family practitioner or for the Government. The U.S. Government had not yet issued standards of care for AIDS-related illness in the late 1980’s. We discovered that, by the time medical research material was published, much of the information was obsolete.

In 1989, we made our first venture into cyberspace as a referral source. Increasingly more AIDS patients are surfing the Net to communicate, regardless of where they live. We can gain access to information that the Government didn't make available: new drugs, protease inhibitors, and experimental techniques. In 1995, "Critical Path" issued new items on the Net on a daily basis.

Ninety percent of all people with AIDS are from the underserved populations. Critical Path decided to offer free training, in addition to information, about the Internet. In fifty three branches of Philadelphia public libraries, there will be Internet access sites installed.

Six hundred and sixty individuals have been given Internet access, and now, two, years later, Critical Path hosts 40 servers and has made links to other health-related sites. We have an AIDS e-mail list that links people in developing countries where the epidemic is most severe.

It costs $2.16 to send out a single copy of our newsletter, but it's only 7 cents for each hit we get on our system. Our entire annual budget for all our services (newsletter, website, 24-hour hotline, advocacy work, and free dial-up access to the Internet) is less than $150,000. Annotation and moderation are used to create standards for material on web sites as we try to gain medical validation. We have tried to block inappropriate forums.

We try to reach younger people. The Internet will reach 50 percent of U.S. households by the 10th year from its inception, whereas the telephone took 50 years to get into 50 percent of American households. The Internet is expanding rapidly.

Alpert: As a regulator, quality is the issue to be focused on in communications environments where the goal is to reach people. The FDA is not trying to prevent dissemination.

How does the FDA determine what is quality? Labeling, marketing, and promotion are issues that this agency is charged to investigate by Congress. I am not making policy, and I am not proposing policy. The FDA is still evaluating information, and we are not making hasty decisions. That's my claim and disclaimer. I have different perspectives. Yes, I am a regulator, but I am also a consumer.

There is a T-shirt for sale at this conference showing a patient trying to do his own diagnostics, sitting in front of a monitor. By the way, I bought that shirt, and it is very telling about how widespread the use of the Internet is for medical information.

I will discuss how the FDA assesses quality. First of all, I would like to state that we are skilled in the new technology, and we have our own web site at www.fda.gov. The following factors should be addressed:

Recognizing quality at FDA:

1. What is it?

• Accuracy in communication

2. Who is responsible?

• Generation of information
• Confirmatory/endorsement role -- does it reflect what we've seen in data

3. How do we know? We review all information and make judgments on the quality of that information

4. Labeling according to the law

• Misbranding & adulteration
• We have a role in overseeing communication between manufacturer and consumer -- there are professional consumers and nonprofessional medical consumers. We can assume a lot about information given to the professional consumer. For the nonprofessional, however, we need to assist by providing more information.
• Labeling is the summation of all the data. It shows how to operate the product, measures the benefits, and provides a history of how the data were generated.
• The FDA has highlighted misbranding and adulteration because we only regulate the person who puts the product on the market. Those companies or individuals are responsible for providing accurate product information. If the claims of the manufacturer do not match the performance of the product, the FDA has a legal obligation to investigate and impose restrictions on the manufacturer.

5. Roles and types of publications

• Referees -- we regulate the producer of the products. Not all journals follow the established rules and guidelines. There are numerous publications that are throwaways. There is no regulation for the authors, data, or other information within the journals.

6. What information?

• Who is responsible for putting the information on the web sites? Many items on the Net are not legal because the manufacturers have not brought the product before the board. Yet the information is still available, and there is little regulation. Accuracy, and how informative the sites are, are issues the FDA looks at.

Other important questions include:

• Who reports the information -- background of authors
• Who evaluates for accuracy -- who's gone behind the information and who endorsed it?
• Who receives the information? -- different population groups, and how you communicate to them affect the usefulness of that information.

The consumer needs to be better educated. Patients need full disclosure in order to make informed decisions. For instance, a patient considering high-risk, nonreversible retinal surgery could benefit from patient information brochures addressing this form of intervention.

Rippen: As you can see, there are reoccurring themes related to the issues. Now I'd like to open the floor for questions for the panel.

Question: This is for John and the panel. If the Internet is a global, international system, it reaches the whole world, and the medical health system is confronted with the debate about "alternative medicines." Eighty percent of patients worldwide get much of their health care and advice from those systems. Thus, shouldn't issues pertaining to these various forms of medicine be included in the debate about quality?

Renner: A large portion of the American population believes in aliens and UFOs, but just because this concept is marketed does not mean that it is appropriate to allow accessibility to those projects. You cannot tell someone to practice Mayo, Chinese, and this type and that type of medicine. We need to help people prioritize. "Chubushu" tea has its place, but then again so does Coke.

"Alternative medicine" is a political term, not a medical term. The definitions of alternative medicine have varied. It used to be that if a particular practice wasn't taught at the best medical schools in the country then it was not a proper medical procedure. Now, you see many schools (even Yale) are including alternative medicine practices in their curriculum.

However, I challenge you about the sale of chloroform tablets to the American public. I think that is unwise and unsafe to make this product available to the public.

Question: With the need for urgency, I share that concern about charlatans. How can we assist the citizen in making the right decisions?

Alpert: We have put in place regulation and information that states what the properties of the products are, as well as information about the manufacturer. The validation of good medicine is not unique; we ought to expect the same quality assurance procedures in reviewing all the information consumers look at. We as an agency are not trying to prevent information from being shared. But we want it balanced, complete, and accurate.

Question: How can people be in control of their own assessments about new technology or medicine so that the Government is not the sole regulator?

Alpert: We have in place many panels, and we are involving consumers; we are having open public meetings to allow people to come and speak about issues related to regulation.

Renner: I would just like to add that the AIDS message is very important. I was picketed by ACT-UP, and there was a time when we all were a bit uncomfortable with the AIDS activists' stance and their methods of getting the most up-to-date information about possible treatments to their constituents. And I just feel that it is important to address patients’ needs and new arenas on the Internet for AIDS groups to share medical advice or new drug discoveries.

Rippen: There are a great number of resources, and we at the Health Information Technology Institute are in the process of putting together a white paper that will be published soon, outlining the resources.

Question: Who rates the raters? Consumers look to brands as shortcuts to get the substance of information. Where is the evidence of rating sites going?

Lundberg: People can rate sites by the criteria outlined in the article I handed out. These criteria are based on the standard concepts found in Journalism 101. JAMA would be interested in working with other organizations that make good criteria that makes sense for journalism and for medicine. We have a software review column, journal evaluation review criteria, and we would agree to publish other reviews if all interested parties had agreed to the standards of publication. Donald Trump used to be fond of saying, "Don't give people what they need; give them what they want." JAMA can provide the venue of validation for review and for evaluators.

Rippen: An important issue -- Certifiers cannot be biased and must be able to have a broad base. They must show with whom they are affiliated.

Alpert: You can come up with guidelines of self assessment and a means of certification to meet voluntary standards. Then the web masters can choose if they wish to follow those standards.

Audience: Might I suggest a new book called Medical Safari? This book provides guidelines on evaluating medical sites on the web. There is a ranking system offered.

Renner: Yes, it is true. Many of the award givers are lacking for a category on accuracy.

Lundberg: They often give awards for form, not content.

Rippen: We hope you enjoyed the forum. We are certainly glad to have had you here today. Thank you.

Back to Summaries and Transcripts page.