From Primary Sources to Anecdotes: Making Sense of Treatment Information

Spring 2004

This article is part of The Body PRO's archive. Because it contains information that may no longer be accurate, this article should only be considered a historical document.

Staying "informed" about treatment options -- whether for your own health decisions or to help clients and friends with theirs -- can be overwhelming. I don't know whether it's more frustrating to find that there's no information on a treatment you're interested in or to find that there's so much information that you don't know where to start -- let alone how to sort out the good stuff from the fluff. Is the information really not there or are you not looking in the right places? What if there's a ton of information but it seems contradictory and messy?

I am trained in epidemiology and in social science research and have spent a lot of time teaching college and graduate students how to think about finding and evaluating information. I also had cancer a few years ago and was unexpectedly thrown into a situation in which I needed to make complicated decisions that involved a whole area of health and treatment I knew nothing about -- and I needed to move quickly. In this article, I suggest some strategies for locating reliable information that I have found useful both in teaching and in making my own treatment decisions. I also try to untangle some of the terminology about information sources that always seem to pop up but aren't usually defined -- terms like "primary sources" and "anecdotal evidence."

Journals, Books, Newsletters, Web Sites -- What Difference Does It Make?

There are obvious differences between a journal and a book, but there are some important "hidden" differences, too. Even a lot of my college students aren't sure about the difference between a journal and a magazine. Each of these sources has a different process that authors have to go through in order to get published, and that has an effect on how reliable the information is.

Start with journals. Usually, a "journal" refers to a publication that goes through a process of "peer-review." Peer-reviewed journals are the gold standard for reporting on scientific research. Publishing an article in a peer-reviewed journal is much more difficult than publishing in other sources (a magazine, newsletter, Web site, or even a book). That is because the article must go through a process of systematic review by other scientific experts.

The process looks like this: a scientist (or usually a team of scientists) writes up research results and sends them to a journal where they'd like to publish. The editor of the journal takes the name of the scientist(s) off the study and sends it to two or three experts in the particular field of research for detailed comments and criticism. The main job of the scientific reviewers is to make sure that the article is clear, gives enough details about how the study and analysis were done, and takes into account other relevant research in the field. It is quite unusual for a peer-reviewed article to be published exactly as it was originally written. If the results of one study contradict other research that has been done or are very surprising given current scientific ideas, then the review will be particularly important. Reviewers should make sure that the article gives a reasonable scientific rationale for how to make sense of this new study in light of other research.

The process is a very conservative one, which is troubling in some ways. On the one hand, it's hard to get results out quickly -- especially very new or surprising results -- and it's hard to use lively or enthusiastic language, which makes these journals pretty boring to read. On the other hand, when it works as it should, the peer review process pushes scientists to explain how new work fits in with existing studies and also makes sure that scientists can give clear and plausible explanations of how they reached their results. This helps to minimize scientific fraud (faking results to get grants, credit, or Nobel Prizes). Since the review is done anonymously, by multiple independent experts, this means that information presented in a journal has been held to a higher standard than information from any other source, so it is considered the most reliable. That doesn't necessarily mean that it is the most useful information, though.

Magazines, newsletters, books, and Web sites have editors -- some very knowledgeable and strict -- who might do a lot of the same things as peer reviewers. Some of these sources also use "fact checkers" whose job it is to check that the information is accurate. While all peer-reviewed journals have a pretty similar process, there is endless variety in how these other sources decide what to publish. This is called the "editorial policy" and it is completely up to the person or organization who publishes the material. Which brings me to another major consideration -- who is the publisher?

There's a huge difference between a Web site or newsletter published by an individual and one published by an organization that has a formal process for review and standards for what kind of information to include. Before you even start reading any Web site you visit or print material you pick up, ask, "Who published this?" (Another thing to pay attention to is when it was published; obviously you want the most current information you can find, all other things being equal.) Is the publisher an organization you recognize? Many medical associations, universities, trusted non-profit organizations, and government programs (such as institutes within the National Institutes of Health) run very reliable Web sites. If you aren't familiar with the organization that published something, is there information about the organization's mission and the editorial process anywhere in the publication or on the Web site? Do they list the original sources for any claims that are made about how a treatment works or for information that is presented as "fact" (whether about how a treatment has been tested, how it works, how the body works, disease process, or anything else)? If you've never heard of the organization that publishes the material, you should definitely find out more about them before you make any treatment decision based on their information.

The same goes for books. Academic presses (those run by universities) usually have a stricter process for making sure that there is accurate information in their books than do "commercial" or "trade" presses. Beware of presses or Web sites that are run by people trying to sell health or medical products, especially if ads or "testimonials" about products appear alongside treatment information. And what about knowing who funds the publisher or author, whether it's a Web site, press, or organization? It's a good idea, on the one hand, to "follow the money" so that you have an idea of how independent the publisher is from companies that have an interest in selling their products. On the other hand, pharmaceutical companies often give large grants for treatment education that have no strings attached -- in some cases, the organizations who get these grants are very thoughtful and careful about keeping their information as unbiased as possible. How well they manage this kind of separation depends on the overall commitment of the organization that acts as the publisher and on the kind of editorial process they have in place.

You may have heard people use the terms "primary sources," "secondary sources," and "anecdotal evidence." In the treatment world, "primary source" refers to the original publication that establishes any treatment-related "fact." Primary sources (usually peer-reviewed journal articles) are the first and strongest link in a chain of information. The best way to explain this might be to think of the game telephone, in which a simple message gets distorted just because it has gone through multiple people who try to repeat it, but change it slightly -- usually without meaning to. The same thing can happen with science. But sometimes information has to go through additional steps to be useful. For example, in a game of telephone, if the first person says something in French, and you don't understand French, it's not a very helpful message for you. If someone in between the first person and you translates it into English, there's a chance something is lost in translation, but there's also a chance you can get something out of the message. Likewise, primary sources aren't the most useful source for every reader or every purpose.

Not Sure If You're Reading It Upside Down? Maybe It's Not The Most Useful Source

So what are the disadvantages of primary sources? They are sometimes too limited in terms of the particular conditions or drugs they consider, too technical, and too focused on specific "study situations" to be useful for making an individual decision. Each primary source is an original report of a single study. Secondary sources are more varied, but what they have in common is that they draw directly on one or more primary source. Sometimes "secondary" reports are written just to simplify or shorten a primary report, and sometimes they are written to draw a broader picture of a treatment issue. Sometimes they do both. For example, the Fall 2003/Winter 2004 issue of ACRIA Update -- "Drugs! Drugs! Drugs!" -- used material from hundreds of "primary" sources to provide an easy-to-understand overview of all currently approved antiretrovirals available in the United States.

"Anecdotal evidence" is the slipperiest, because this refers to information that doesn't come from scientific study, but from personal observation, experience, or even opinion. Sometimes anecdotal evidence can be pretty convincing, especially when it's evidence that comes from your own body! And anecdotes often lead to specific studies or more carefully-considered "case reports." But in general, an "anecdote" specifically means that the information has not been tested in a systematic way, and the results may not be reproducible even if they were studied in a systematic way.

Although primary research reports are the gold standard for distributing specific information, they aren't meant to address an individual's medical decisions. To be useful, these primary reports must be put in the context of other research and clinical practice. Most people won't want (let alone have the necessary time and training) to read many primary research reports and synthesize them for themselves. So what to do? Look for clearly written reviews that pull together multiple primary sources (rather than "testimonials" and anecdotes or single studies). The most useful reviews are not just a laundry list of studies or results, but those that actually synthesize multiple studies -- that is, they help the reader make sense of patterns in the findings, major remaining questions, and so on.

There are always some science buffs who prefer to dip into the primary literature, at least now and then. Once you've found a peer-reviewed journal article on a topic that is important to you, how do you make sense of it? If you don't have much experience reading science, it's not going to be easy. But it's usually possible to get something out of it. Your best bet at first might be to get someone who is experienced to talk you through it. Take an interesting or important-looking article to your favorite community-based treatment organization, to a nurse practitioner, or to a friend who is more experienced and comfortable reading science. The most important sections to read are the "Discussion" and "Conclusion," because this is where the results of the study will be discussed in relation to other research and the implications for clinical care or future research will be addressed. The hardest part to read is going to be "Methods," and unless you're trying to become a treatment expert, you can happily skip over that.

Another issue with primary sources is that they aren't as accessible as other material. Medical journals are expensive, they often aren't available on the Web, and most public libraries can't afford them. But there are ways that regular people can get their hands on them. Lots of college and university libraries, as well as public libraries in some large (or affluent) cities, buy subscriptions that offer online access to a huge number of journals. With college and university libraries, there's a chance that you can get a visitor pass. Call the library at the college to find out if they have a visitor program. Some research institutes and non-profit organizations also have libraries, and these are sometimes open for visitors by appointment. One of the best library collections is the National Library of Medicine, which has great databases available free and without a subscription on the Web, including some free links to journals.

Safe and Satisfying Surfing

The Web is like the ocean -- it's vast, it's always moving, and there's lots of nourishment in there, as well as quite a few sharks. And don't forget how much garbage is dumped in the ocean! Seriously, though, the Web is a fabulous source of information if you have some basic knowledge about how to use it effectively. As a teacher, one of the most interesting and frustrating things I've learned about how students use the Web is that most seem to approach it as a "free-for-all" -- type a keyword or two into and let it rip! And while Google is one of my favorite search engines, there are usually better places to start if your goal is to find useful treatment information. In this issue of ACRIA Update, for example, there is an annotated resource list that includes dozens of Web sites run by reputable organizations -- start your search by picking some of those Web sites. Go directly to them and then search within the site for information that's relevant to you.

This kind of search limits the information you get, which is a mixed blessing. If you just type "Crixivan" into a search engine (Lycos, Google, MSN, Yahoo, etc.) you'll get thousands upon thousands of entries, and it will definitely contain some great links to useful information. But there is no quality control -- you may wind up with a list of sites that ranges from AEGIS (see the resource list!) to the long-winded and inexpert ranting of Joe Anybody on his personal Web page. Plus, even if they were all great, thousands of entries are overwhelming. Where do you look first? Lots of folks just go down the line, first-come, first-served fashion. It's helpful to know that most search engines list results in two different ways: "sponsored links" versus the regular search results. "Sponsored links" are paid advertisements. Companies or organizations pay other Web sites to list their sites whenever certain keywords are entered. The regular search results usually list Web pages in order of how many "hits" they have -- that is, how many times other people have looked at that particular page.

It's worth a brief aside here to note that doing a "Web search" is not the same as searching an electronic database. A Web search will bring up relevant material from any publicly accessible Web site. Electronic databases, on the other hand, are "closed systems" that only include information from specific sources. For example, the National Library of Medicine gives you access to databases such as PubMed, a gigantic collection of citations and abstracts from biomedical journals. The full texts of some of the articles listed on PubMed are even available for free. Databases like PubMed are great for finding primary source material (those peer-reviewed journal articles), but aren't always best for finding very user-friendly overviews of treatment issues. For that kind of search, you'll need to surf the Web or visit the Web sites of specific organizations.

Regardless of how you start your search, there are a few more specific tips for evaluating the Web sites you end up on. Remember to check out who runs the site (the "publisher" or "sponsor"), who pays for it, and a description of the editorial policy. There will often be a tab somewhere near the top called "About Us" that should contain this information. The best sites will also give you information about the qualifications or experience of the people writing for or editing the site and give you some way to contact the site for more information. Ideally, there will be citations listed for any claims made on the site. If they accept advertising, it should always be clearly distinguished from the "information" they are providing.

Finally, a pet peeve of mine and lots of other educators is that stuff on the Web circulates forever, but can deceivingly look as fresh as the day it was published. Beware of outdated material! Know that people and organizations can pay Web sites to make sure that their material stays accessible long past its useful shelf life. All the best Web sites date their pages -- this is usually a little line somewhere near the bottom of the page that reads "last updated on (date)." If you're looking for information that was listed in a "back issue" of a journal or a newsletter, you might look for a tab or link to "archives." Some journals allow free access to archives even if you have to pay for current issues, while others give up the new stuff for free and make you pay to get things from the archive. Go figure.

My Way or the Highway? Naaaah

It's no surprise that there is no single right way to get treatment information. But there are definitely more and less useful ways. My biggest concern in this article has been to help you understand the tradeoffs you make by going to one source versus another. Nobody wants to spend all their waking hours pouring over treatment information (unless there's some treatment-fetish crowd I'm not aware of, which I guess I shouldn't rule out ...). I hope these tips help you make the most of the time and energy you spend zooming around the information superhighway. Happy Cruising!

Rebecca Young, Ph.D. has been an HIV/AIDS activist, educator, and researcher since 1987, when she co-founded a consortium for street outreach workers in Washington, D.C. She works at the Center for Drug Use and HIV Research at NDRI, Inc. and the department of Women's Studies at Barnard College, both in New York City.

This article was provided by AIDS Community Research Initiative of America. It is a part of the publication ACRIA Update. Visit ACRIA's website to find out more about their activities, publications and services.


The content on this page is free of advertiser influence and was produced by our editorial team. See our content and advertising policies.