Evaluating Internet Research Sources
Evaluating Internet Research Sources
Robert Harris
Version Date: January 21, 2015
Previous: December 27, 2013; November 6, 2013; Nov. 22, 2010 and June 15, 2007
"The central work of life is interpretation." --Proverb
Introduction: The Diversity of Information
Information is a
Commodity Available
in Many Flavors
Think about the magazine section in
your local
grocery store. If you reach out with your eyes closed and grab the
magazine you touch, you are about as likely to get a supermarket
as you are a respected journal (actually more likely, since many
journals don't fare well in grocery stores). Now imagine that your
is so accommodating that he lets anyone in town print up a magazine and
put it in the magazine section. Now if you reach out blindly, you might
get the Elvis Lives with Aliens Gazette just as easily as Atlantic
Monthly or Time.
Welcome to the Internet. As I hope my analogy makes clear,
an extremely wide variety of material on the Internet, ranging in its
reliability, and value. Unlike most traditional information media
magazines, organizational documents), no one has to approve the content
before it is made public. It's your job as a searcher, then, to
what you locate, in order to determine whether it suits your needs.
Information Exists on a
of Reliability and Quality
Information is everywhere on the
Internet, existing
in large quantities and continuously being created and revised. This
information
exists in a large variety of kinds (facts, opinions, stories,
interpretations,
statistics) and is created for many purposes (to inform, to persuade,
sell, to present a viewpoint, and to create or change an attitude or
For each of these various kinds and purposes, information exists on
levels of quality and reliability. It ranges from very good to very bad
and includes every shade in between.&
Adopting a Skeptical Attitude
You might have heard of the term information warfare,
the use of information as a weapon. Now, as I just said above, there is
a lot of high quality information available through the Internet.
Unfortunately, however, there is also a large amount of misinformation
(honest people mistakenly spreading false information), together with
the information warfare ammunition: disinformation (dishonest people
knowingly spreading false information), half truths, distortions, urban
legends, fallacies, exaggerations, and plain old lies. So you need to
adopt a skeptical attitude toward all knowledge claims. You don't have
to believe everything you read, hear, or see. (Nor do you have to
become completely cynical and disbelieve everything.) Just be very
cautious. Instead of putting each knowledge claim into your "accept" or
"reject" pile, put it in a "this is claimed, but I neither accept nor
reject it right now" pile. Seek confirmation and disconfirmation from
other sources and your own thinking. Remember that a Web site that
makes up facts, quotes people who don't exist, and cites imaginary
sources could look just as professional and slick as a highly reliable
and credible site. Don't be deceived by appearances. Develop your
critical thinking skills.
Getting Started: Screening Information
Pre-evaluation
The first stage of evaluating your
sources takes
place before you do any searching. Take a minute to ask yourself what
you are looking for. Do you want facts, opinions (authoritative or just
anyone's), reasoned arguments, statistics, narratives, eyewitness
descriptions? Is the purpose of your research to get new ideas, to find
either factual or reasoned support for a position, to survey opinion,
something else? Once you decide on this, you will be able to screen
much more quickly by testing them against your research goal. If, for
you are writing a research paper, and if you are looking for both facts
and well-argued opinions to support or challenge a position, you will
which sources can be quickly passed by and which deserve a second look,
simply by asking whether each source appears to offer facts and
well-argued
opinions, or just unsupported claims.
Select Sources Likely to
be Reliable
Becoming proficient at selecting
sources will
require experience, of course, but even a beginning researcher can take
a few minutes to ask, "What source or what kind of source would be the
most credible for providing information in this particular case?" Which
sources are likely to be fair, objective, lacking hidden motives,
quality control? It is important to keep these considerations in mind,
so that you will not simply take the opinion of the first source or two
you can locate. By thinking about these issues while searching, you
be able to identify suspicious or questionable sources more readily.
so many sources to choose from in a typical search, there is no reason
to settle for unreliable material.
But Wait a Minute
Remember that to locate fair,
objective material, you must be fair and objective, too. A major error
too many researchers make is to look only for sources whose ideas,
findings, or arguments they already agree with. It's fine to have a
sense of where you think you are going, but you should be open to
opposing ideas and not discount them just because you don't like them
or because they conflict with your planned direction. The best
researchers usually don't start out "to prove X." Instead, they start
out "to find out about X." Be careful not to fall into that circular
reasoning trap by thinking, "Books expressing that view are unreliable."
Source Selection Tip:
Try to select sources that offer as much of the following information
as possible:
Author's Name
Author's Title or Position
Author's Organizational Affiliation
Date of Page Creation or Version
Author's Contact Information
Some of the Indicators of Information Quality (listed below)
Evaluating Information: The Tests of Information Quality
Reliable Information is
You may have heard that "knowledge
is power,"
or that information, the raw material of knowledge, is power. But the
is that only some information is power: reliable information.
Information
serves as the basis for beliefs, decisions, choices, and understanding
our world. If we make a decision based on wrong or unreliable
information,
we do not have power--we have defeat. If we eat something harmful that
we believe to be safe, if we avoid something good
we believe to be harmful, we have needlessly restricted the enjoyment
our lives. The same thing applies to every decision to travel,
or act, and every attempt to understand.&
Source Evaluation is an
Source evaluation--the determination
of information
quality--is something of an art. That is, there is no single perfect
of reliability, truthfulness, or value. Instead, you must make an
from a collection of clues or indicators, based on the use you plan to
make of your source. If, for example, what you need is a reasoned
then a source with a clear, well-argued position can stand on its own,
without the need for a prestigious author to support it. On the other
if you need a judgment to support (or rebut) some position, then that
will be strengthened if it comes from a respected source. If you want
facts, then using facts from a source that meets certain criteria of
will help assure the probability that those facts are indeed
reliable.&
The CARS Checklist
The CARS Checklist (Credibility,
Accuracy, Reasonableness,
Support) is designed for ease of learning and use. Few sources will
every criterion in the list, and even those that do may not possess the
level of quality possible. But if you learn to use the criteria in this
list, you will be much more likely to separate the high quality
information
from the poor quality information.
The CARS Checklist for Information Quality
Credibility
Because people have always made
important decisions
based on information, evidence of authenticity and reliability--or
credibility,
believability--has always been important. If you read an article saying
that the area where you live will experience a major earthquake in the
next six months, it is important that you should know whether or not to
believe the information. Some questions you might ask would include,
about this source makes it believable (or not)? How does this source
this information? Why should I believe this source over another? As you
can see, the key to credibility is the question of trust.
There are several tests you can apply to a source to help you
how credible and useful it will be:&
Author's Credentials
The author or source of the
information should
show some evidence of being knowledgeable, reliable, and truthful. Here
are some clues:&
Author's education, training, and/or experience in a field
relevant to
the information. Look for biographical information, the author's title
or position of employment&
Author provides contact information (email or snail mail
address, phone
Organizational authorship from a known and respected
organization
(corporate,
governmental, or non-profit)&
Author's reputation or standing among peers.&
Author's position (job function, title)
Evidence of Quality Control
Most scholarly journal articles pass
a peer review process, whereby several readers must examine and approve
content before it is published. Statements issued in the name of an
organization
have almost always been seen and approved by several people. (But note
the difference between, "Allan Thornton, employee of the National
Oceanographic
and Atmospheric Agency, says that a new ice age is near," and "The
Oceanographic and Atmospheric Agency said today that a new ice age is
The employee is speaking for himself, whereas a statement in the name
NOAA represents the official position of NOAA.)&
Evidence of quality control of Internet material includes
these items:&
Information presented on organizational web sites&
On-line journals that use refereeing (peer review) by
editors or
Postings of information taken from books or journals that
control process
Note: Appearances can be deceiving. Don't assume that
a great-looking Web site is automatically credible. Very professional
and sophisticated Web page templates are available for a few dollars,
so that anyone and his pet skunk can put up a site that looks expensive
and authoritative. Good looks are not evidence of credibility.
Metainformation
Metainformation
is information about
information.
Information workers (sometimes called knowledge workers) all over the
are constantly poring over, processing, and evaluating information--and
making notes. As the challenges produced by the increasing quantity of
information continue, access to high quality metainformation will
increasingly important. Metainformation can take many forms, but there
are two basic types, summary and evaluative.&
Summary metainformation includes all the shortened forms of
information,
such as abstracts, content summaries, or even tables of contents. This
type of metainformation gives us a quick glance at what a work is about
and allows us to consider many different sources without having to go
them completely.&
Evaluative metainformation includes all the types that provide
judgment or analysis of content. This type includes recommendations,
reviews, and commentaries. Even the search results order of pages from
a search engine like Google represents a type of evaluative
metainformation,
since pages are ranked in part by the number of other pages linked to
(and hence "voting" for them in some sense).
And, of course, these two types can be combined, resulting in
form of metainformation, providing us with a quick overview and some
evaluation
of the value. An examples would be a World Wide Web yellow pages or
which describes each selected site and provides evaluations of its
Indicators of Lack of Credibility
You can sometimes tell by the tone,
competence of the writing whether or not the information is suspect.
are a few clues:&
Anonymity&
Lack of Quality Control&
Negative Metainformation. If all the reviews are critical,
be careful.
Bad grammar or misspelled words. Most educated people use
well and check their work for spelling errors. An occasional split
infinitive
or comma in the wrong place is not unusual, but more than two or three
spelling or grammar errors is cause for caution, at least. Whether the
errors come from carelessness or ignorance, neither puts the
information
or the writer in a favorable light.
Emotional earnestness accompanied by exaggeration or
absolutes. Even in very controversial areas (gun control, global
warming, abortion, capital punishment) and promotional contexts
(product claims and evaluations) we expect reasons, data, and emotional
restraint. Articles where the writer's feelings have clearly taken over
from thinking make us wonder if we are reading ideology instead of
information and arguments that might persuade us. Breathless, sweeping
generalizations should set off your baloney detector. For example, "Did
you know that none of the vitamins and supplements sold in stores work
correctly with your body chemistry? Only SuperDuperVite has been
formulated to blah blah blah."
of unique, secret information (which is now on the
Web site) or claims of such dramatic implications that you should
expect widespread discussion. For example, "The CIA was responsible for
the assassination of President Kennedy." Conspiracy theories in
general, because they run counter to official reports and often counter
to reason, should be met with great caution.
The goal of the accuracy test is to
assure that
the information is actually correct: up to date, factual, detailed,
and comprehensive. For example, even though a very credible writer said
something that was correct twenty years ago, it may not be correct
Similarly, a reputable source might be giving up-to-date information,
the information may be only partial, and not give the full story. Here
are some concepts related to accuracy:
Timeliness
Some work is timeless, like the
classic novels
and stories, or like the thought provoking philosophical work of
and Plato. Other work has a limited useful life because of advances in
the discipline (psychological theory, for example), and some work is
very quickly (such as technology news). You must therefore be careful
note when the information you find was created, and then decide whether
it is still of value (and how much value). You may need information
the past ten years, five years, or even two weeks. But old is not
necessarily
bad: nineteenth-century American history books or literary anthologies
can be highly educational because they can function as comparisons with
what is being written or anthologized now. In many cases, though, you
accurate, up-to-date information.&
An important idea connected with timeliness is the dynamic,
fluid nature
of information and the fact that constant change means constant changes
in timeliness. The facts we learn today may be timely now, but tomorrow
will not be. Especially in technology, science, medicine, business, and
other fields always in flux, we must remember to check and re-check our
data from time to time, and realize that we will always need to update
our facts.
Note: Many Web pages display today's date automatically,
regardless
of when the content on the page was created. If you see today's date on
a page other than from a news site, be extra careful.
Comprehensiveness
Any source that presents conclusions
claims (explicitly or implicitly) to give a full and rounded story,
reflect the intentions of completeness and accuracy. In other words,
information should be comprehensive. Some writers argue that
researchers
should be sure that they have "complete" information before making a
or that information must be complete. But with the advent of the
information
age, such a goal is impossible, if by "complete" we mean all possible
information.
No one can read 20,000 articles on the same subject before coming to a
conclusion or making a decision. And no single piece of information
offer the truly complete story--that's why we rely on more than one
On the other hand, an information source that deliberately leaves out
facts, qualifications, consequences, or alternatives may be misleading
or even intentionally deceptive.
Audience and Purpose
For whom is this source intended and
purpose? If, for example, you find an article, "How Plants Grow," and
are the intended audience, then the material may be too simplified for
your college botany paper. More important to the evaluation of
information
is the purpose for which the information was created. For example, an
titled, "Should You Buy or Lease a Car?" might have been written with
purpose of being an objective analysis, but it may instead have been
with the intention of persuading you that leasing a car is better than
buying. In the latter case, the information will most likely be biased
or distorted. Such information is not useless, but the bias must be
into consideration when interpreting and using the information. (In
cases, you may need to find the truth by using only biased sources,
biased in one direction and some biased in the other.) Be sure, then,
the intended audience and purpose of the article are appropriate to
requirements or at least clearly in evidence so that you may take them
into account. Information pretending to objectivity but possessing
hidden agenda of persuasion or a hidden bias is among the most common
of information in our culture.
Indicators of a Lack of Accuracy
In addition to an obvious tone or
style that
reveals a carelessness with detail or accuracy, there are several
indicators
that may mean the source is inaccurate, either in whole or in part:
No date on the document&
Vague or sweeping generalizations&
Old date on information known to change rapidly
Very one sided view that does not acknowledge opposing
views or respond
Reasonableness
The test of reasonableness involves
the information for fairness, objectivity, moderateness, and
consistency.
Fairness includes offering a
balanced, reasoned
argument, not selected or slanted. Even ideas or claims made by the
opponents should be presented in an accurate manner. Pretending that
opponent has wild, irrational ideas or arguments no one could accept is
to commit the straw man fallacy. A good information source will also
a calm, reasoned tone, arguing or presenting material thoughtfully and
without attempting to get you emotionally worked up. Pay attention to
tone and be cautious of highly emotional writing. Angry, hateful,
spiteful tones often betray an irrational and unfair attack underway
than a reasoned argument. And writing that attempts to inflame your
to prevent you from thinking clearly is also unfair and manipulative.
Objectivity
There is no such thing as pure
objectivity,
but a good writer should be able to control his or her biases. Be aware
that some organizations are naturally not neutral. For example, a
professional
anti-business group will find, say, that some company or industry is
overcharging
for widgets. The industry trade association, on the other hand, can be
expected to find that no such overcharging is taking place. Be on the
for slanted, biased, politically distorted work.&
One of the biggest hindrances to objectivity is conflict of
Sometimes an information source will benefit in some way (usually
financially,
but sometimes politically or even emotionally or psychologically) if
source can get you to accept certain information rather than the pure
objective truth. For example, many sites that sell "natural" products
(cosmetics,
vitamins, clothes) often criticize their competitors for selling bad,
or dangerous products. The criticism may be just, but because the
will gain financially if you believe the message, you should be very
careful--and
check somewhere else before spending money or believing the tale.
Moderateness
Moderateness is a test of the
information against
how the world really is. Use your knowledge and experience to ask if
information is really likely, possible, or probable. Most truths are
If a claim being made is surprising or hard to believe, use caution and
demand more evidence than you might require for a lesser claim. Claims
that seem to run against established natural laws also require more
In other words, do a reality check. Is the information believable? Does
it make sense? Or do the claims lack face validity? That is, do they
to conflict with what you already know in your experience, or do they
too exaggerated to be true? "Half of all Americans have had their cars
stolen." Does that pass the face validity test? Have half of your
had their cars stolen? Is the subject on the news regularly (as we
assume it would be if such a level of theft were the case)?&
It is important, of course, to remember that some truths are
spectacular
and immoderate. Over the past few decades, Michel Lotito, a French
with the stage name of Monsieur Mangetout (French for "eats
everything")
has actually eaten 18 bicycles, several TV sets, a few shopping carts,
and a small airplane by first having them ground into a fine powder and
sprinkling a few teaspoonfuls on his breakfast cereal each morning. So
do not automatically reject a claim or source simply because it is
astonishing.
Just be extra careful about checking it out.
Consistency
The consistency test simply requires
argument or information does not contradict itself. Sometimes when
spin falsehoods or distort the truth, inconsistencies or even
contradictions
show up. These are evidence of unreasonableness.
World View
A writer's view of the world
(political, economic,
religious--including anti-religious--and philosophical) often
influences
his or her writing profoundly, from the subjects chosen to the slant,
issues raised, issues ignored, fairness to opponents, kinds of
and so forth. World view can be an evaluative test because some world
in some people cause quite a distortion in their view of reality or
world view permits them to fabricate evidence or falsify the positions
of others. For some writers, political ideology or political agenda
takes precedence over
truth and sometimes even over fairness.
If you are looking for truth or a whole picture, such sources are not
Indicators of a Lack of Reasonableness
Writers who put themselves in the
way of the
argument, either emotionally or because of self interest, often reveal
their lack of reasonableness. If, for example, you find a writer
a book he opposes by asserting that "the entire book is completely
claptrap," you might suspect there is more than a reasoned disagreement
at work. Here are some clues to a lack of reasonableness:
Intemperate tone or language ("stupid jerks," "shrill cries
opponents")&
Overclaims ("Thousands of children are murdered every day
in the United
States.")&
Sweeping statements of excessive significance ("This is the
idea ever conceived!")&
Conflict of Interest ("Welcome to the Old Stogie Tobacco
Company Home
To read our report, 'Cigarettes Make You Live Longer,' click here." or
"The products our competitors make are dangerous and bad for your
The area of support is concerned
with the source
and corroboration of the information. Much information, especially
statistics
and claims of fact, comes from other sources. Citing sources
strengthens
the credibility of the information. (Remember this when you write a
Source Documentation or Bibliography
Where did this information come
from? What sources
did the information creator use? Are the sources listed? Is there a
bibliography
or other documentation? Does the author provide contact information in
case you wish to discuss an issue or request further clarification?
kind of support for the information is given? How does the writer know
this? It is especially important for statistics to be documented.
Otherwise,
someone may be just making up numbers. Note that some information from
corporate sites consists of descriptions of products, techniques,
technologies,
or processes with which the corporation is involved. If you are careful
to distinguish between facts ("We mix X and Y together to get Z") and
advertising
("This protocol is the best in the industry"), then such descriptions
be reliable.&
Corroboration
See if other sources support this
source. Corroboration
or confirmability is an important test of truth. And even in areas of
or opinion, if an argument is sound, there will probably be a number of
people who adhere to it or who are in some general agreement with parts
of it. Whether you're looking for a fact (like the lyrics to a song or
the date of an event), an opinion (like whether paper or plastic is the
more environmentally friendly choice), or some advice (like how to grow
bromeliads), it is a good idea to
triangulate your findings: that is,
at least three sources that agree. If the sources do not agree, do
research to find out the range of opinion or disagreement before you
your conclusions.
What you are doing with corroboration, then, is using
information to
test information. Use one source, fact, point of view, or
interpretation
to test another. Find other information to support and reconfirm (or to
challenge or rebut) information you have found.
Corroboration is especially important when you find dramatic
or surprising
information (information failing the moderateness test, above). For
the claim that a commonly used food additive is harmful should be
with skepticism until it can be confirmed (or rebutted) by further
The claim may be true, but it seems unlikely that both government and
organizations would let the additive go unchallenged if indeed it were
External Consistency
While the test of corroboration
involves finding
out whether other sources contain the same new information as the
being evaluated, the test of external consistency compares what is
in the new source with what is familiar in other sources. That is,
information
is usually a mixture of old and new, some things you already know and
things you do not. The test of external consistency asks, Where this
discusses facts or ideas I already know something about, does the
agree or harmonize or does it conflict, exaggerate, or distort? The
is that if a source is faulty where it discusses something you already
know, it is likely to be faulty in areas where you do not yet know, and
you should therefore be cautious and skeptical about trusting it.
Indicators of a Lack of Support
As you can readily guess, the lack
of supporting
evidence provides the best indication that there is indeed no available
support. Be careful, then, when a source shows problems like these:
Numbers or statistics presented without an identified
source for
Absence of source documentation when the discussion clearly
needs such
documentation&
You cannot find any other sources that present the same
information or
acknowledge that the same information exists (lack of corroboration)
Summary of The CARS Checklist
Research Source Evaluation
Credibility
trustworthy source,
author’s credentials,
evidence of quality control, known or respected authority,
organizational
support. Goal: an authoritative source, a source that supplies some
evidence that allows you to trust it.
up to date, factual,
exact, comprehensive, audience and purpose reflect intentions of
completeness
and accuracy. Goal: a source that is correct today (not yesterday), a
that gives the whole truth.
Reasonableness
fair, balanced,
objective, reasoned,
no conflict of interest, absence of fallacies or slanted tone. Goal: a
source that engages the subject thoughtfully and reasonably, concerned
with the truth.
listed sources,
contact information,
available corroboration, claims supported, documentation supplied.
a source that provides convincing evidence for the claims made, a
you can triangulate (find at least two other sources that support
Living with Information: The CAF? Advice
Here is one last piece of advice to
live well in the world of information: Take your information to the
(Challenge, Adapt, File, Evaluate).
Challenge information and demand
accountability.
Stand right up to the information and ask questions. Who says so? Why
they say so? Why was this information created? Why should I believe it?
Why should I trust this source? How is it known to be true? Is it the
truth? Is the argument reasonable? Who supports it?
Adapt your skepticism and
requirements for quality
to fit the importance of the information and what is being claimed.
more credibility and evidence for stronger claims. You are right to be
a little skeptical of dramatic information or information that
with commonly accepted ideas. The new information may be true, but you
should require a robust amount of evidence from highly credible sources.
File new information in your mind
rather than
immediately believing or disbelieving it. Avoid premature closure. Do
jump to a conclusion or come to a decision too quickly. It is fine
to remember that someone claims XYZ to be the case. You need not worry
about believing or disbelieving the claim right away. Wait until more
information
comes in, you have time to think about the issue, and you gain more
knowledge.
Evaluate and re-evaluate regularly.
New information
or changing circumstances will affect the accuracy and hence your
evaluation
of previous information. Recognize the dynamic, fluid nature of
information.
The saying, "Change is the only constant," applies to much information,
especially in technology, science, medicine, and business.
Books you need:
w w . v i r t u a l s a l t . c o m
About the author:
is a writer
and educator with more than 25 years of teaching experience at the
and university level. RHarris