Evaluating data sources

As part of their informational literacy education, high school and college students are provided with an evaluation tool called the C.R.A.A.P test. The tool’s intention is to help students evaluate the validity and usefulness of the information sources they use in their research papers.

The protocol can also be used to evaluate the information streams businesses use when making decisions about entering new markets, developing sales and marketing campaigns, developing new products, managing the health of the customer base, etc. In all of these cases decision makers have a number of data sources available to inform their decision ranging from the day-to-day experiences of the sales force to the management team’s expertise to third party research and reports. Each source addresses the question from a different angle and they can come to different conclusions and occasionally conflict with each other. This contributes to the analysis paralysis that plagues many organizations. Decisions-makers must untangle these differences so they can move forward. The C.R.A.A.P test can help evaluate the strengths, biases and limitations of each data source.

The five elements of evaluation are:

• Currency: How timely is the information? Does it describe how things are today or how they were in the past?
• Relevance: How important is the information to the decision at hand?
• Authority: Who is the source of the information? What are the source’s credentials?
• Accuracy: What is the information based on? Is it qualitative or quantitative data? Is it personal opinion?
• Purpose: What is the intention of the information? Is it impartial? Is it biased?

Some practitioners recommend scoring each data source on these dimensions. However, simply evaluating them in a systematic manner can help identify where and why disconnects exist. Often seemingly conflicting data sources are actually talking about different things. Understanding these differences adds to the overall information available for the decision.

As a primary research firm we recommend using a systematic approach to incorporating data sources into any business decision. None of the data streams available ever has all the answers and none are ever completely wrong. If evaluated and used correctly each additional source of data increases the organization’s ability to make a better decision.