Choosing the Best Analytic Suite: Best of Breed vs. an Integrated, All-in-One Solution

In the past, choosing a web analytic tool or service was simple as there were very few vendors. Today, however, there are literally hundreds of services and tools available, costing the customer anywhere between zero and tens of thousands of dollars each year. After focusing on vendors in my two previous posts, I would now like to focus on the customers, and the decisions they have to face when choosing a Web Analytic Suite.

 

From the customer’s perspective – Web Analyst or Marketer – having such a huge variety of tools and services to choose from makes their mission almost impossible. Let’s say, for example, that we want to test and optimize Landing Pages. We can choose between Optimizly, Unbounce, Visual Website Optimizer, Monetate, LiveBall, Reedge, or any number of other tools. If that’s not enough, we can choose a free solution, such as Google Website Optimizer, or download an open-source project such as Genetify. We even have the option of choosing an all-in-one package such as Unica or Omniture.

 

Each tool has its own architecture and advantages, and of course, its own limitations. Some are on-premise solutions while others are cloud based. Some require technical involvement for the set up period and then for each and every test we wish to run. Others require the mapping of a dedicated subdomain and then host the landing pages externally – or on-site, depending on the tool.

 

Confusing? I agree. And that’s just for testing and optimizing Landing Pages. Imagine what happens when we have to choose a complete web analytic package.

 

There are two ways of choosing a complete web analytic package. One approach is to choose the best tools available (AKA Best of Breed), while the other approach is to purchase an all-in-one suite (AKA an All-in-One Solution), which contains all or most of the tools needed.

 

If you choose to create a Best of Breed package, there are numerous issues that need to be addressed. The first thing is the importance of having a JavaScript Guru. I addressed this matter in an earlier post, in which I talked about my personal selection of tools – which includes Visual Website Optimizer, KISSinsight, Google Analytics and others. However, although having a JavaScript guru who can implement and integrate the chosen tools is a necessary condition, it is insufficient, as unfortunately there are many additional obstacles along the way:

 

Discrepancies

The most common question asked by my clients relates to discrepancies between different tools. Although there are many reasons for discrepancies, one thing I can say with great certainty is that in most cases, the problem is not a technical one (although that is the first thing we tend to check).

The problem begins with the unfortunate fact that there are no set standards for web analytics – neither terminology standards, nor technical implementation and common metric standards. What solution A defines as “Unique Users” is probably something completely different to solution B’s definition. You have visitors, Unique Visitors, Users, Visits, Sessions, Pageviews, Events, Unique Pageviews, First Pageviews, Bounce Rate, Exit Rate, Pan-Session, Multi-Session, Multi-Channel, Funnels, Micro-Funnels, Upper-Funnels, Cookies, Flash Cookies, and Ever-Cookie. The list is endless, and each solution uses its own terminology, metrics and unique way of identifying users. Here are just a few examples:

  • MixPanel uses IP to identify the user, while KISSinsights uses cookies. Using them together will probably result in discrepancies in the Unique Visitor metric.
  • According to Google Analytics, a visit ends when the user uses a different traffic source, while other solutions define the end of a visit after 30 minute. Using them together will probably result in discrepancies in the Visit metric.
  • Clicky defines a bounced visit as a visit with only one pageview on the site and that lasts less than 30 seconds. Other solutions define the bounce rate differently. Again, using Clicky and another tool together could result in discrepancies in the Bounce Rate.
  • Google Analytics calculates Time on Page differently than other tools (i.e., without the last page in session). GA might tell you that on average, visitors stayed on a page for 9 seconds, yet ClickTale states that they actually stayed for 3 minutes. If this happens, please don’t blame your technical person, and don’t assume they didn’t implement the tags correctly.

It’s not easy to eliminate discrepancies, I know. But do not despair – I have 3 basic steps to suggest to make things easier:

  1. Read the f** manual (no offense). Study the definition and make sure you understand how the tool identifies the user, how it calculates the different metrics, and which terminology it uses. You need to become familiar with all the tools you are going to be using.
  2. Try to recognize what can be compared. For example, if one of your tools uses cookies to identify users, while another uses Etags, then trying to compare Unique Users will probably be useless. On the other hand, Pageviews or other metrics such as Visits or Unique Pageviews might be similar. You might even find that you can compare two different metrics – for example, Visits on tool A and Unique Pageviews on tool B.
  3. Compare Trends instead. That way, even if there are differences in the amount of Visits or Conversions, you will almost always receive the same trend.

Pricing

The differences in pricing are insane. You can get a decent package for almost no licensing fee, or alternatively you can pay $140,000 annually just for Google Analytics. And that’s before you’ve added the extra costs for complementary tools and the actual work. Because of the huge difference in pricing, some companies end up paying more than $150K, while other companies only pay around $10K each year for the same set of features and monthly pageviews.

 

The most important thing to remember is that there are no free meals. You have to pay for your web analytic suite in some way: either for licensing or for implementing, or both. If you choose Omniture or Unica, then in addition to the licensing fee, you also have to pay for a technical person to customize, maintain and operate the tool. On the other hand, If you go with Piwik as your primary reporting platform, BTBuckets for behavioral targeting, Genetify for testing, ClickHeat for heat maps and so on, you might not have to pay anything for licensing but you will probably have to pay programmers to implement, integrate and maintain them.

 

Although you have to pay for web analytics one way or another, the good news is that you should see the Return on Investment pretty quickly. If you don’t, you should check the tools you are using and/or your Web Analyst/Optimization Manager.

 

I personally don’t think that more than a few thousand dollars a month should be spent on a comprehensive suite, which is why I’m not sure I understand why using Google Analytics Premium should cost $140,000 per year.

 

If you choose Unbounce or visual website optimizer (AB and MVT), KISSmetrics (Funnels and Customer retention), Google Analytics (the free version), KISSinsights (Surveys), ClickTale (visitor recording), GoSquared (Real time), Crazy Egg (heat maps) and hire a JavaScript freelancer to hook you up with all necessary customized codes – you will probably end up with a $40-50K solution over 3 years. That’s around $1-1.5K a month, and you will probably have one of the best suites available. If you are a relatively small business, you can probably even get all this for less.

 

Synergy

Let’s say we have now picked the best tools, have hired a JavaScript Guru and a Web Analyst, have successfully nailed the discrepancies, and understand what can be tracked, and how and what can be used together. There is, unfortunately, still one catch: Synergy.

 

For example, KISSinsights and ClickTale do not use the same infrastructure, nor do they use the same repository or talk in the same language. ClickTale identifies a certain user as User X, while KISSinsights identifies the same user as User Y. This makes it difficult to create a pop-up survey for a specific visitor or segment that has been defined in ClickTale. In fact, this makes it impossible.

 

How about if we want to create a segment in Google Analytics and run a test for that segment alone, using Unbounce? We may want to capture Real Time activity, or record visitors for just for that specific segment. The same goes for wanting to display KISSinsights information on a Google Analytics Dashboard, or defining goals using GA, and then evaluating the performance of the Visual Website Optimizer tests. Again – almost impossible to achieve.

 

A few days ago, I came across what seemed like a really great offer that included Uncounce, KISSmetrics, Olark and Hello Bar. But can they really be integrated, and can information be shared and combined between them? The answer is: Probably Not.

 

The problem with a Best of Breed approach is that in most cases we lose the ability to obtain true synergy between the tools with regards to all aspects, including Support and Resources. The only thing we can do is hire full-time developers to keep all these tools in sync, or manually check and calculate goals, segments and all other information across all the tools.

 

Conclusion

I am a long time customer of web analytic services. Besides the fact I am also part of Intlock, which is a web analytic vendor for SharePoint based sites, I have been using a variety of web analytic solutions for the past decade – for my clients’ sites and my own. From experience I can honestly say that there are some really amazing tools. However, when it comes to a holistic solution, the only option is to go with an all-in-one package, such as Unica or Omniture. Unfortunately, this option requires spending tens of thousands of dollars each year – which in most cases is not worth the investment.

 

At Intlock we believe that the best way to move forward is by providing a synergized, all-in-one package, and at a low price, in order to overcome all the issues mentioned above, and free the customer from the need to hassle with 5 different and non-connected products in order to achieve a holistic analytic goal. When you create a segment you can create surveys, conduct tests, and target content, and later on even view all of the reports for this segment – without the need to export and import, or perform any type of integration between different tools. (You won’t even have to keep a technical person nearby…).

 

From a pricing perspective, we provide a much more reasonable alternative to expensive tools such as WebTrends or Omniture, which makes the question “Should we use the Intlock solution?” a no-brainer decision – in my humble opinion – but I guess I am not that objective 🙂

 

Just before I sign off, I’d like to mention a few interesting and relevant posts that I have recently read: