The Dangers of Reviewing Open Source CMS

The April issue of Adobe Edge contains the article, Review of open source content management systems. The article provides an overview of what the author describes as "five of the top open source software (OSS) solutions". The five open source CMS included in the author's list are CMS Made Simple, Drupal, Joomla!, WordPress, and XOOPS. After reading the article, I found myself wondering how we "reviewers" can actually improve our reviews of open source CMS. More importantly, I've come to the realization that I can no longer claim to be non-biased in which CMS I believe is the best out there.

The author does a fine job in the article describing the similarities and differences between the CMS being reviewed. However, one of the issues I have in this article and many others I've read that review CMS is the big jumps in the conclusion:

Drupal, Joomla!, and XOOPS are best for building an e-commerce site because all three offer:

  • Inventory management
  • Support for third-party payment processing mechanisms (such as PayPal)
  • Modules for shipping and sales tax calculators
  • Shopping cart functionality

While it is true that Drupal, Joomla! and XOOPS can do e-commerce, none of these CMS can do that straight out of the box. I can just imagine a shop owner or design company trying Drupal, Joomla!, or XOOPS for the very first time and wondering, "how the heck do I get a shopping-cart into the CMS?". While the author does hint in the article that third-party modules are needed to make the e-commerce work, I think the author would have been better off better explaining that "some work is required" to get those features into the CMS.

It is reviews like this where open source loses many of their potential customers. When we oversell what a CMS can and cannot do for first time users, I think many of those customers "move on" before getting to know the CMS better. Reviews like these can get the expectations of first time users so high that after the initial install, they're disappointed in what they see in the CMS. Is it no wonder that people stick with shopping-cart only solutions like osCommerce and Magento? You tell me, am I out of line with my criticism of this review? In fairness to the author, she does also indicate that the point of her article was not to spell everything out for the reader, but to give "some sense of direction as you navigate the many options and determine the CMS that best suits your requirements".

After reading this article, I also have found myself admitting my own shortcomings. The author concludes that she agrees with Packt Publishing's 2007 award putting Drupal at the top of the list of best open source CMS.

I tend to agree with Packt Publishing's award. I think Drupal is the best overall CMS available — based on its high level of customization, flexibility, scalability, taxonomy, SSL support, e-commerce functionality, and extremely enthusiastic user community that develops modules and provides technical advice and support.

I was on Packt's panel of judges for that award. During my tenure as judge, I was fair and non-biased to which CMS should be ranked on top of the list. In the end I picked Drupal. Even now as I look out on the landscape of open source, I still don't see any overall open source CMS out there better than Drupal. But that experience sitting as a judge on the panel in the Summer of 2007 changed me. I fell in love with Drupal and Drupal's community of developers, advocates, and users.

I now have to admit to myself that it will be a very long time before I can review a CMS without using Drupal as my reference. If asked today to sit on a panel judging the best PHP-based CMS, I would have to excuse myself from that nomination. I can no longer judge or review a PHP-based CMS without showing my bias for Drupal. At the moment, my belief that Drupal is the best CMS out there has lost me my reputation of being non-biased in my own reviews. I say for this, damn you Drupal.