In the beginning there was the data warehouse. It was designed to alleviate the problems caused when companies had dozens of different systems containing disparate data based on which staff needed to make important business decisions. Each system could store data in its own quirky way, and worse yet, each system's data frequently could be inconsistent with that in other systems. Chaos reigned.
Warehouses were thus designed to let you gather all the disparate data into one giant virtual processing plant, clean it up, and then put it into a new, orderly warehouse. Here everyone (with permission, that is) could access the data and trust that what he or she got was good, consistent information across the board. The data Joe in cubicle was A obtained to determine July sales forecasts would be consistent with the data that Susan in office B used to judge Joe's performance against his peers. Everyone would be happy (as long as Joe was good at his job).
Unfortunately, because the process of bringing together data from innumerable systems is a challenging and time-consuming one, data warehouses initially came with a high price tag and long implementation time. As people discovered the competitive edge to be gained by having up-to-date data, they became impatient to put together a warehouse quickly. Because management usually hadn't yet been convinced of the value of the warehouse's high price tag, they were impatient for a less pricey solution.
Out of this need arose the data mart, which let customers build a miniwarehouse of data from a vertical segment of the company's data, say, accounts payable or sales. Marts took less time and cost less money. They took off like hotcakes. Battles arose between the mart and warehouse companies: Which is better, a warehouse or a mart?
Although data marts provided a quick fix, they created the very same problem data warehouses were designed to solve. Joe in cubicle A works in a sales department that just rolled out a solution for its data mart project. Meanwhile, completely unaware of what the same department is doing, the accounting department across the country has also decided it needs a data mart and chooses a different vendor. Once again you have two different systems within the same company. Where did the added value go?
Data mart companies heard the cry and started altering their products so that you could eventually join them, like Legos, into one big data warehouse. They argued that companies could start their warehouse small by implementing just one mart at a time, not entering into warehouse matrimony until they were ready.
This sounds good in theory, but let's not forget that companies are organic entities made up of human beings who are, by their very nature, unpredictable. If companies were to remain populated by a static group of employees who never changed, and if the companies themselves never changed ownership, the mart-to-warehouse plan might work brilliantly. I do not doubt that many companies have achieved success taking this route, and I do not intend to belittle their achievement.
But what guarantee is there that within six months you won't have a new CTO who doesn't like the previous mart choice and wants a new system from his or her preferred vendor? Or what if your company is acquired by a larger company that already has its own full data warehouse project in place and wants nothing to do with your existing mart technology? You're back at square one. Although the technology to help you migrate from system to system has evolved nicely, you still must redo work you've already done.
Maybe you did succeed in populating your organization with several compatible marts and martlets. Like giving birth to sextuplets, your next challenge is to keep track of them. Only this time, each mart or martlet may involve its own team of sextuplets. People discuss ways to keep marts in sync, magazines publish extensive articles on it, and a whole new segment of the market has devoted itself to it. The fact remains, however, that there are still too many ways for data to get out of sync, period.
I anticipate that the pendulum may swing away from data marts. It's only a question of what form it will take and when it will arrive. I'm not predicting the demise of data mart companies or recommending that you kill your data mart projects and run for the warehouse. I'm merely saying that the "mart market" will evolve into a new form.
What will that form be? It's probably too soon to tell. Send me your own predictions (to cparkes@earthlink.net), and together, we might be able to catch a rising trend before it gets its own buzzword. Even better, we can create the term ourselves.