The concept of data warehouse was at its peak of investments and hype in the ‘90s. Back then, enterprises spent a fortune on data warehouse. Seeing the potential that data offered, average investment by organizations ranged from USD 30 to USD 40 million on data warehousing capabilities. Enterprises started building infrastructures to assimilate and consume data on a common platform, but they had no clear idea about its utilization and life cycle management. They eventually used it as a central place for enterprises to collect transactional data and distribute it to those who needed it.
Why are traditional warehouses failing?
CIOs often ask this, because CFOs question them where the ROI is. When we speak to CXOs, they point out that only forty to fifty percent of their business needs are being met by a data warehouse - this is a low number for such an investment.
Though a centralised repository of data system was put in place, the investment didn’t live up to its promise. About ninety percent of enterprises failed in creating value because they implemented the wrong strategy. A data warehouse was perceived as a central base to collect all types of data across an enterprise. With a lot of unnecessary data coming in, the data overgrew its size. It worked as a data aggregation platform rather than a data information hub. There’s a structural flaw in this design, while data volume, governance and sustenance went for a toss. Enterprises didn’t consider a data warehouse as a strategic infrastructure which meets all the business data requirements of an enterprise.
Emergence of data lakes
Over the last five to six years, enterprises have been breaking data warehouse into data lakes. These are a new type of cloud-based enterprise architecture which creates data in specialized forms to cater to different lines of businesses. For instance, in a pharmaceutical organisation, data is split and stored separately for finance, commercial, HR, management etc.
While data lakes increase security and manageability of data, the whole purpose of a data warehouse is defeated because of silos and it’s reached a chaotic situation again. Information that should essentially be in the warehouse gets stored in lakes. As a result, a user who needs real time information ends up confused about where to pull out the data from.
There’s a need for data to be sorted in a strategic way. We need to strike a balance between data warehouse and data lakes and reap the benefits of both.
Trends driving next-gen environments
The dream of making real time data available to a user wasn't accomplished with the first wave of data warehouses due to wrong priorities and strategies. However, it can be fixed in most cases. You needn't dissolve the entire data warehouse system to build a new one, but rather, take advantage of the existing investment with data warehouse modernization. It helps enterprises to build on its existing infrastructure and leverage its big data and analytics capabilities. The ‘modernization’ part refers to structural changes which will bring more economic and business value to an enterprise.
Organisations will benefit well from a hybrid strategy - a structured manner to identify the kind of data which should be centralised in a data warehouse and the data that should be isolated in lakes using technology. The challenges of upgrading to a hybrid model are mainly technological ones, but are easily manageable as available solutions are mature and stable.
Recently, people started emulating each other’s model as the best practice, forgetting that some problems are peculiar to an individual organisation. Each business has a different requirement. Hence, enterprises must note that one strategy doesn’t fit all and instead, a customised solution serves as a ‘best practice’ for all.
Organizations need to strategize the need of business data and figure out how much insights an enterprise has. Identify critical business entities, and find out what kind of data is critical, what can be in transit, and which data must be stored in the warehouse. For this, one needs to categorise residual data and transient business information.
Regardless of how sophisticated your data warehouse is, it likely needs modernization. Getting your data warehouse modernization strategy right will help your enterprise increase the percentage of data needs being supplied to 90 per cent. Evidently, it is important to modernize a data warehouse for an enterprise to stay forward, competitive and relevant in the industry. It is vital to take the right steps to lead to success.
The author is CEO, Incedo
Disclaimer: This article is published as part of the IDG Contributor Network. The views expressed in this article are solely those of the contributing authors and not of IDG Media and its editor(s).