By Gil Press. This article originally appeared on Forbes, September 17, 2015.
Results of the largest worldwide survey to date of Hadoop users were released today by AtScale, which was assisted in this project by Cloudera, MapR and Tableau. The answers from more than 2,100 participants provide insights into the current and future use of Hadoop and where and how it brings business value to the companies investing in this foundational big data technology.
“Enterprises today have reached a new stage where they are worried about providing access to Hadoop to business users,” says Bruno Aziza, AtScale’s Chief Marketing Officer. “Self-service is the next big thing for Hadoop and the companies that understand how to deliver on the promise of democratizing access to the Data Lake will win.”
So what’s up with Hadoop?
Focusing on revenues, providing self-service, and an executive mandate are key to getting value from Hadoop
Companies that deploy Hadoop to drive revenues are about 30% more likely to achieve value from Hadoop than those who are looking to cut costs. Those that provide business users with self-service access to Hadoop are nearly 50% more likely to realize a tangible value. The presence of an “executive mandate” (i.e., telling people they are going to do Hadoop rather than just “sponsoring” it) boosts an enterprise’s chances of deriving value from Hadoop by 20%.
The more you invest in Hadoop, the likelier you are to get value out of it
72% of the enterprises surveyed that have deployed over 500 nodes are getting value out of Hadoop, but only 33% of those with up to 10 nodes say they got value out of their investment. 49% of all survey participants say they are already deriving tangible value from Hadoop and 94% think they will continue or start to get value from Hadoop.
Hadoop is moving to the mainstream of enterprise IT
In its early days, Hadoop was talked about as “replacement for tape,” a better-performing data archiving solution. As its capabilities and ecosystem have evolved, it is now maturing into a mainstream enterprise data management tool.
76% of current Hadoop users plan on doing more with it over the next 3 months and almost half of those who haven’t yet deployed Hadoop say they will do so in the next 12 months. While 26% describe Hadoop as “experimental,” 47% regard it as “strategic” and 15% say it is “game changing.”
Business intelligence is rising again
When big data and data science took over the airwaves five years ago, Hadoop rose as a technology of choice for two specialized workloads: The traditional Extract-Transform-Load (ETL, the process by which data is taken from different sources, formatted, and uploaded to a data warehouse or database) for the folks in IT dealing with unstructured data, and the new data science process used by the new breed of data scientists to find new insights in the data, somewhat overshadowing the business analysts performing traditional business intelligence tasks.
This is reflected in where Hadoop has been deployed so far. 74% of the companies participating in the survey that have already deployed Hadoop are using it for ETL, 62% for data science and 65% for business intelligence.
In another indication of the mainstreaming of Hadoop, however, business intelligence may rise again to take advantage of the new new thing. The picture is somewhat different for those that have not yet deployed Hadoop, with 69% intending to use it for business intelligence, 56% for data science and only 51% for ETL.
Online companies are Hadoop-savvy, financial firms not so much
As could be expected, the companies that make their living on that vast repository of unstructured data, the World Wide Web, are ahead of other sectors in Hadoop maturity. Surprisingly, the financial services sector which is typically the first to invest in and learn new enterprise IT tools and applications did not do well in terms of Hadoop maturity in this survey. It may well be that financial services companies hold themselves to very high standards or that the survey captured many small financial services firms that do not have the same resources as their larger brethren.
As ones and zeros are eating the world and enterprises worldwide are analyzing private and public data to find new sources of revenues and new ways to streamline operations, time-to-access (for data) and time-to-value (for analysis) are shrinking rapidly. Hadoop and similar big data tools are graduating into an enterprise IT environment where the underlying technologies merge to collectively provide more data and better analytical tools to more employees.