More and more organizations are increasing the use of data in their decision making. EMA describes this as having a data-driven strategy. Recent research has revealed that almost 63% of respondents included data-driven strategies in their organization at a significant level. EMA has been tracking the growth of big data with annual research since 2012. From our 2015 research, you can see in the chart below how organizations are moving forward with their big data projects.
As big data matures, the number of organizations with a big data project in serious planning or in full operation is increasing while the number of organizations in initial investigation stage decreases. Our 2014 research revealed that more than 55% of respondents reported a big data strategy was either “adopted and essential” to their organization or “adopted and important.” The number of projects within these organizations has also been on the rise, as shown in the chart below.
In 2013, the majority of organizations had one or two active big data projects. In 2014, the majority reported two or three projects, and there was a significant rise in the percentage of organizations with four or more projects. While the number of projects has been increasing, the number of issues and frustrations has been rising as well. The chart below shows the obstacles facing organizations heading down the big data path.
Poor data management was the most mentioned obstacle with 22%. Also significant was the complexity of implementation for multistructured data platforms with 12.7%. Big data is more than just lots of data. The value of big data is that it combines previously unconnected data points into a cohesive view. Complexity results from the variety of data sources that must be sourced, ingested, and combined to achieve this value.
EMA defines data management as “the development and execution of architectures, policies, practices, and procedures in order to manage the information lifecycle needs of an enterprise in an effective manner”. With 22% of respondents struggling with poor data management, it is clear that there is room to improve practices, procedures, and the underlying tools used in managing the information lifecycle.
Automation can often help manage complexity, bring consistency, and reduce costs by making IT staff more efficient. However, IT organizations often struggle to achieve automation. EMA research has shown that a lack of proper tools is often a significant factor in preventing automation from happening more quickly. The chart below shows that 44% of organizations either are struggling with homegrown tools or lack tools for automation altogether.
Workload automation (WLA) tools and other IT and business process automation tools have been expanding their capabilities to address the growth of big data. While Hadoop and the many commercial tools have some built-in scheduling capabilities, they do not come close to matching the features and controls of the WLA solutions. Many of the workloads managed and monitored with these tools relate to data movement. As big data gets bigger and more automation is applied to these workloads, more awareness and control of data and the file transfer processes (and even the ability to make minor manipulations) are included in these WLA tools. Many WLA solutions either are integrated with broader process automation tools or are packaged with companion process automation tools to further address automation in support of data management processes.
As big data continues to mature, further automation will be added to the infrastructure enterprises use to collect the high volumes of data required to achieve a data-driven strategy. Enterprises should focus both on better integration with enterprise applications and on incorporating third-party tools that can mature by learning from a large user base and with the resources available to the software vendor. While homegrown tools can often provide a quick, direct fix, they can be difficult to keep current and mature with limited in-house resources (especially as staff members likely have production support responsibilities that always seem to override infrastructure enhancement work). The scope of third-party tools are also a benefit as they often look beyond specific IT operations tasks to encompass a broader view of business value with automation supporting reports and dashboards that allow non-technical people to access the data.
Many enterprises have discovered the value of a data-driven strategy. Many have also discovered the difficulty of consolidating all of their disparate application data and data from other sources into a useful form. Help is available in the form of process automation tools designed to automate processes across systems and integrate critical business applications to create comprehensive and centralized automation of all these processes. Take advantage of better tools to ensure that you are in the driver’s seat and in control of your data-driven strategy.
