By Romi Mahajan
Nowadays one has to be completely turned off to avoid talk of the revolutionary aspects of what is termed “Big Data.” If Data is intelligence and the ability to collate and connect data is wisdom, then Big Data indeed is omniscience. While the term has been bandied about by marketers and technologists alike, it represents the positivist path that constitutes a “future condition” that is possible by the right investments and algorithms.
It appears we are seeking a Data Nirvana.
Projecting into the future is a fun exercise no doubt but it can obscure the present. And while we don’t have a Data Nirvana in the present, we do have a set of tools that can improve the current situation so greatly that we can approximate what real data agility is.
Take for instance the issue of corporate data. Corporations (in fact organizations of any size) typically have reams and reams of data that could be enormously useful if discovered, deciphered, and deployed. These stores of data represent not only historical intelligence but also a great deal of current and contextual knowledge that could be put to use in sales, marketing, finance, operations; in fact, in any department, this knowledge could be beneficial.
But most “business users” in an organization are not hardcore data archaeologists. They cannot excavate the seams of history or roam forever in the cavernous present. They need data in a useable form in a “pre-decision” time-frame. Put simply they need nicely sculpted and curated data and they need it now.
These data-hungry users are also in some ways fire-walled from the IT department which is the steward of the data sources, data processing, and reporting. They make constant and needy requests from IT, which is itself busy and backlogged. Most IT departments after all are told to “do more with less,” so they demure, suggesting that business users figure out their needs once and upfront. In their framework, this makes sense since they are not privy to the ever changing data needs of the business nor are they capable of pulling a rabbit of out of the IT hat. They are bound by the systems they have.
These constraints don’t seem to deter the business teams from wanting what they want when they want it. Again, this is perfectly consistent with their mandate – to grow and drive the business. If Data is intelligence and data combinations wisdom then surely the teams deserve it. But we know that new data sources are added consistently and existing sources change. So by definition the needs of the business users are dynamic and ever-changing, ideas anathema to IT which is already fatigued and can’t manage to make changes in real-time.
This impasse-a real and clear problem- creates a situation of “data wastage.” Data becomes the tree that fell in the forest and no one heard. It’s present but not optimally. It’s abundant but still thin. It’s updated regularly but shields itself from ready viewing. This hampers the ability of the organization to be agile, to be dynamic, to learn quickly and change. This further hampers innovation which itself creates a stagnation that is hard to get out of.
This scenario- pressing and ever-present–flies into the face of the data nirvana story. The current state does not lead to brilliant outcomes. If the gap between IT and Business persists (and it does, despite the best efforts of both), then there is no agility, no nirvana, no realization of the promise inherent in the marketing brochure.
What then of solutions that in fact do bridge this gap? Clearly, they are placed well to help organizations get to good and then get to great. As the gap is bridged, as the war settles, and as we enter data-peace, we start to see that nirvana might in fact be attainable.