Unravel Data co-founders Dr. Shivnath Babu and Kunal Agarwal talk about the role of observability in resolving IT infrastructure performance challenges

Dr Shivnath Babu, CTO & Co-Founder of Unravel Data and Kunal Agarwal, CEO & Co-Founder of Unravel Data are experts in the field of data observability. Their company, Unravel Data provides comprehensive observability solutions for modern enterprises’ complex IT infrastructures, including cloud-native, multi-cloud, and hybrid environments.

Unravel Data uses AI and machine learning algorithms to analyze millions of details in context, identify performance issues, pinpoint root causes, predict risks, and provide prescriptive recommendations. In this interview, Dr. Shivnath Babu and Kunal Agarwal discuss the key challenges of IT infrastructure performance, how observability platforms can resolve them, the differences between observability and monitoring, and the market acceptance for observability tools in India

Some edited excerpts:

What are some of the key challenges with respect to application and IT infrastructure performance and how can observability platforms resolve them?

Modern enterprises are heavily digitalized and increasingly data-driven. Their IT infrastructure tends to be a complex mix of cloud-native, multi-cloud and hybrid environments, requiring their information and data teams to optimize performance on both legacy and new systems.

Modern applications are increasingly powered by data. When application performance hits a roadblock, data engineers must resolve issues fast to minimize business impact. The data and operations teams struggle to get the right visibility from legacy application performance monitoring tools that were not designed for modern data pipelines, and as a result spend way too much time firefighting manually. Something like 70-75% of their time is spent tracking down and resolving problems through manual detective work and a lot of trial and error. And with 20x more people creating data applications than fixing them when something goes wrong, the backlog of trouble tickets gets longer, SLAs get missed, friction among teams creeps in, and then finger-pointing and blame game begins.

This is where data observability fits in. Modern data observability platforms like Unravel are designed to help enterprise IT and data teams stay on top of their data pipeline performance, cost and quality. Many of these observability platforms go far beyond simple data observability – referring to data profiling and quality monitoring – and cover other aspects of observability too. These include application and pipeline performance, operational observability into how the entire platform or system is running end-to-end, and business observability aspects such as ROI and—most significantly—FinOps insights to govern and control escalating cloud data costs.

Finally, armed with holistic data observability platforms like Unravel, software engineers and architects can design, build, and deploy optimal infrastructure to reduce the risk of bottlenecks using AI-driven actionable insights.

How is observability different from monitoring?

Traditional application performance monitoring (APM) and observability tools are great for diagnosing web applications. But APM and observability tools were never designed to troubleshoot or optimize performance for data applications/pipelines. While they tend to seem alike and they leverage telemetry data such as logs, metrics, and traces, in reality, data observability goes far beyond monitoring. Data observability tools like Unravel gather deep metadata about data jobs, pipelines, clusters, partitions, and users, then correlate those details across your data stack, source code, and infrastructure to create a holistic view of how everything works together.

Monitoring tools have traditionally used telemetry data to detect and solve the challenges of web applications. As applications and data pipelines become more and more complex, they can hardly be managed effectively with legacy tools and methods. Tracking a data system’s overall health and detecting the known failures with monitoring is not enough today, as it cannot provide sufficient insight to resolve them.

While monitoring tools help businesses detect if something is wrong, observability enables them to get down to why the issue exists and how to resolve it. Data observability is designed to not only show data application/pipeline performance, cost, and quality, but also provide data teams with precise, prescriptive fixes so they can quickly and efficiently solve problems and get on to the real business of putting their data to work.

How can observability be used to identify and diagnose performance issues?

Observability is increasingly critical for modern businesses as it helps them improve the stability and agility of their data- and analytics-driven applications. The complex and distributed nature of modern data architectures makes it increasingly difficult to understand the behaviour of data pipelines and data applications.

As a result, troubleshooting and resolving performance issues is based on trial-and-error and is very time consuming. Roughly 70-75% of data engineers’ time is spent tracking down and resolving problems through manual detective work.

Data observability on the other hand provides the necessary visibility into data flows, dependencies, and performance metrics, allowing data teams to identify issues and optimise their data pipelines. The right data observability platform can even help people developing data applications/pipelines manage performance themselves via self-service capabilities, reducing the burden on data engineers. Further, AI and machine learning algorithms built inside the platform can analyze millions of details in context to detect anomalies and identify performance problems, pinpoint root causes, predict risks, and automatically provide prescriptive recommendations.

What is the market acceptance for observability tools in India? What are some of the opportunities you see for your firm in this space?

Much like their global counterparts, Indian businesses both large and small are rapidly digitalising, and the country is already one of the largest markets for public cloud today. Industry body Nasscom had projected in 2021 that India’s Cloud market would reach US$5.6bn in 2022. Research firm IDC has further projected that the overall India public cloud services market will grow at a CAGR of 23.1% to reach $13.0 billion by 2026. These figures demonstrate the growing cloud adoption as well as escalating spending on cloud services by Indian businesses.

Now, cloud bills are usually an organization’s biggest IT expense, and the sheer massive size of data workloads is driving most of the cloud bill. In the US, monthly cloud budget overruns of 40% or more are common, and many companies are even exhausting their three-year data cloud budget in just 12 to 18 months. In India too, many large businesses are already realising the pressures to regulate and control their cloud expenses and here, data observability in conjunction with FinOps, a cloud financial management practice, is the ideal solution to achieve that.

Fundamentally, as Indian businesses across industries evolve and use data as a central fulcrum of their business and operational models, they will need modern data observability tools to effectively and efficiently manage the performance, cost, and quality of their data pipelines – whether on-cloud or on-premises. Notably, banking, financial services, and insurance (BFSI), healthcare, and e-commerce industries are far ahead in their digitalisation journeys and are prime candidates for using DataOps and FinOps (DataFinOps) observability to extract full value from their data stacks. Unravel Data can help companies in these industries in optimising their data operations, reducing their cloud costs, improving application performance, and most importantly help them grow their top lines and bottom lines.