Harnessing the Data Explosion with Software Defined Storage

If data is the new oil, then there is a massive shortage of resources to manage it.

The scope of digital services extends to everyone from the CEO of a Fortune 500 company to a small business in a remote corner of the world. According to analyst firm IDC, the total worldwide data is expected to grow to 175 zettabytes (175,000,000,000 gigabytes) by 2025 — 10 times more than what we have today.

The end result is not happy for any modern-day enterprise: it creates multiple complexities for data centre operations, as massive demand for data storage surges across the organization. Adding to this, the majority of data being generated from the huge range of new sources such as machine learning, video streams, and the Internet of Things is unstructured data, where the immediate business relevance is not necessarily obvious, but still needs to be stored for later reference and cross-reference. All these factors need to be considered when deciding on the best strategy for proper data storage and protection.

Needless to say, there is no straightforward solution, but leading organizations and eminent analyst firms like 451 Research suggest that Software Defined Storage (SDS) might just be the solution to the storage dilemma. Modern SDS solutions such as the open source Ceph project are ideal for addressing exponential growth of data when organizations are faced with expensive proprietary storage hardware. Ceph is massively scalable, enabling organizations to automatically and cost-effectively grow from terabytes to exabytes and beyond. It is self-healing and self-managing with no single point of failure. It is designed as a horizontal scale-out system which means that performance and throughput capacity increases as storage capacity is added into a cluster of servers. This makes the system almost infinitely scalable without the introduction of bottlenecks that can be seen in traditional “head-and-shelves” storage appliances.

It is important to differentiate between SDS and hyper converged infrastructure (HCI). HCI provides single ‘building blocks’ for growing a cluster of compute, network and storage capacity at the same time, which make sense when the usage models for all three are roughly equivalent, but does lock the enterprise in to a single hardware supplier for any expansion, putting them in a similar position as when expanding traditional storage systems.  SDS, by contrast, is a software-only approach which provides a layer of functionality that can be used to build a massively scaled storage enterprise storage system using commodity server hardware from different manufacturers with no lock-in. When facing the data explosion, having the flexibility to find the best-value hardware at a given point of time is vital to contain costs.

Moreover, organizations can configure SDS based on their storage requirements and policies, and then let automated management processes help achieve more with the same resources. According to a report by 451 Research, 43% of organizations decided to choose SDS because it simplifies data provisioning and 39% of organizations think it lowers their acquisition costs when compared with traditional storage appliances.

Irrespective of whether the storage software layer is open source or proprietary, it’s important to have an SDS strategy to tackle long term data explosion and to better support next generation Mode-2 application workloads. But it’s important to consider that open source based SDS will empower enterprises to pick the most suitable hardware, operating system, and support.

An open source solution can enable tremendous advantages in economics versus traditional proprietary hardware centric approaches, along with greater flexibility in management, and more efficiency in operations. As we continue to see organizations take a multimodal approach to IT infrastructure – enjoying the advantages offered by both the cloud and on-premise data centres – software-defined storage will become a strategic piece of the puzzle, enabling smarter, less complicated, and scalable yet cost effective solutions.

Authored by Peter Lees, Chief Technologist, APAC & Japan, SUSE