A move from RDBM to NoSQL DB with Big Data capabilities empower the Snapdeal portal to achieve the crucial aspects of scale, speed and real time information
By Abhishek Raval
October 6, 2014—the day when India’s online retail market saw one of the biggest sales and the Indian consumers got the opportunity to buy a range of exciting products at unbelievable prices! Lakhs of products got sold in a single day.
It was an action packed day at the popular online retailer, Snapdeal: a smartphone got sold every 6 seconds; laptop every 20 seconds; tablet every 30 seconds; sari every 30 seconds; footwear every 10 seconds.
With such high sales happening for the first time in the history of India’s online retail, few glitches were expected. Some consumers complained that they could not buy their favourite products because of technical issues. But overall, October 6, 2014, was a highly successful day—it can be seen as a coming of age day for India’s online retail industry.
A November 2013 report from CLSA indicates that e-commerce will grow to as much as $22 billion (excluding travel) in five years from $ 3.1 billion currently. Such high growth can only happen on a secure online platform that uses cutting edge technology to provide real time information to the customers and suppliers.
Building the infrastructure
Snapdeal could service so many customers in a single day because of the innovative technology that the website has deployed. Before the big sales day, the company had moved its systems from Relational Database Management System (RDBM) to a real time In-Memory NoSQL DB, which has Big Data processing capabilities. This new system enabled the Snapdeal website to handle massive amounts of traffic and continue to serve its customers with seamless efficiency.
Initially the website was using 10 MongoDB servers to process the data, but as the number of transactions continued to rise, the servers saw a performance degrade. This resulted in an unsatisfactory buying experience for some of the consumers. Few customers were unable to get the latest price related information.
Apart from the customer based interactions on the e-commerce portal, there is also the activity from sellers, who are listing new products, updating their catalogues, changing the nature of their offerings based on the fluid demand-supply dynamics. Considering that the approximate seller base on Snapdeal is one lakh, it is obvious that the site sees a lot of action from the seller side.
In the database world, these activities on the website are expressed as “Writes” and “Reads”–“Writes” are the changes made in the inventory and pricing management system and “Reads” are the webpage loading time with all the latest changes being made on the site in real time. When Snapdeal started encountering a drag on the website due to the growing business volume, the need was felt to have a re-look at the database model.
Tech for Serving Exponential Growth
As Snapdeal continues to grow exponentially, there is a massive amount of rise in the data from consumers, resellers and other partners in the ecosystem. The website is also prone to sharp spikes in consumer and supplier traffic during the big sales days. Thus there was need to upgrade the systems to ensure that the transactions are conducted without any delay. “Only when the systems fail to cope up that you realise that you need to have Big Data technology and a team in place within the company,” says Ankit Khanna, Senior VP, Product Management, Snapdeal.
The company began to explore for solutions that could help it handle the regular load as well as the sharp spikes in traffic during the holiday seasons. The IT team at Snapdeal was clear from the beginning that they wanted a solution that could provide efficient real-time service to its consumers as well as sellers.
Thus the quest began for a DB. The company looked at solutions like Couchbase, Hazelcast, Cassandra, Redis and few others. After a lot of deliberation and comparative evaluation, Snapdeal finally signed up with Aerospike.
The In-Memory NoSQL DB player was capable of meeting the exact requirements from Snapdeal.
The implementation had to be done quickly as the online retail market in India is growing at an exponential pace. At the time of festivals like Diwali there is sudden explosion in the consumer and supplier traffic. “Snapdeal had the goal of thirty thousand TPS. The system that can reach that level was good enough for them,” says Sunil Sayyaparaju, Tech Lead, Aerospike.
After the incorporation of Aerospike, there has been a significant improvement in the transaction process at Snapdeal. On October 6, 2014, Snapdeal was expecting that there will be a maximum of thirty thousand TPS, but the demand went up to seventy thousand TPS. The good thing is that the system from Aerospike was able to withstand the spike. Thus Snapdeal was able to serve maximum number of consumers on that day.
“Aerospike is helping us to serve real time data in large volume. The Snapdeal applications and the resellers can now make real time updates in inventory, catalogue etc. This in turn is enabling us to serve billions of records to the consumers in real time,” says Khanna.
The other technology that has been implemented at Snapdeal is the recommendation engine. The Big Data element of the engine provides intelligent recommendations to the frequent visitors.
Real time updates
Currently, the Java enabled marketplace platform used by Snapdeal, includes sub-systems for order and catalogue management, inventory and pricing management, fulfilment centre management, shipping, delivery and tracking management and TrustPay, a buyer-seller protection platform.
Snapdeal is now capable of ensuring that its sellers are able to constantly update their products and services. The consumers are having a seamless and efficient buying experience. They are also able to rate the sellers on the basis of price, time for delivery and the overall consumer-satisfaction. After every page click, the website combines the updates from shoppers and sellers to display the most relevant products.