Dear Customers,

Quite a few of our customers faced issues in trading in the early market hours. Let me explain what exactly happened in today’s unforeseen and unfortunate incident. 5Paisa hosts its entire trading and back-office systems in Netmagic datacenter, part of NTT, a global IT infrastructure company. Netmagic is the largest Data center in the country, with more than 70000 square meters in over 10 locations. NTT serves over 5000 global customers. We have 2 different data centers (Both in Mumbai) in Netmagic, with the 3rd under construction in Chennai. Each data center (DC) is equipped to handle twice our existing load and can easily grow to more than 10X. Though the 2nd Data center is a back-up, we use both the data centers in Active – Active mode. In the present scenario DATA CENTER 1 (DC1) takes more than 65% - 70% of our existing load and balance is on Data Center 2 (DC2). We are in process of migrating clients to DC2 so that both data centers take equal load at any given time.

Today morning around 3.50 a.m. one of the Netmagics Hybrid cloud leaf switch went into hung mode which had an impact on storage switch which resulted in storage cluster being not accessible for a few of Netmagic’s customers including us. Netmagic team isolated the faulty hybrid leaf switch post which our network segments of DC2 came up around 7 am. DC1 which is our primary DC with all backoffice processes running on it was still inaccessible. The access to the same was established around 8.20 a.m. so basically all our backoffice processes both EOD post commodity trade processing and BOD could not run from 3.50 a.m. to 8.20 a.m. Normally all our backoffice processes which update ledger, payments, holdings, net positions of derivative segments, etc takes around 60 – 70 mins to get completed. As we just had 40 mins before market started, we could not complete our backoffice processes which had the following impact.

On another front, as a part of our project to migrate more customers to our DC2 so that we have equal load, there was a planned activity on 19th Dec night to migrate around 4 lac customers from DC1 to DC2. As the migration happened but backoffice processes could not get completed the customers which got migrated had to face following issues apart from the ones highlighted above.

As the market started we observed login issues and slowness in our DC2 for the first few minutes, impacting almost all the users on DC2. Customers on DC1 were not impacted at all. The slowness on DC2 was resolved by 9.30 a.m. post that logins and orders went through smoothly throughout the day. There were other issues like E-dis for sell authorization, margin mismatch & intraday orders not visible in the order book. Most of these issues were resolved before 12 p.m.

Though today’s incident was because of failure at our DC provider and was not restricted just to us but other companies as well, we are in discussion with Netmagic to provide detailed RCA and also steps to be taken to mitigate such risks in future. We are also setting up our 3rd data center in Chennai which is expected to be ready by feb 2023 (Got delayed due to global chip shortage). This DC will host not just our trading systems but also our backoffice and other affiliated systems. As the DC will be completely independent and in a different geographic and seismic location, it will give us much more redundancy for any kind of failovers. Alternatively, we are working with Netmagic to identify every single point of failures and creating redundancy for the same. As a technology company it is our responsibility to ensure a smooth trading experience to our customers and we are 100% committed to it.    

 

Regards

Prakarsh Gagdani

CEO – 5Paisa Capital Ltd.