Update as on 21st Dec 11 p.m.
Dear Customers,
In my previous communication
(mentioned below) I had stated the reasons for the issue we faced on 20th Dec.
It was Netmagic's
hybrid cloud leaf switch which went into hung mode and had an impact on storage
switch which resulted in storage cluster being not accessible. This affected 13
nodes out of total 20 nodes of storage component. Though we regained
accessibility around 8.30 a.m. yesterday but the performance for this cluster
remained extremely slow for high resource consuming activities like our
backoffice processes. From yesterday morning till now the performance was very
poor leading to non-updation of margins, trades and holdings. Despite this, we
ran our trading systems to almost 90% efficiency throughout the day and completed
all exchange related processes. As the backoffice was not updated clients had
to face some issues as listed below.
1.
Credit arising out of trading profits and option writing for trades done on 19th
and 20th were not added back to margins.
2.
Stocks bought of T2T segment on 19th were not allowed for selling. Clients had
to call our trading desk to place orders.
3.
Funds Payouts were only processed once a day to avoid any excesses.
4.
Margin pledge done before 9 a.m. today were not added to margins
5.
Trades done for 19th and 20th were visible in Net Position.
6.
100% margin was not released for stocks sold on 20th Dec
7.
Holdings were not updated but “My stocks” and “portfolio” section was updated.
Our technology team along with Netmagic have been working
for past 30 hours to get the system up and running. This involved a new set of
storage clusters shipped, installed and updated in our existing primary datacenter.
This activity will be completed by tomorrow evening i.e 22nd Dec. Alternatively,
we are creating the entire set up in our new data center of Chennai which will
also be ready in next few days.
STATUS AS OF NOW
Today post market hours we have updated all trades, margin
and holdings till 20th Dec. Our post market activities of trade
processing, Early payin, Ledger and margin updation for trades done on 21st Dec i.e. Today is underway and I expect a smooth trading day for all our
clients tomorrow onwards. We will continue to work on strengthening our
infrastructure and build multiple failover mechanisms to avoid such issues to
occur in future.
Lastly, I truly understand the pain and
frustration of customers as any technical incident even for few minutes
disrupts trading opportunities but the hard fact of our reliance on technology
is that there are multiple moving parts in a trading system today like Networks,
switches, databases, servers, gateways, connectivity, etc. Though we try and
build redundancies, failovers, disaster recoveries but still there may be times
when systems don’t work the way it should be. This has not just happened to us or
other brokers, exchanges, depositories but biggest of the big product companies
like whatsapp, facebook, googles of the world. I appreciate your patience and
trust with 5paisa and we will continue to work hard towards making your trading
and investing experience seamless.
Update as on 20th Dec
Dear Customers,
Quite a few of our customers faced issues in trading in the early market hours. Let me explain what exactly happened in today’s unforeseen and unfortunate incident. 5Paisa hosts its entire trading and back-office systems in Netmagic datacenter, part of NTT, a global IT infrastructure company. Netmagic is the largest Data center in the country, with more than 70000 square meters in over 10 locations. NTT serves over 5000 global customers. We have 2 different data centers (Both in Mumbai) in Netmagic, with 3rd under construction in Chennai. Each data center (DC) is equipped to handle twice our existing load and can easily grow to more than 10X. Though the 2nd Data center is a back-up, we use both the data centers in Active – Active mode. In the present scenario DATA CENTER 1 (DC1) takes more than 65% - 70% of our existing load and balance is on Data Center 2 (DC2). We are in process of migrating clients to DC2 so that both data centers take equal load at any given time.
Today morning around 3.50 a.m. one of the Netmagics Hybrid cloud leaf switch went into hung mode which had an impact on storage switch which resulted in storage cluster being not accessible for a few of Netmagic’s customers including us. Netmagic team isolated the faulty hybrid leaf switch post which our network segments of DC2 came up around 7 am. DC1 which is our primary DC with all backoffice processes running on it was still inaccessible. The access to the same was established around 8.20 a.m. so basically all our backoffice processes both EOD post commodity trade processing and BOD could not run from 3.50 a.m. to 8.20 a.m. Normally all our backoffice processes which update ledger, payments, holdings, net positions of derivative segments, etc takes around 60 – 70 mins to get completed. As we just had 40 mins before market started, we could not complete our backoffice processes which had the following impact.
- All trades done on 19th December were visible in the Net position screen.
- Holdings was not updated with transactions of 19th December
- Payouts requested by customers post market hours on 19th December were not processed and payout was not done.
- Customers could not login upto 8.55 a.m.
- On another front, as a part of our project to migrate more customers to our DC2 so that we have equal load, there was a planned activity on 19th Dec night to migrate around 4 lac customers from DC1 to DC2. As the migration happened but backoffice processes could not get completed the customers which got migrated had to face following issues apart from the ones highlighted above.
- Orders placed on 19th December were not seen in order book and net position
- Margins were not getting released in case of square offs for certain clients
- EDIS process for Sell authorization was not working
As the market started we observed login issues and slowness in our DC2 for the first few minutes, impacting almost all the users on DC2. Customers on DC1 were not impacted at all. The slowness on DC2 was resolved by 9.30 a.m. post that logins and orders went through smoothly throughout the day. There were other issues like E-dis for sell authorization, margin mismatch & intraday orders not visible in the order book. Most of these issues were resolved before 12 p.m.
Though today’s incident was because of failure at our DC provider and was not restricted just to us but other companies as well, we are in discussion with Netmagic to provide detailed RCA and also steps to be taken to mitigate such risks in future. We are also setting up our 3rd data center in Chennai which is expected to be ready by feb 2023 (Got delayed due to global chip shortage). This DC will host not just our trading systems but also our backoffice and other affiliated systems. As the DC will be completely independent and in a different geographic and seismic location, it will give us much more redundancy for any kind of failovers. Alternatively, we are working with Netmagic to identify every single point of failures and creating redundancy for the same. As a technology company it is our responsibility to ensure a smooth trading experience to our customers and we are 100% committed to it.
Regards,
Prakarsh Gagdani
CEO – 5Paisa Capital Ltd.