Update regarding outage on 15 June
DNB’s online services were down for several hours on 15 June.
“We apologise to our customers for any inconvenience caused by the outage. Our focus during the weekend has been to establish the root cause and take all necessary actions to reduce risk and stabilise our systems,” says Liv Fiksdahl, Group EVP IT services.
Our preliminary findings indicate that the problem originated in DNB’s central storage unit. The storage system which caused the outage is delivered by Dell EMC. The preliminary report concludes that the root cause was a human error made as part of hypercare operations performed by Dell EMC staff, one of HCL’s sub-vendors.
The failure resulted in HCL, DNB’s IT partner, having to restart several thousand servers. This is a complex task requiring the management of dependencies and sequencing between multiple different applications. This took considerable time, but was managed well without further incident.
Dell EMC is one of the world’s primary technology providers with more than half of the Global 500 companies using its technology to deliver highly available technology systems.
DNB has high availability systems that are designed to kick in and ensure service continuity in the event of failures. However, the failure on 15 June affected the key components that ensure such high availability, hence the disruption to services. DNB and HCL are taking all actions necessary to reduce risk related to this issue. The same component also caused the lack of high availability on 16 March.
Dell EMC had provided a bug-fix to eliminate the risk that caused the outage on 16 March, and this was successfully implemented on 17 June. This upgrade activity was assessed to be non-disruptive and not risky to run in parallel with DNB’s Internet bank running in production. Unfortunately, there was some instability Saturday evening as a result of the fix.
“The inconvenience caused to DNB customers, both in March and last week, is highly regrettable,” says Rajeev Nanda, EMEA Head of Infrastructure Services, HCL.
“It is important for DNB to be transparent in connection with system failures, whether the problem is attributable to DNB or our sub-vendors. However, DNB takes full responsibility for the outage,“ says Liv Fiksdahl.
This whole system is under continuous monitoring and HCL is working with DNB and the technology partners to ensure system stability.
Contact:
Vidar Korsberg Dalsbø
+47 993 80 389