- Case Study
- Critical Power Incident
Critical Power Incident
Case StudyBackground
In the early hours of a Sunday Morning, the Facilities Management department at Sudlows received an urgent call from the data centre team at one of the UK’s leading major retailers.
A major incident had occurred at their primary data centre facility, resulting in a total loss of all backup power to a number of their systems. To compound this, all major corridors and local routes in the surrounding area of the city centre facility were subject to rolling road closures in order to accommodate a cycling event with an estimated 20,000 participants and spectators attending.
The Brief
What follows is a timeline record of the event and details of the rapid response that Sudlows critical power team put into action for the emergency replacement of the failed UPS system. All undertaken at a facility that Sudlows had no prior visibility of and little hope of immediate access to.
From the initial emergency call, all the way through to a complete system restoration, this highly critical project was all successfully achieved in a very impressive 57 hours. The new critical power system installed and commissioned by Sudlows now comprises of 4 x 100kVA UPS in a 300kVA (N+1) configuration.
Timeline
Sunday
09:30: First call received by Sudlows
09.45: Road closures established, alternate route identified with local council collaboration.
10:15: Sudlows site attendance.
10:30: Engineering solutions co-ordinated.
11:00: Replacement equipment sourced (on a Sunday and during the Olympics when spares are scarce).
12:30: First materials delivered to site.
13:00: Preliminary orders placed.
14:00: Removal of the existing UPS system commenced.
Monday
10:00: Sudlows engineers on site to assist in removal of equipment.
11:00: Sudlows engineers source all final materials.
12:00: Confirmation equipment is being dispatched.
16:00: New UPS system on site.
18:00: UPS systems positioned.
20:00: UPS and battery build 50% complete.
22:00: Comms sent to data centre team to confirm works progress.
Tuesday
07:30: Sudlows engineers resume installation works.
08:30: Second consignment of materials delivered.
10:00: Remaining battery systems built.
12:00: Initial electrical testing started.
16:00: UPS pre-commissioning checks undertaken.
17:30: UPS final commissioning.
17:45: Sudlows electrical testing resumed & completed.
18:15: Client training and inspections.
18:30: UPS system put into service and with no interruption to service.
19:45: Retailers IS team successfully transfers the business system back online.
The Conclusion
The retailer’s business systems were restored 45 minutes earlier than planned by the IS team, a great result with 100% from everyone involved.
Without the dedication from The Client Team, Sudlows Project Team and the hard work of the on-site Sudlows Engineers, all of whom were supported by the efforts of key supply chain partners. This could quickly have become a complicated situation with, ultimately, the clients business exposed to risk for a significant period.
The success of this project highlights the need for all organisations to regularly test their own critical power arrangements and demonstrates the true benefits of taking out a facilities management contact with embedded contingency power arrangements to BS25999 with a data centre specialist such as Sudlows.
Testimonial
“Firstly, a simple MASSIVE, thanks for all your help. An excellent example of a great team and a great strategic partner for our Group. I agree, it’s a record and a testimony to your team’s professionalism! Please pass on our sincere thanks for their efforts, which were definitely in the ‘Over and above the call of duty’ territory.
-Head of Servers, Storage and Desktops, Group IT Services
“Wholeheartedly agree, a true measure of what can be achieved with the level of commitment and attitude demonstrated by all parties in turning around what was a grave situation. Thank you all.
“We made the right calls and decisions with professional technical advice from highly respected supply chain partners, from the crucial enabling works before the raw power restoration, to selecting a buy policy instead of a panic influenced hired-set, with all of its locked in costs and unknown quantities. Thanks again to everyone.”
-Data Centre Manager, Group IT Services
“I wholly support my IT colleagues comments, a phenomenal achievement by any standards and raising the bar for our other suppliers, thanks again all.”
-Technical Services Manager, Workplace Services
Andy Hirst, Managing Director of Critical Infrastructures at Sudlows concluded;
“Sudlows presented a design that provides our University with a cost effective and innovative solution. Our commitment to sustainability objectives is demonstrated by the new data centre’s target PUE of between 1.4 and 1.2. This is a substantial improvement to the power efficiency in our old data centre.”
Location
North West, UK
Value
£120,000
Incident
UPS Replacement
Find out how we can release the full potential of your building and communications infrastructure.
