Strasbourg datacentre: latest information

Roubaix, France | March 18, 2021

Strasbourg datacentre: latest information

Strasbourg datacentre: latest information

Roubaix, France | 18 March 2021 - 16:30 (CET)

Customer information:

1. Service Status

  • In the OVHcloud Control Panel, each affected customer can view the status of their services located in Strasbourg (SBG) - as detailed in the FAQ.
  • The OVHcloud Community platform is now available to interact with community members and support you.
  • By following the security protocol defined for on-premises operations: the restart will be carried out room by room, aisle by aisle and rack by rack.
  • Procedures for restarting services will be communicated to affected customers as details become available in the coming days.
  • Real-time monitoring of the restart of SBG by room and planned rack is available at: status.ovh.com/vms/sbg.

2. Commercial actions

  • At the beginning of the week, all customers concerned received a communication on the commercial measures that will be applied to their situation. For customers who have not received the email, a procedure has been indicated in the FAQ.

3. Plan of action

Priority 1: restoring services to SBG1, SBG3 and SBG4

  • Status summary for each datacentre

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

Datacentre operational.

 

Accomplished

Accomplished

First racks tested with success.

Handover to the customer today with visibility of the status in their OVHcloud Customer Control Panel.

SBG-2

Building non-operational.

Site secured.

 

 

Replacement of infrastructure in other datacentres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Datacentre operational.

 

Servers undamaged.

 

 

 

Accomplished

 

Accomplished

Test today on one rack and if conclusive provisional estimate for Friday 19 March for a progressive restart of the services.

SBG-4

Datacentre operational.

 

All floors and servers undamaged.

Accomplished

 

 

Accomplished

 

Provisional ETA: Monday, 22 March for gradual restart of services.

 

  • 136 people are mobilised on premises today
    • The team rotation schedule for the coming weekend has been established.
  • Cleaning of site:
    • Cleaning teams are active in SBG1 and SBG3 in all affected areas.
    • Some of the servers in SBG3 require specific cleaning prior to their return to service. We have started this process.

Priority 2: Provisioning of infrastructure for our customers in other datacentres

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), Londres (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.
  • Update on quantities delivered since 10th March:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

4519

 

NAS HA

TB

246

Public Cloud

VPS

VM

12353

Hosted Private Cloud

Host

Servers

485

 

Datastore

Number of zpool

522

 

 

Volume of TB

3067

  • 400 servers will be produced today.

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.

On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres. 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have set up a support plan for customers who have not yet activated their DRP.

Communications

  • The OVHcloud Community platform is now available to interact with community members and support you.
  • An FAQ is available here.
  • To follow updates about this situation in real time:
    • Our main communication tool remains "Task Status"
    • Status videos are available here:

- FR : https://twitter.com/olesovhcom/status/1371908744775360518

- EN : https://twitter.com/olesovhcom/status/1371934994751500288

Our mission remains to provide our customers with the highest quality of service, to support their online business.  and we know how important this is to them. We sincerely apologise any inconveniences related  this situation An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

------------

17 March 2021 - 16:30 (CET)

Customer information:

1. Service Status

  • Octave Klaba, founder of OVHcloud, has released a new video on the situation in Strasbourg (SBG).
  • In the Control Panel, each affected customer can view the status of their services located in Strasbourg (SBG) - as detailed in the FAQ.
  • The OVHcloud Community platform is now available to interact with community members and support you.

By following the security protocol defined for on-premises operations: restarting will be carried out room by room, aisle by aisle and rack by rack.

  • Procedures for restarting services will be communicated to affected customers as details become available in the coming days.
  • Real-time monitoring of SBG restart by room and planned rack via: status.ovh.com/vms.
  • Progressive updates to the status page for services and backups in Strasbourg (SBG), from today onwards, accessible via the FAQ.
  • Updating and tracking the percentage of services restarted per datacentre scheduled in the “Task Status” interface in the coming days.
  • A new order form has been set up (for Bare Metal Cloud) to offer concerned customers services similar to those they had at the Strasbourg (SBG) site.

2. Commercial actions

  • All concerned customers received, at the beginning of the week, a communication on the commercial measures that will be applied to their situation. For customers who have not received the email, a procedure has been indicated in the FAQ.
  • For each customer concerned:
    • A billing suspension notification for the month of March has been added in the Customer Control Panel.
    • Commercial actions have been indicated, the details of which will be communicated this week.

3. Plan of action

Priority 1: restoring services to SBG1, SBG3 and SBG4

  • Status summary for each datacentre

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

4 of 12 rooms were damaged

 

- Rooms 61AD/62AD: servers not affected

- Rooms 61E/62D: cleaning of servers underway

- Restored power to datacentre

 

- Backbone: reconnected

- Redeployment of the local network (Cabling finalised this Wednesday) and configuration today

 

 

A rack was tested today and if conclusive a provisional estimate of Friday, 19 March for a gradual restart of services after cleaning and drying servers.

 

SBG-2

Building out of use Site secured

 

 

Replacement of infrastructure in other data centres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Servers undamaged

 

 

 

- Restored power to datacentre

 

 

- New network power restored

- Redeployment of the local network this Wednesday 17th March (Cabling finalised).

 

Test tomorrow on a rack and if conclusive a provisional estimate of Friday, 19 March for a gradual restart of services.

SBG-4

All floors and servers were spared

- Restored power to datacentre

 

Redeployment of the local network carried out.

Provisional ETA: Monday, 22 March for gradual restart of services.

 

  • 116 people are mobilised at the premises today
  • Network redeployment: 
    • The SBG1, SBG3 and SBG4 datacentres will be connected to the network tonight.
  • Watercooling:
    • The entire system is operational in the SBG1/SBG3/SBG4 datacentres.
  • Cleaning of site:
    • Intensive mobilisation (24/24).
    • SBG1: finalisation of cleaning Wednesday, 17th March.

Priority 2: providing infrastructures in other datacentres for our affected customers

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), Londres (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks. Update on quantities delivered since 10th March:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

4200

 

NAS HA

TB

276

Public Cloud

VPS

VM

11998

Hosted Private Cloud

Host

Servers

434

 

Datastore

Number of zpool

466

 

 

Volume of TB

2950

 

  • 400 servers will be produced today.

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.

On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres. 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have set up a support plan for customers who have not yet activated their DRP.

Communications

- FR: https://twitter.com/olesovhcom/status/1370045226929299461

- EN: https://twitter.com/olesovhcom/status/1370045891806175232

Our mission remains to provide our customers with the highest quality of service, to support their online business.  and we know how important this is to them. We sincerely apologise any inconveniences related  this situation An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

-----------

16 March 2021 - 16:00 (CET)

Customer information:

1.  Service Status

  • Progressive updates to the status page for services and backups in Strasbourg (SBG), from today onwards, accessible via the FAQ.
  • Monitoring of the % of service restarts per datacentre to be displayed in the "Travaux" interface in the coming days. Restarts will be carried out room by room, aisle by aisle and rack by rack.
  • Finalized procedures for restarting services will soon be communicated to customers.

2. Commercial actions

  • For each customer concerned:
    • A billing suspension notification for the month of March has been added in the Customer Control Panel.
    • Commercial actions have been indicated, the details of which will be communicated this week.

3. Plan of action

Priority 1: restoring services to SBG1, SBG3 and SBG4

  • Status summary for each datacentre

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

4 of 12 rooms were damaged

 

- Rooms 61AD/62AD: servers not affected

- Rooms 61E/62D: cleaning of servers underway

- Power restored

- Damaged UPS have been checked and are being replaced

- Backbone: temporary connection to SBG on 15th March

- LAN redeployment and configuration today

 

 

Provisional ETA: Monday, 22 March for gradual restart

SBG-2

Building out of use. Structural expertise carried out. Site secured.

 

 

 

 

Infrastructure replacement in other data centres Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Servers undamaged

 

 

- Power restored today

- All inverters are operational

 

 

- New network room powered up today

- Redeployment of the local network on 17th March

Provisional ETA: Monday, 22 March for gradual restart

 

 

SBG-4

All floors and servers were spared

Power restored today

 

Redeployment of the local network today

Provisional ETA: Monday, 22 March for gradual restart

 

  • 101 people are mobilised and on premises
  • Network redeployment:
    • SBG connection to the backbone today (alternative routes to Paris and Frankfurt underway)
    • Reconnection of buildings and rooms:
      • Fibre optic redeployment in progress 
      • Configuration underway
  • Watercooling:
    • The entire system is operational.
  • Cleaning of site:
    • Intensive mobilisation (24/24).

Priority 2: Provisioning of infrastructure for our customers in other datacentres

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres: Gravelines (GRA), Roubaix (RBX), Londres (LON), Warsaw (WAW), Francfort (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks. Update on quantities delivered since 10th March:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

3939

 

NAS HA

TB

242

Public Cloud

VPS

VM

10731

Hosted Private Cloud

Host

Servers

372

 

Datastore

Number of zpool

393

 

 

Volume of TB

2704

 

  • 350 servers will be produced today.

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have set up a support plan for customers who have not yet activated their DRP.

Communications

- FR: https://twitter.com/olesovhcom/status/1370045226929299461

- EN: https://twitter.com/olesovhcom/status/1370045891806175232

Our mission remains to provide our customers with the highest quality of service, to support their online business and we know how important this is to them. We sincerely apologise for any inconveniences related to this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

-------

15 March 2021 - 17:00 (CET)

Customer information:

1 - Server status

  • The status page for services and backups in Strasbourg (SBG) is now available in the FAQ.
  • For VPS service, a question about “zone vs datacentre (SBG)” has been added for clarification in the FAQ.

2 - Commercial actions

  • For each customer concerned:
    • A billing suspension notification for the month of March has been added in the Customer Control Panel.
    • Commercial actions have been indicated, the details of which will be communicated this week.

3 - Action plan

Priority 1: restoring services to SBG1, SBG3 and SBG4

  • Status summary for each datacentre

 

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

4 of 12 rooms were damaged

Temporarily repowered on 13th March and will be restored permanently on 16th March

- Backbone: temporary connection to SBG on 15th March

- Internal network to be redeployed on 16th March

Provisional ETA: Monday, 22 March

for gradual restart

SBG-2

Building out of use

Structural expertise carried out.  Site secured

 

 

Replacement of infrastructure in other datacentres

SBG-3

Servers undamaged

Temporarily repowered on 12th March and will be restored permanently on 16th March

- New network room to be deployed and powered on 16th March

- Internal network to be redeployed on 17th March

Provisional ETA: Monday, 22 March

for gradual restart

SBG-4

Servers undamaged

Power to be restored on 16th March

- Internal network to be redeployed on 16th March

Provisional ETA: Monday, 22 March

for gradual restart

 

  • 111 people are dedicated on-site to connecting network and electricity units (in rotation, 24/24), for: 
  • Power redeployment:
    • The electrical installations required to restart SBG1/SBG4 have been verified.
    • The high-voltage power supply for SBG3 is deployed and operational.
    • Inverter re-commissioning protocol underway today (verification of all UPS).

 

  • Network redeployment:
    • The SBG1 network room is being tested.
    • The new network room is being completed and will be supplied with electricity as soon as the inverters are verified.
    • The installation of the network equipment will be finalised this evening.
    • The connection of the backbone is scheduled for tomorrow.
    • The local network is being redeployed.

 

  • The cleaning of any soot in the SBG3 server rooms will begin on Tuesday, 16 March. The clean-up teams have arrived on site and are beginning deployed.

 

Priority 2: providing infrastructures in other datacentres for our affected customers

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our Roubaix (RBX) and Gravelines (GRA) datacentres.
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.
  • 500 servers will be produced today.

 

  • Update on quantities delivered since 10th March:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

3658

 

HA-NAS

TB

220

Public Cloud

VPS

VM

8903

Hosted Private Cloud

Host

Servers

328

 

Datastore

Number of zpool

361

 

 

Volume of TB

2598

 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers, and have set up a support plan for customers who have not yet activated their DRP.

Communications

- FR: https://twitter.com/olesovhcom/status/1370045226929299461

- EN: https://twitter.com/olesovhcom/status/1370045891806175232

 

Our mission remains to provide our customers with the highest quality of services to support their online activities, and we know how important this is to them. We sincerely apologise for the problems caused by this incident. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

-----------

14 March 2021 - 15:00 (CET)
 

Customer information:

1 - Server status

  • A page has been created listing the backups of services hosted in Strasbourg (SBG) managed by the customer or by OVHcloud along with their status, which will be available in the FAQ within the next few hours.
  • For Public Cloud and VPS customers, a question about “zone vs datacentre (SBG)” has been clarified in the FAQ today.

2 - Commercial actions

  • All affected customers will be notified about their suspended March bill in their Control Panel.
  • Details on how these measures will be implemented will be sent next week, and will also apply to March bills for impacted services that were already issued before 10th March.

3 - Action plan:

Priority 1: restoring services to SBG1, SBG3 and SBG4

  • Status summary for each datacentre

 

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

4 of 12 rooms were damaged

Temporarily repowered on 13th March and will be restored permanently on 17th March

- Backbone: temporary connection to SBG on 15th March

- Internal network to be redeployed on 17th March

Provisional ETA: Monday, 22 March

for gradual restart

SBG-2

Building out of use

 

 

Replacing infrastructures in other datacentres

SBG-3

Servers undamaged

Temporarily repowered on 12th March and will be restored permanently on 16th March

- New network room to be deployed and powered on 16th March

- Internal network to be redeployed on 17th March

Provisional ETA: Monday, 22 March

for gradual restart

SBG-4

Servers undamaged

Power will return permanently on 17th March

Internal network to be redeployed on 17th March

Provisional ETA: Monday, 22 March

for gradual restart

 

  • A team of 60 people is working on site to connect mobile network and electricity units (working in rotation, 24 hours a day)
    • Cabling in progress.
    • Connection planned for tonight.
  • The drying out of the SBG3 buildings will be finished tonight.
  • The cleaning of any soot in the SBG3 server rooms will begin on Tuesday 16th March.
  • The cooling system is operational :
    • Water cooling is protected from the cold weather (forecast locally on Thursday, 18 March and Friday, 19 March).
    • The pumps are operational.
  • Priority 2: providing infrastructures in other datacentres for our affected customers
     
  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our Roubaix (RBX) and Gravelines (GRA) datacentres.
  • We are committed to providing around 15,000 new servers in the coming weeks.
  • Update on quantities delivered since 10th March:

 

Statement

 

  • Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers
    • Our Support teams and Technical Account Managers are in contact with our customers, and have set up a support plan for customers who have not yet activated their DRP.

Communications

- FR: https://twitter.com/olesovhcom/status/1370045226929299461

- EN: https://twitter.com/olesovhcom/status/1370045891806175232

 

Our mission remains to provide our customers with the highest quality of services to support their online activities, and we know how important this is to them. We sincerely apologise for the problems caused by this fire. We will continue to communicate with the utmost transparency about the cause of the fire and its consequences.

An investigation is ongoing. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

-------

Roubaix, France | 13 March 2021 - 16:00 (CET)
 

Customer information:

1 - Server Status

  • We have sent emails to all our customers informing them of the status of their services.

2 - Commercial actions

  • Billing has been suspended for Strasbourg (SBG) datacentre customers from the date of the incident.
  • Commercial measures have been set out to each affected customer.
  • Details on how these measures will be implemented will be sent next week, and will apply in particular to March bills for impacted services that were already issued before 10th March.
  • We have updated the FAQs.

 Our three priorities are as follows:

  • Priority 1: restoring services to SBG1, SBG3, and SBG4
  • Priority 2: providing infrastructures in other datacentres for our affected customers
  • Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

Action plan:

Priority #1:

  • Status summary for each datacentre

Datacentre

Situation

Electrical restart

Server restart

SBG-1

4 of 12 rooms were damaged

From Monday, 15 March

Progressive restoration of undamaged servers by Monday, 22 March

SBG-2

Building out of use

Audit and inventory of premises

Replacing infrastructures in other datacentres

SBG-3

Servers undamaged

Tests for the high voltage supply will be carried out this weekend.  Provisional restart on 15th March

A progressive restart of all services is estimated from Monday, 22 March.

 

SBG-4

Servers undamaged

Scheduled for Monday, 15 March

A progressive restart of all services is estimated from Monday, 22 March.

 

 

  • Mobile high-voltage network and electricity units arrived at the Strasbourg site today at 12pm.
  • 60 experts are currently working on site (in rotation over the weekend) in four teams, with the following missions:
  1. Team 1 (network) is tasked with connecting the mobile network unit and testing the network room in SBG1
  2. Team 2 (electricity) is working on high-voltage electricity connections
  3. Team 3 (electricity) is working on low-voltage electricity connections
  4. Team 4 is working on protecting and restoring the cooling systems

The aim is to reconnect, repower and restart the backbone access network from Monday, 15 March (following the authorisation of the ESR (Électricité de Strasbourg Réseaux)).

To repower the site:

  • Two generators have been rented since 11th March to power the temporary office.
  • One additional generator was rented to recharge the batteries of our generators following their activation during the incident.

 

  • Several service providers are currently working on site (30 people)
    • The water on the floor of the SBG3 building has been cleaned up
    • The SBG3 building is currently being dried out, which should be completed by Sunday evening
    • The cleaning of any soot in the server rooms of SBG3 will begin on Monday.
       
  • Repair of the damaged cooling system:
    • A crane has arrived on site.

Priority #2:

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our Roubaix (RBX) and Gravelines (GRA) datacentres.
     
  • We are committed to providing around 15,000 new servers in the coming weeks.
     
  • Update on quantities delivered since 10th March:
•	Update on quantities delivered since 10/03

Priority #3:

  • Backup mechanisms:
    • An overview of all backup mechanisms will be published in the FAQ today.
    • For Public Cloud and VPS customers, questions about “zone vs datacentre (SBG)” will be clarified in the FAQ today.
    • An ad hoc mechanism is in place to help our customers set up their DRP system.

Communications

- FR: https://twitter.com/olesovhcom/status/1370045226929299461

- EN: https://twitter.com/olesovhcom/status/1370045891806175232

Our mission remains to provide our customers with the highest quality of service, to support their online business. And we know how important this is to them. We sincerely apologise for the problems caused by this fire. We will continue to communicate with the utmost transparency about the cause of the fire and its consequences.

An investigation is ongoing. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

-------------

Roubaix, France | 12 March 2021 - 14:00 (CET)

Customer Information:

  • Octave Klaba, founder of OVHcloud, has uploaded videos providing information about the situation.

FR: https://twitter.com/olesovhcom/status/1370045226929299461

EN: https://twitter.com/olesovhcom/status/1370045891806175232

  • Today, we will inform our customers by email about the status of their services. We also have a 24/7 support team in place to support them.

As a reminder, our three priorities are as follows:

  • Priority 1: restoring services to SBG1,SBG3, and SBG4
  • Priority 2: providing our customers with infrastructures in our non-affected datacentres
  • Priority 3:  Implementation of all DRP (Disaster Recovery Plan) mechanisms with our customers.

Priority 1: restoring services to SBG1, SBG3 and SBG4

  • Diagnostics and inventory are ongoing.
  • To guarantee the safety of our teams, cleaning and repairs to the damaged buildings are in progress.
  • The tasks of cleaning the foam and the removal of water on the public road have been carried out. The supply tank of the hydraulic system has been drained.
  • All long-distance fibre connections are operational.
  • SBG1 was partially damaged (4 out of 12 rooms):
    • The network room was checked yesterday.  An isolated electrical test run, without connectivity, is planned for this weekend.  We estimate an orderly restart of connectivity early next week.
    • Reactivation of the electrical network is tentatively planned for  Monday, 15 March.
    • Servers will be audited and reactivated one by one.
  • SBG3 was spared from the fire
    • A network room will be set up by the end of this week, in order to be operational at the beginning of the week of 15 March.
    • Tests for the high voltage power supply will be carried out this weekend. Provisional estimate for the reestablishment of power: Monday, March 15.
    • A progressive restart of all services is estimated beginning Monday, 22 March.
  • SBG4 was spared from the fire
    • We will finalise our service reactivation plan by the end of this week.

All affected Email Pro services began being restored on Thursday, March 11, starting at 4:55 p.m. No email was lost.

Priority 2: providing our customers with infrastructures in our non-affected datacentres.

  • For customers that have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our Roubaix (RBX) and Gravelines (GRA) datacentres.
  • We have already delivered to our impacted customers:
    • Wednesday 10 March: 2100 new servers.
    • Thursday, 11 March: 630 new servers.
  • The new organisation of our assembly lines has been operational since the morning of Friday, 12 March and has tripled our production capacity (3,000 to 4,000 servers per week).
  • We are committed to producing nearly 15,000 new servers in the coming weeks.
  • Delivery times for the new services will be communicated as soon as possible.

Priority 3:  Implementation of all DRP (Disaster Recovery Plan) mechanisms with our customers.

  • By the end of the week, our support teams and Technical Account Managers will define a support plan for our customers which have not yet activated their DRP.

Communications

- FR: https://twitter.com/olesovhcom/status/1370045226929299461

- EN: https://twitter.com/olesovhcom/status/1370045891806175232

Our mission remains to provide our customers with the highest quality of service and  to support their online business and we know how important this is to them. We sincerely apologise for the problems caused by this fire. We will continue to communicate with the utmost transparency about the cause of the fire and its consequences.

An investigation is ongoing. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

---------

March 11 2021 – 15:30 (CET)

People :

  • HR teams arrived on site today to support the teams.
  • Additional industrial and technical teams arrived from other offices and datacenters to reinforce the on-site staff and provide specific expertise.
  • Logistic and food services have been provided to all on-site teams.
  • A temporary office has been established.
  • We’ve reinforced the staff on the French production site (Croix) with internal volunteers to address the current production plan and additional workload caused by the Strasbourg incident.
  • The production will be carried out in extended working hours and during the weekends.

Production and Datacenters :

  • We’ve started to propose replacement infrastructure (Dedicated servers, Hosted Private Cloud, and Public Cloud) in Roubaix and Graveline datacenters to our customers.
  • To address the demand, additional assembly lines will be in place in the next 48 hours. This will triple our production capacity.

On the Strasbourg site:

  • Diagnostic and inventory currently in progress.
  • We are in the process of clearing up and repairing the damaged buildings to guaranty the safe working environment for our teams.
  • We have finished removing the extinguishing foam and water from the street. We will remove the water from the water tank today.
  • SBG2 will need to be almost entirely reconstructed.
  • SBG1 was heavily damaged.
    • Following a preliminary audit, Network room SBG1 will be restored at the beginning of next week.
    • The provisional date for restoring the power supply is Monday, March 15th.
    • In the coming days, the servers will be activated room by room after the audit.
  • SBG4: preliminary audits did not reveal issues. Our ambition now is to restore power during week 12 and then gradually reactivate all services.
  • SBG3 was not impacted by the fire. We aim to restore the power and network during week 12 and then gradually reactivate all services.

Our three priorities remain as follows:

  1. Reserve infrastructures at our other datacentres for our affected customers: we have a stock of new servers at the Roubaix and Gravelines sites, ready to be delivered to the majority of affected customers. We will further enhance availability in these datacentres, with the production of nearly 10,000 new servers in the coming weeks. Affected customers will be notified about this process as soon as possible.
  2. Secure the site now that we have regained access, clean it up, and reconnect the electricity and the network for the three affected datacentres.
  3. Continue to assess the impact on our customers’ servers at the affected datacentres, in order to find the best solutions and communicate to all customers
     

Additional actions taken by OVHcloud:

Network:

  • All fiber paths that enter the datacenter site are operational.
  • We have placed orders with different suppliers – due date for delivery on-site, March 11th.

Communication:

Our mission is to provide our customers with the highest quality of services to support their online activities and we know how important this is to them. We sincerely apologise for the issues caused by this fire. We will continue to communicate with the greatest transparency about the cause of the fire and its consequences.

An investigation is ongoing. Regardless of the cause of the incident, OVHcloud is currently evaluating the potential technical and operational measures aimed to bring solutions to the impacted customers.

An FAQ is already available here.

---------------

10/03/2021 – 21:00 (CET)

Summary:

  • At 00:47 on Wednesday 10 March 2021, a fire broke out in a room at one of our four OVHcloud datacentres in Strasbourg (SBG2). 
  • The fire was contained by the early hours of the morning.
  • There are no injuries.
  • The fire mostly destroyed the SBG2 datacentre and partially damaged the SBG1 datacentre (4 of the 12 rooms destroyed). The two other OVHcloud datacentres in Strasbourg were not affected by the fire; SBG3 and SBG4 servers are currently switched off but undamaged.
  • The site is not classified as a Seveso site. 
  • The cause of the fire has yet to be established and an investigation has been launched as mandated by the authorities.

Actions taken by OVHcloud:

  • The technical and commercial teams have been working since this morning to inform our customers and handle the unavailability of our Strasbourg site.
  • The company’s founder, Octave Klaba, has been on site since this morning with the industrial and technical teams.
  • To follow updates to this situation in real time:
  • Incident tracking platform: http://travaux.ovh.net/?do=details&id=49484
  • Twitter: https://twitter.com/ovhcloud
  • A customer announcement, along with an FAQ, will be available tomorrow morning.
  • To help us handle customer requests, we recommend using the ticket feature on our website or consulting our help centre.

Our three priorities are as follows:

    1. Reserve infrastructures at our other datacentres for our affected customers: we have a stock of new servers at the Roubaix and Gravelines sites, ready to be delivered to the majority of affected customers. We will further enhance availability in these datacentres, with the production of nearly 10,000 new servers in the coming weeks. Affected customers will be notified about this process as soon as possible.
    2. Secure the site now that we have regained access, clean it up, and reconnect the electricity and the network for the three affected datacentres.
    3. Continue to assess the impact on our customers’ servers at the affected datacentres, in order to find the best solutions.

We are doing everything we can to ensure a continuity of service to our customers:

  • We are working on a plan to relaunch the three unaffected datacentres (SBG3 and SBG4), the partially affected datacentre (SBG1), as well as our network, as quickly as possible.
  • We ask that our customers exercise caution around the emails they receive: in times of crisis, it is common for malicious activity (phishing, spam, etc.) to increase. It is more important than ever to stay alert. 

Impact on our operation:

  • We are continuing to assess the impact of this incident, particularly for the customers whose data was located in the datacentre destroyed by the fire.
  • Our Web-based VOIP services in France were not affected.
  • All of our services in our other France-based datacentres and across the world (including 15 datacentres in Europe) are fully operational.

Our mission is to provide our customers with the highest quality of services to support their online activities and we know how important this is to them. We sincerely apologise for the issues caused by this fire. We will continue to communicate with the greatest transparency about the cause of the fire and its consequences.

We are assessing the environmental impact by working with the relevant authorities on a procedure to confirm that no pollution was caused.

At this stage, we can confirm that the local residents are not at any risk.

We are continuously assessing the impact of this incident and will communicate as transparently as possible on the progress of our analyses and the solutions to be implemented.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

(Incident tracking platform: http://travaux.ovh.net/?do=details&id=49484)

-------------

10/03/2021 – 11:00 (CET)

At 00:47 on Wednesday, March 10, 2021, a fire broke out in a room in one of our 4 datacenters in Strasbourg, SBG2. Please note that the site is not classified as a Seveso site.  Firefighters immediately intervened to protect our teams and prevent the spread of the fire.

At 2:54 am they isolated the site and closed off its perimeter.  By 4:09 am, the fire had destroyed SBG2 and continued to present risks to the nearby datacenters until the fire brigade brought the fire under control. From 5:30 am, the site has been unavailable to our teams for obvious security reasons, under the direction of the prefecture. The fire is now contained. We are relieved that no one was injured, neither among our teams nor among the firefighters and the services of the prefecture, whom we thank for their exemplary mobilization at our side.

With our 15 datacenters in Europe, our technical and commercial teams are fully invested to support our customers, implement solutions and mitigate the impact of the unavailability of our Strasbourg site. Our mission is to provide our customers with the highest quality of services to support their online activities and we know how important this is to them. We sincerely apologize for the difficulties and challenges caused by this fire. We are committed to continue to communicate with the greatest transparency about the cause of the fire and its consequences.

We are currently assessing the impact of this incident and will communicate as transparently as possible on the progress of our analyses and the implementation of solutions. All of our communication channels, starting with our incident tracking platform (travaux.ovh.com), are accessible so that you can stay informed of developments in real-time. 

Press Contact | OVHcloud

press@ovhcloud.com

My customer accountContact SalesWebmail OVHcloud Blog

Welcome to OVHcloud.

Log in to order, manage your products and services, and track your orders

Log in