Following the two incidents on the 9th of November 2017

Roubaix, November 13, 2017

Following the two incidents on the 9th of November 2017

Last Thursday, the 9th of November two of our sites were impacted by two simultaneous but unrelated incidents: a power failure in Strasbourg (SBG) and a software bug on our optical equipment in Roubaix (RBX).

This had some direct consequences on other services. For example, it was especially difficult for our customers to receive emails.

All our teams were summoned immediately. They worked non-stop to resolve the situation as quickly as possible, while also communicating transparently, as soon as the incident started, via the voice of Founder and CEO, Octave Klaba. Customers were informed of the development of the situation in real time, via Twitter and then via status updates (travaux.ovh.net).

Thanks to our teams’ intervention we could get the Roubaix site up and running again in under two and a half hours and the Strasbourg site in under four hours. Apart from some customers hosted in Strasbourg, whose services are still being rebooted, our services have now returned to normal. We have now activated the compensation package.

OVH’s technical and commercial teams are still fully mobilized to support our customers, to facilitate their understanding, and to answer their questions.

We have of course learnt some lessons from this exceptional situation. We will be making some changes as a consequence. An action plan has been decided upon, as follows:

  • In Roubaix: we have fast tracked the project to redistribute the optical multiplexing system on two separate systems, to limit the scope of a failure. At the same time, we are also conducting investigations with the equipment manufacturer to fix the bug by upgrading the incriminating software.

  • In Strasbourg: we have reconsidered the electrical design of our site and decided to once and for all stop using shipping containers, a technology used in datacentres SGB 1 and 4. Instead, we will make way for “fixed”, large capacity datacentres, designed for large scale and maximum reliability. The servers in SBG 1 and 4 will be migrated to the SBG3 datacentre, which is under construction, to ensure that every aspect of the site meets the same standards as all the company’s other sites.

  • On all our 12 sites: a preventive audit of the electrical systems is underway, to ensure that they fully comply with internal quality standards.

Our objective is to deliver the technical standards and excellent service level that our customers expect, each day, anywhere in the world. We are all aware of how important our services are to our customers. We would once again like to offer our most sincere apologies for these incidents and the damage they may have caused.

Our job is to ensure that the services you trust us to host are protected in all circumstances. Even the most unlikely. It is also our job to learn the necessary lessons when the measures we have put into place do not work as they should. The 9th of November incidents have caused us to further perfect our infrastructure, to take care of each detail and to push our standards above and beyond those of the industry. Every day we must earn your trust. And even more so in these types of situations.

About OVH:

OVH is a global provider of hyperscale cloud, which offers businesses a benchmark for value and performance in the sector. Founded in 1999, the group manages and maintains 27 datacentres in 12 sites across 4 continents, deploys its own global fibre optic network, and manages the entire supply chain for web hosting. Running on its own infrastructures, OVH provides simple, powerful tools for businesses, revolutionising the way that more than 1 million customers work across the globe. Respect for individuals’ right to privacy and equal access to new technologies are central to the company’s values. For OVH, “Innovation is Freedom”.
OVH Press Contact: guillaume.gilbert@corp.ovh.com