How do we back up a million databases every day?
Here is an average day, from the perspective of a VAC system (the OVH anti-DDoS protection system): legitimate incoming traffic in green and in red, illegitimate traffic (DDoS attacks) filtered by our protection system.
DDoS attacks - an unavoidable phenomenon
2013 saw a surge of DDoS attacks, some peaking at unprecedented levels. It made us realize that OVH had been under-investing in anti-DDoS protection technologies.
For the record, we later discovered that some of these attacks – particularly those targeting customers in the online gambling sector – originated from an unscrupulous competitor. As soon as the attack finished, they would contact the victims to offer their services while, of course, making much of their DDoS protection.
Never mind. We’d already decided that the courts couldn’t be the only way to fight the attacks. For sure, legal authorities and their special cyber-crime units are attempting to fight back against DDoS attacks. But let’s be realistic. Investigations are long and complex, both technically and due to the international networks that give anyone who can pay the ability to carry out wide-ranging attacks.
The real battles have to be fought at a technical level. DDoS attacks are inherent to our business as a hosting provider. It’s therefore our responsibility to implement adequate protective measures to mitigate the impact of attacks as much as possible, without raising false hopes that they will deter attacks.
In 2017 we’ve seen an average of 2000 attacks per day, of which 20 can be considered powerful (dozens of Gbps).
This figure has never dropped. You’d be surprised to see the list of countries hosting the largest numbers of bots – the zombie machines behind the biggest DDoS attacks. A recent article pointed out that while God works in mysterious ways, the Vatican’s computers, servers, phones and other connected objects are much easier to read. Why? The city has the world’s highest density of bots per internet user. Perhaps this explains the hacking of Vatican tourists’ phones...proof that there are no miracles in cybersecurity.
Sharing the cost of anti-DDoS protection to benefit everyone
In 2013, we asked our customers for their contribution in order to deploy our anti-DDoS VAC system as quickly as possible and make up for lost time.
We set ourselves apart from our competitors by not offering DDoS protection as an optional paid service. Instead, we offered everyone the service by including it in all solutions as standard, and spreading out the cost.
Octave Klaba said at the time: “It is not a question of whether you need anti-DDoS protection, but when you will suffer your first DDoS attack. OVH is by your side to protect your project 24/7 against any type of DDoS attack, no matter how big it is and how long it lasts.”
Remember, a DDoS attack always results in collateral damage when no protection is in place. Depending on the intensity of the attack, if it is not mitigated, all neighbours of the targeted server rack might temporarily lose their service.
That is why we felt that offering protection to everyone was the best option. Many DDoS attacks have criminal or pointless motives, but they can also be a censorship tool, as demonstrated by American security expert Brian Krebs. Protection against DDoS attacks therefore not only helps to maintain network quality, but also allows us to better protect freedom of expression, which has always been highly important to us.
New-generation anti-DDoS & international deployment
Deploying the VAC in 2013 has given OVH customers a level of protection that was unrivalled at the time and still ahead of the game today. This was demonstrated in September 2016, when OVH fought off a record attack of 1 Tbps.
The VAC system is still performing well, working away in the background to protect OVH customers. For the most part, they don’t realize they’ve been the target of an attack until we email to let them know.
That being said, as you know, OVH has since embarked on an ambitious international development plan, setting up new data centers in Europe, Asia-Pacific and North America. The aim is to have a presence wherever our customers wish to expand their markets.
We therefore had to think hard about how to scale up the VAC technology. It was initially based on three modules in Roubaix, Strasbourg and Montreal, with a fourth added in Gravelines in 2016.
Aware that the proprietary technology deployed in 2013 was reaching its limits in terms of capacity and scalability, we have developed our own anti-DDoS solution over the last two years. This is based on a range of technological features: FPGA filtering (re-programmable computer chips with a more powerful processing capacity compared to CPU), x86 servers with 6WIND software acceleration and the open-source DPDK library, plus the latest generation of Mellanox 100GbE network cards. This now represents over 100,000 lines of code (our mitigation approach is constantly developing) in our next-generation, made-in-OVH VAC.
To scale up our anti-DDoS protection, we started by replacing the first four VAC 40G units with the next-generation VAC 100G units, each with a capacity of 600 Gbps. Then we started deploying these “vacuum cleaners for illegitimate traffic” in data centers in new locations.
The aim of boosting the number of VAC units is to deal with attacks as closely as possible to their source. This prevents moving them through the backbone and thus avoids mobilizing bandwidth unnecessarily, at the risk of saturating certain connections.
The next-generation VAC has been deployed at the same rate as new datacentres. So far in 2017, OVH has deployed five additional VACs: in Singapore, Sydney, Warsaw, Limburg (Frankfurt) and London. This brings the number of VACs to nine, with a total capacity of more than 4 Tbps. Deployments are planned for the East Coast of the USA (Vint Hill datacentre, early October), Spain and Italy.
In parallel with the international deployment of next-generation anti-DDoS systems, we are pressing on with more R&D. The race against time we mentioned in the introduction has no finishing line. As the internet’s size and capacity expands, the intensity of the attacks will continue to grow. And their sophistication will grow too. On the one hand, we’re studying the mechanisms behind the attacks in order to continuously improve our mitigation tactics. On the other, we are well aware that the attackers are trying to understand how our protective mechanisms work in order to get around them more easily. We have to stay ahead of the game.
Our users themselves want us to perfect our anti-DDoS protection systems. VAC does protect all OVH customers by default, but it’s only activated when an attack is detected. It then starts to filter illegitimate traffic to keep the targeted server up and running. Some customers, for example in the finance sector, have opted to keep the VAC permanently activated. The slowdown in service during the three seconds or less that it takes, on average, to detect and react to an attack, is not acceptable to them.
We know this level of reactivity will become standard for an increasing number of users. In the field of the Internet of Things (IoT), for example, detection should lead to an immediate reaction.
Indeed, IoT is quickly going to pose another challenge: how to correctly distinguish a DDoS attack from a massive influx of data from connected sensors, which are multiplying exponentially in industrial sectors in particular.
Today, the VAC system is designed to protect our customers from external attacks, i.e. from outside our network. For attacks originating within the OVH network, we can detect and cut them off at the root in less than 30 seconds by isolating the implicated services from the rest of the network. It works efficiently but we can and would like to offer a higher level of protection. We would position an additional anti-DDoS protection lower down in the network, as close as possible to the server, that allows us to be even more reactive in protecting servers from internal attacks.
It’s actually what we already offer on our GAME servers, whose specific requirements in terms of anti-DDoS protection pose a formidable technical challenge. This has driven our innovation in the same way that advances in the aerospace industry often benefited everyone else a few years later. To attain this additional level of protection and offer it to everyone, we’re going to use our vRouter technology. This is a virtual router based on x86 servers, which we're gradually installing in our data centers. This allows us to deploy high-performance filtering throughout our data centers.
The proof of concept is currently underway in Roubaix, and the technology is already working in 20,000 servers. We’re also experimenting with techniques that allow us to offer full protection against certain types of attacks by anticipating them and activating protection measures before the first packets reach our network.
In essence, not only do we want to provide effective anti-DDoS protection, we want to offer the best possible protection. To prove how serious we are, we are considering including the risk of undergoing a DDoS attack in our SLA. In other words, we would contractually guarantee the availability of our services, even in the case of a DDoS attack.
Increase in interconnection capacities: the hidden cost of anti-DDoS protections
Along with the significant cost of the protection system that mitigates the attacks, i.e. intercepts and filters out illegitimate traffic without affecting legitimate traffic, there is a hidden cost. This relates to the need to increase our interconnection capacities (peering) with ISPs all over the world. The aim is to avoid saturating links between the sources of the DDoS attack (which, as the name suggests, are distributed) and our various VACs. These peerings increasingly come with a charge.
OVH needs a lot of excess capacity to be able to absorb the peaks of high-intensity attacks without saturating parts of the network. Just knowing how to deal with an attack is not enough – you must also have the ability to deal with it. Of our 12 Tbps of network capacity, we actually only use 3.5 Tbps on average.
It’s clear that the investment required to combat DDoS attacks is, at least partially, unrelated to OVH’s growth rate. Increasingly powerful attacks force us to accelerate spending at a quicker rate than the OVH server park is growing.
IT infrastructure security is now, more than ever, a critical concern for businesses. Cyber threats, from ransomware to DDoS attacks, have been regularly making headlines in the last few months. There is, without a doubt, a growing awareness of the sheer volume of these attacks. Certain types of threats are difficult to eradicate, which is incidentally why insurers consider cyber attacks to be a high-potential market. However, today, the issue of DDoS attacks is pretty well covered, as long as you choose a cloud provider that prioritizes and invests in this issue. This is why OVH has decided to request a contribution from customers, via a price increase for VPS, Public Cloud instances and dedicated servers (except Game). We have updated our price lists by taking into account the Anti-DDoS protection as well as currency fluctuation. This increase will be visible on the OVH websites by the end of October and it will apply to new orders of dedicated servers and VPS. The same will be rolled out for Public Cloud at the start of December. Existing customers whose services are affected by the price increase, will receive a personal email notification of the revised pricing due to take effect on December 1st. If these customers have chosen a 3, 6, or 12 month subscription, the increase will become effective on the next renewal date of their services.
How do we back up a million databases every day?
OVH wishes you all the best for 2018, and presents its ambitions for this year
OpenZFS: why is this open source storage technology still so appealing?