1.3 Tbit/s mitigated by the VAC: a recap on the Memcached episode
On Thursday 1 March, at around 02:00 (GMT+1), our VAC mitigated a DDoS attack exceeding 1.3 Tbit/s, which was aimed at one of our customers — thereby breaking the record of 1 Tbit/s previously held by MIRAI. A few hours earlier, GitHub had been the target of a similar attack, again reaching 1.3 Tbit/s. Unfortunately, the Github website was down for several minutes.
These impressive attacks had one thing in common: poorly configured Memcached services were used around the world to launch what are known as amplification attacks. A lot was learned over a tense week marked by numerous investigations.
Memcached: default configuration to blame
Memcached is an open-source key-value store distributed cache. It is typically used to improve performance, especially for websites, store database request results, and improve load times for dynamic pages. The tool is used very frequently by webmasters and by some editors, who regularly incorporate Memcached into their solutions. one example of this is the webmail service Zimbra. But why has such a widely-used service suddenly been exploited?
As early as 2017, security researchers from the Chinese firm 360 identified the possibility of using an incorrect configuration in Memcached to launch amplified denial-of-service attacks. When Memcached is installed, it listens via the public interface by default. If no rules are configured for the firewall, Memcached will thus continue to be accessible to anyone on the internet via a server’s public IP. If the service listened via TCP only, the problem would be confined to attacks on confidential data, since there is no authentication mechanism. Unfortunately, the service listens via UDP as well, and this is where the situation gets more complicated, because it means that amplification can be carried out at an impressive factor of up to 51,000; NTP only allows a factor of 557.
What is an amplification attack?
Amplification is a phenomenon that aims to generate a response larger than the request. Figure 2 shows a factor 2 amplification, since the response is twice as large as the request.
To make the most of this phenomenon and use it in denial-of-service attacks, attackers typically combine an amplification mechanism with a reflection mechanism. Reflection involves spoofing an IP address, so that the response is sent to the spoofed IP. A server will respond blindly to a packet’s source IP, even if that IP has been spoofed.
Reflection is common practice in UDP as it is a ‘non-connected’ protocol, i.e. there is no ‘three-way handshake’ mechanism, contrary to TCP. Therefore, all you have to do to obtain a response from your correspondent is send one single packet, as opposed to the minimum of 4 packets that need to be exchanged between the client and the server in TCP, in order for both sides to agree to establish a connection. If the IP were spoofed, the response from the server (SYN-ACK) would reach the victim, and the person initiating the connection would not be able to confirm (ACK) that the connection had been established; this would prevent the connection being established, and mean any spoofing of the IP would be ineffective. For more information, the Wikipedia page gives a relatively good explanation on how to establish a TCP connection.
Action plan at OVH
In the wake of this abnormal increase in the number of amplification attacks using Memcached, we quickly implemented an action plan to ensure we would be able to resist even the largest attacks. Amplification is a very easy vector to identify; that said, we need to ensure that our interconnections with other operators (peering) are not saturated, in order to maintain our quality of service. Before long, our action plan was put to the test when, as shown in Figure 1, one of our customers was targeted by an attack of more than 1.3 Tbit/s, which was intercepted perfectly by the VAC without any service interruptions.
This plan also ensured that those of our customers who had an incorrect Memcached service configuration were not involved in the attacks. To achieve this, we needed to:
• Send out a targeted communication to our customers who were potentially concerned, in order to help them configure their service. We did this via a guide ;
• Avoid waiting several hours/days until our customers fix the problem by detecting the OVH IPs acting as reflectors, and intervening directly on the network without reducing the quality of service.
After discussing the issue, we decided not to block the UDP/11211 port (the port used by Memcached) on our network, as it can also be used to initiate communications as a ‘customer’ port. Blocking it could have a negative impact on the quality of service, particularly for online gaming, VoIP, streaming and other services that use UDP. We preferred to take a different approach, which consisted of placing IPs receiving a very large number of Memcached requests per second (and thus identified as contributing to denial-of-service attacks) under permanent mitigation. By filtering out these malicious requests as and when they arrive, we ensured that our customers’ services did not act as reflectors.
Investigations carried out
On 26 February 2018, we observed the first attacks generated by ‘get’ requests, as shown in Figure 4. Just like Cloudflare, we found that this command could be used to read a key containing a .zip file, which was password-protected and contained a .gif file.
What was this .zip file doing in the caches? Straight away, we thought that the services targeted had been prepared beforehand in order to be used during the attacks, and we are currently trying to pinpoint the source of those attacks. Currently, we know that Memcached had already been used for amplification purposes for several days. The illustrations in Figure 5 show the bandwidth for one of our customer’s servers, which had been used since 24 February 2018 to launch attacks against the Chinese IP “59.56.19.xxx” (we have deliberately censored the last byte).
We realised that this IP was being attacked regularly by machines in our network in a very synchronised manner, using the amplification phenomenon (LDAP, NTP, SNMP, etc.). There are several explanations for this phenomenon: either this IP hosted a service that stirred up a lot of hatred, or it is an IP that was used for testing purposes during the preparation of the Memcached amplification attacks.
Taking a closer look, the attacks on this IP are generally very brief: lasting between tens of seconds to a few minutes at the very most, whereas other attacks typically last for several minutes. The largest attack observed so far lasted for three hours. A lot of factors thus lead us to the conclusion that this IP is a ‘dstat’, i.e. an IP used to measure the effectiveness of a DDoS attack in real-time.
This means we can consider it likely that the admins of the ‘booter’ (the DDoS sales platform) probably wanted to compare their various amplification attacks with Memcached amplification. Of all the tests that were carried out over the same time period, we observed NTP (UDP/123), LDAP (UDP/389), SSDP (UDP/1900) and BitTorrent.
As we continued our research on specialised DDoS forums, we finally found a relatively explicit conversation, set out in Figure 6, where a user acknowledged that his DDoS sales platform was the first one to have provided Memcached amplification.
Progress of the threat
In a week, things changed a lot: although the concept of Memcached amplification appears to have originated from a single ‘booter’, other administrators very quickly implemented the feature in order to sell it to their customers. From the very first hits onwards, our honeypots uncovered an extensive number of ways to exploit the Memcached services available on the internet, as shown in Figure 7.
In certain services, the .zip file contained in the ‘a’ key had now been replaced by an ‘abcdefghij’ chain repeated thousands of times, but we did not detect any write attempts:
We also observed keys containing messages asking for a payment of 50 XMR, a popular cryptocurrency also referred to as Monero. Could this be an attempt at holding databases that by definition hold only temporary data to ransom? Or was it just a way of filling the cache in order to increase the amplification factor?
Although a limited number of Memcached services open were rapidly corrected, a very large number of them can still be used to launch attacks.
Like NTP, the Memcached threat looks set to last for a while. Even though the protections implemented by hosts have drastically reduced the size of the attacks — most of them are now under 100 Gbit/s — it is still vital that users rectify their own configurations. However, the NTP case showed us that even 5 years on, some servers are still vulnerable, in spite of the patches available.
So, now that you have finished reading this article, why not check the status of your own firewall, and take a look at the services you are exposing across the internet?