Open Season on Malware and Ransomware!

When we celebrate innovation and rejoice in the democratization of technology, we often forget this also benefits the malicious actors on the network. Despite advances in antivirus software, launching malware or ransomware has never been easier. For a year and a half OVH has contributed, at its level, to make the web safer by trapping malicious programs and when possible, their authors and discourage them from using the OVH network. Frank Denis, security expert within OVH’s Security Operation Center (SOC), which is made up of a 20 member team spread over three continents, provides us with some insight.

Malware a more widespread problem than DDoS attacks

We regularly report on DDoS attacks, which are occurring with increased frequency and growing intensity. However, today these denial of service attacks are well controlled through the use of equipment put in place by OVH that detect and mitigate (lessening the consequences of an attack by vacuuming illegitimate traffic) such attacks. This protection of course has a significant cost, and moreover OVH must add enough additional network capacity to be able to absorb monstrous volumes of requests without affecting all other users. But overall, DDoS attacks can be contained as long as the proper means are in place: equipment and humans – because to combat these requires vigilance, in order to identify new types attacks and response by updating the algorithms that govern the equipment providing the protection (a real and notable challenge when it comes to anti-DDoS Game).

The problem which the OVH SOC team has decided to tackle for the past year and a half is more devious. It has been working to put the brakes on the proliferation of malware and ransomware with the latter being especially in vogue for the past three years [see sidebar]. These malicious programs infect computers and servers, encrypting their data and ransoming it from their owners or sending it to third parties through an array of complexity comparable to that of offshore financing. Also, they sometimes take control of machines with the goal of creating a botnet to carry out Distributed Denial of Service attacks (DDoS). To catch the authors of these programs, a majority of which come from Eastern European countries (notably Russia and Ukraine), Frank must be clever and his techniques borrow as much from computer science, with reverse engineering as good old fashioned police work.

Malware traps and spam nets

Based on the principal of a mouse trap, baiting undesirable rodents with a piece of cheese, OVH intentionally places on its network machines that are very easy to hack. “These machines record all activity and allow us to better understand how our users’ servers are compromised and what purpose they serve afterwards.” Additionally, OVH has created and released on the web (forums, mailing-lists…) thousands of valid email address (even entire domains), so that, they are available to spammers. All that remains is to raise the nets regularly: “The emails received are analyzed. Those that contain interesting attachments are stored, grouped and dissected. This allows us to recognize current campaigns and identify those involving servers present on OVH infrastructures.

Frank explains he has developed a sandbox environment, based on virtual machines, which makes it possible to observer malware without contaminating the local network or actually infecting any machines. Many tasks are automated, but analysis is largely entrusted to humans. “Machine learning does not yet work well enough to automate the process of blocking malware, which continues to evolve to limit the risks of it being detected.”

Ongoing investigations

Once evidence has been collected, an investigation can begin. The investigation is complicated, because in almost all cases, the mice that take the cheese offered by OVH have done so unwittingly. “Most often, it’s a server that has been infected and reconfigured to act in a malicious manner. For example, by becoming the host of a phishing page designed to collect banking information, before sending the stolen data to another server. The network of machines is vast and complex, with servers which diffuse malware and control infected machines, others recuperate information and play the role of proxies by relaying the connection to a third machine via a VPN, then to a fourth via the Tor network… And of course, servers are spread out among different hosting providers, datacenters, geographical regions and jurisdictions.

When an OVH machine appears to display the characteristics of a server involved in the distribution of malware, control of infected machines or data exfiltration – whether it be a top level botnet–the ability to intervene directly is very limited. In fact, OVH does not have access to data stored on the hard disks of its customers’ servers. Frank informs the authorities, who may initiate a criminal investigation. This consist, firstly, to identify the server administrator through a police request, to determine their level of involvement (victim, negligence or part of a criminal network). Then, if they judge it necessary, the authorities proceed to seize the hard drives and/or take legal action. Operations are carried out in a strictly regulated manner, under the control of a magistrate and the OVH legal department. “Through forensic analysis and international cooperation agreements, police services attempt to track down botnet operators to dismantle criminal networks which are at the source and derive substantial profits, which often finance other criminal activity outside of cyberspace…” OVH’s actions and those of legal authorities are complimentary: “Our expertise consists of identifying, in accordance to technical characteristics, the machines situated at the top of cybercriminal networks which are the source of malware campaigns, whereas the police hold the power to investigate. Our cooperation allows authorities to focus on the most interesting machines.

Analysis of a server's memory infected by Locky showing IP addresses of the machines contacted by the malware in order to get a key allowing to encrypt data.

Reverse-engineering to spot hacks as quickly as possible

“On our side, except in cases where the authorities ask us not to act, servers distributing certain ransomware are permitted to continue to operate temporarily in order to collect evidence. We take all measure necessary to stop the propagation of malicious software and the theft and sale of data through machines which are under our authority.”
So when the server of an OVH customer appears to have been hacked, the owner is alerted and encouraged to take corrective action by cleaning or reinstalling the machine, otherwise, OVH suspends their server. The same goes for reoccurrences. “When a WordPress instance, for example, is hacked, we provide the customer concerned with a list of the pages added by the hacker and we identify the modifications made to their system’s code that is used to create backdoors through which hackers operate.”
Sometimes it happens that OVH does not know the identity of the end user administering the compromised server. This happens in cases where the server is rented by an intermediary who resales to their own customers. “We get in touch with the reseller, so that they can alert the administrator of the server. If there is no response on the part of the reseller and upon further review we find that the reseller has an accumulation of machines involved in a cybercriminal network, we do not hesitate to terminate any contracts binding us to the customer. From experience, we know that 99% of cybercriminals do not directly rent servers from a hosting provider but rather through intermediaries. Therefore, we are especially adamant in this type of situation.”

Tracking cybercriminals is necessary, but it’s a lengthy process. Interventions are only carried out when an alert has been made – this is the policy of most ISPs – and is often ineffective: the URLs sent are already no longer valid, the servers involved have been returned and for the most part the malware campaign over (this is without counting false positives!). “Therefore, two actions must be taken: educate our customers and discourage cybercriminals from using the OVH network.” This is where Frank takes off his detectives cap and puts on his computer engineer’s. “The other part of my job consist of reverse engineering the malware that we’ve captured in out traps or has been sent to us by other security researchers.” The goal is to adopt a proactive approach, capturing weak signals to identify new operating methods and cut the ground from under the cybercriminals, “without resorting to intrusive methods and automating the maximum amount of out process to increases our efficiency,” says Frank. “At the end of 2015, banking malware started to propagate from some servers of which a majority were hosted at OVH. We had managed to understand how the hacked servers were configured to do harm and we were able to cut them off before they were used. The result was that we never saw this malware return here. If we detect malware before it can take action, it can be a bit discouraging to cybercriminals.”

Educate users to minimize the human factor

On the education front, there is also work to be done. Often the cause of infection is the fault of a human – or at least a lack of vigilance. Regarding personal computers, e-mail still proves to be an effective contamination vector, through malicious banner ads (malvertising) and by the exploitation of software vulnerabilities (exploit kits). Regarding the servers, we can distinguish two categories of offending administrators: those who leave the key in the door, using very simple passwords and those which leave the windows open, forgetting to update their WordPress or Joomla installations to the latest versions. “It is not that the code for these applications is poorly developed, rather it is the massive deployment of these applications that arouses the appetite of hackers, with more people looking for security vulnerabilities there are more to be found. Hackers are people like everyone else, they are concerned about efficiency.”

Obviously, the most exploited security vulnerabilities are known quickly, but it is impossible to perform a network scan, it is illegal in France, not to mention such an action would not be in line with OVH’s own data security policies, nor the customers’ expectations to privacy. It is therefore very difficult to take precautionary measures… “Nevertheless we do the maximum. For example, if we become aware of a leak of usernames and passwords being broadcast on specialized forums or contained within servers collecting stolen information, we contact any OVH customers which may be effected with an alert.”

OVH could resort to passive network analysis, that is to say observe network traffic without interfering with it (unlike a scan that queries all machines one by one). “Passive analysis is relatively simple to put in place on a small network, but on a network of the size of OVH’s, it is a different story. It is not impossible, just complicated.” Another track: detect hacks and alert server owners immediately, this time using the monitoring tools installed by default on the dedicated servers. Effective, but not foolproof: malware can be coded to deactivate these tools or modify what they examine… Also, Frank advises to use sandboxes, which permit the execution of questionable files within an isolated environment, to observe what an application actually does. This is not always the panacea: “In order to alter behavior, malware is capable of detecting sandboxes. There are constantly new techniques emerging to detect security measures…”

Is this a useless fight? “Not really,” adds Frank.  “We will always be at the mercy of targeted attacks, for which, by definition, there does not exist a model of detection, and the attacks exploit human flaws. That said, through our work, OVH makes these criminal activities a bit more difficult and more expensive. Since they know they are hunted down and that we can find and suspend some of the servers and domains involved in any illegal activity, criminals are forced to update their infrastructures. This certainly does not create an obstacle for those who derive their income exclusively from such activity. They go elsewhere. But it put the brake on others.” What this means for OVH, the European leader of the cloud with 250,000 servers and a million customers, is that the proportion of its customers involved in illegal or questionable activities has been greatly reduced with the vast majority of customers using its services for acceptable and legitimate projects.

Machine learning used to discriminate against hackers that order a server

If it is very difficult to model patterns to detect malware before it acts (this is due to its diversity and constant evolution). But OVH is set on using machine learning to develop a robot-portrait of a hacker and block a large portion of users wishing to employ OVH services to host malware or engage in other illicit activities. Through big data, OVH studies, posteriorly, the characteristics of user accounts which have been linked to illicit activities, comparing them to legitimate user accounts. This makes it possible to detect common points, similar behaviors and makes the machines taking new orders more intelligent, soliciting human analysis when suspicion arises.