While working onsite with a client something popped into the Incident Response Queue and the client had to leap into action.Looking at why one of their applications was running slow in a cloud environment they went through the standard bit of debugging and determined the application had been under load from increase in users. This meant their server was a little underpowered now and standard amount of expansion was expected so they threw another CPU set into the mix and upgraded the RAM. Performance monitoring was added and the application started working better with the new load.
2 days later, the application is performing poorly again. Again the determination was made that it was just probably more users. I was onsite and asked them a few standard questions around their hypothesis. How many more transactions had been completed? What was the average transaction time? Turns out, there were no more users than normal but the application was slower and slower. We started looking for possible memory leaks in the middleware, upgraded a few components to newer versions, none of this helped.
What finally took us toward a resolution was looking at the open ports on the server. We saw a port open to a well known cryptocurrency mining service. After inspecting processes we figured out that a cryptocurrency miner was running on the application server. Once we locked that service down, ripped out the miner and did a password reset on all cloud services, including adding Multi-Factor Authentication to the console, we were able to determine the miner was running on two other servers as well.
So, how did we get here? Well, single factor authentication was a big part of it. A bot had attempted various combinations of logins and finally found a username and password combo that worked. Once in it installed the miner and started pulling work files and finding solutions. Bots used to be used to find email servers for spam points, often they are used to locate unsecure Word Press instances to store malware as part of a phishing campaign. Bots are always bumping up against websites looking for weaknesses.
In this case the biggest market drivers today are crypto currency and running as many crypto miners as possible can be lucrative. As a site owner you might not realize its happening but having good monitoring can help. Just imagine if this organization had truly elastic cloud infrastructure. The Miner would have been more and more effective and eventually the bill for running the site would have been greater than the revenue it generates.
Be careful out there, have good hygiene and if you get stuck, call the experts.