Your Linode, linode1234, has exceeded the notification threshold (5) for outbound traffic rate by averaging 8.67 Mb/s for the last 2 hours. The dashboard for this specific Linode is located at: >https://manager.linode.com/linodes/dashboard/linode1234> This is an automated message, please do not respond to this email. If you have questions, please open a support ticket. You can view or change your alert thresholds under the "Settings" tab of the Linode Manager. This is not meant as a warning or a representation that you are misusing your resources. We encourage you to modify the thresholds based on your own individual needs.
I tried to log in to the machine, but ssh was hanging. I logged in on the dashboard, and found that both the outbound traffic and CPU usage had been pegged at a high plateau for about 4 hours.
I rebooted the machine, and after 15 seconds or so I could log in and start top -c. After about 10 seconds a program called /bin/unama started hitting 100%. That was likely the culprit. I killed it, defanged the file and moved it to a different directory for analysis, and saw that it had started up again. Running 'ps' would just hang. Time for another reboot.
This time I ran find to find all files in /bin and /usr/bin that were newer than 3 days, my last login. The list gave /bin/ps, /bin/netstat, /bin/unama, /usr/sbin/lsof, and /usr/sbin/ss. Another reboot, and I defanged all those files. All of them were copies of /bin/unama. Under /etc/rc.d I found that multiple instances of DbSecuritySpt and selinux were starting up /bin/unama. I deleted all those, rebooted, and it looks like unama was fixed.
Viewing the logs suggest that the hackers found a vulnerability in my ancient rails/nginx stack that let them run as root and walk over my server. unama is jackhammering a couple of dozen IP addresses, which a sample shows are all telecoms in China.
As for now, bentframe.org is on hiatus, and over the holidays I'm going to rebuild it using devops principles to make it easier to stay abreast with new vulnerability reports.
Sorry no screen shots. They would've been instructive, but I just wanted to fix the problem and get some sleep.
