Fixing the unama botnet hack

Depending on which is more important to you, your own health or that of your servers, you should always check your email before heading to bed.  I did late Saturday night, and got this:

Your Linode, linode1234, has exceeded the notification threshold (5) for outbound traffic rate by averaging 8.67 Mb/s for the last 2 hours. The dashboard for this specific Linode is located at: >https://manager.linode.com/linodes/dashboard/linode1234>

This is an automated message, please do not respond to this email.  If you have questions, please open a support ticket.

You can view or change your alert thresholds under the "Settings" tab of the Linode Manager.

This is not meant as a warning or a representation that you are misusing your resources.  We encourage you to modify the thresholds based on your own individual needs.

I tried to log in to the machine, but ssh was hanging. I logged in on the dashboard, and found that both the outbound traffic and CPU usage had been pegged at a high plateau for about 4 hours.

I rebooted the machine, and after 15 seconds or so I could log in and start top -c. After about 10 seconds a program called /bin/unama started hitting 100%. That was likely the culprit. I killed it, defanged the file and moved it to a different directory for analysis, and saw that it had started up again. Running 'ps' would just hang. Time for another reboot.

This time I ran find to find all files in /bin and /usr/bin that were newer than 3 days, my last login. The list gave /bin/ps, /bin/netstat, /bin/unama, /usr/sbin/lsof, and /usr/sbin/ss. Another reboot, and I defanged all those files. All of them were copies of /bin/unama.  Under /etc/rc.d I found that multiple instances of DbSecuritySpt and selinux were starting up /bin/unama. I deleted all those, rebooted, and it looks like unama was fixed.

Viewing the logs suggest that the hackers found a vulnerability in my ancient rails/nginx stack that let them run as root and walk over my server. unama is jackhammering a couple of dozen IP addresses, which a sample shows are all telecoms in China.

As for now, bentframe.org is on hiatus, and over the holidays I'm going to rebuild it using devops principles to make it easier to stay abreast with new vulnerability reports.

Sorry no screen shots. They would've been instructive, but I just wanted to fix the problem and get some sleep.

Nuns and Porn

Amsterdam, July 20, 1979

This work is licensed under a Creative Commons Attribution 4.0 International License.

Uber and the Ongoing Eutrophication of the Middle Class

There were three of us in the early 90s, all fathers of toddlers, hanging out one summer evening in an Ottawa backyard. Mike was on his way to becoming a world-renowned neurophysiologist. Dave was explaining how you could never be bored as an ER doctor, never knowing what the next shift would bring you. I was working with a startup that was helping figure out how HTML and the web could be something more than the next Gopher.

And as happened almost every time, talk turned back ten years to the one things all three of us had in common: our cab-driving days in Vancouver. That sushi restaurant on Powell no one knew about, the ride-and-runs, the big-money nights. Given that undergrad degrees led to scholarships and grad school TA jobs, you could say those three cab-driving jobs helped finance seven university degrees. If you didn't mind the hours, knew the city, and could deal with the occasional belligerent drunk customer, it was a great, low-risk job. At the end of an 11-hour shift you kept half the gross on the meter and all the tips. The owner of the cab got the other half, minus the cost of gas, and of course was responsible for the other costs, like insurance, repairs and maintenance, and even that cab license. If you knew what you were doing and didn't have elaborate tastes, you could make one month's rent on a Friday night, put away the Saturday evening earnings for that month's groceries, and the next few shifts during the less lucrative week would cover everything else.

I haven't ridden in an Uber car. I haven't even seen one, as Vancouver is one of the last jurisdictions to stand up to the company. But I hear reports that the Uber cars are nicer than licensed cabs in most cities, that the drivers are more courteous, the service is better, etc. Could be very true - Uber destroys the decent lower-rung jobs that cab-driving offered and replaces it with a business opportunity. You buy a relatively new car. You're responsible for the maintenance and insurance. You pay the gas. Maybe after Uber pays out your cut of the fares, you're ahead. But this is not an opportunity for a budding scientist.  It's more like the kind of business someone with a recent undergrad degree resigns himself to when he can't afford a post-graduate degree, nor find something better to do.