I just spent the entire weekend re-building a server for the Triangle Linux Users Group.

We first noticed that something was wrong when the machine stopped responding over the network. A couple of our admins took a trip to the data center and noticed that we had a firehose of data on port 6667 (an IRC port), originating from a process owned by the “apache” user.

So we’d been pwned. Now what?

We figured the best way to proceed would be a complete re-install of the operating system. I happened to be free the next day, so I was volunteered to lead in the clean-up duty.

So I drove out to the data center to camp out in the cold air conditioning for a while. I saved away the old infected partitions (we use LVM) and I allocated new space for the fresh install. After I had the OS installed and responding over the network, I went home to finish. I worked frantically over the weekend to restore many of the services that we enjoyed. My priorities were clearly restoring our 250 user accounts and then getting email working (securely). In the process, I gave myself a crash course in LDAP, since that is what we use for user authentication.

Within about 48 hours, we had everything restored except our web pages. After all, we knew the break-in had allowed someone to create a rogue process owned by apache. So we must have had some problem with one of our web-based applications. We did not know whether it was our Drupal-based web page, our web mail client, our wiki, a user application, or something else.

I dug through the log files on the infected partitions, and soon it became apparent that there was a cron job set to run every minute, owned by the ‘apache’ user. The script simply looked to see if its IRC program was running, and if any part of it was damaged or deleted, it would reinstall a new copy of itself somewhere else on the disk… somewhere no one would look, like /var/tmp/.s/something.

Finally, the apache error logs showed what the problem was. It seems that we were running an unpatched version of “RoundCube“, a web-based IMAP e-mail client with a nice AJAX interface. There is a vulnerability in this package that allows a visitor to upload a package to your web server and then run their programs on your server.

Fortunately, the process runs as the “apache” user, and not as “root”. Otherwise, the rogue software would have had permission to do a lot more damage than it actually did. As it stands, the bot simply chatted with a lot of other infected machines. Thankfully, it did not seem interested in the files on our machine.

I learned a lot from this experience. As one admin said, the forced cleanup was a “much-needed enema”, something we had avoided for a long time. As a shared system, system administration was something that was handled by a loose group, and was handed off to new members every year. This break-in was enough to attract our attention, but it was not destructive. And it inspired us to simplify our existing system. And it inspired me to set up nightly backups.