This morning, when I scanned my email, I ran across a report from rkhunter, a tool that runs on my web server that periodically checks to see if system files have changed, or if users have been added, what processes are listening on ports, and a litany of other tests to detect whether a system might be infected. I get these reports fairly often, usually the day after I do an upgrade, or after I add a new user.

However, today’s email was a little bit alarming. It said that several files had been changed recently. And these files all seemed to do with the same sort of things: running stuff behind the scenes, showing library dependencies, elevating privileges. Basically, these were tools that you would want to modify if you wanted to cover your own tracks.

Warning: The file properties have changed:
         File: /bin/sh
         Current hash: 23603f77da4ca37705146fd8a4ed951c8b037156
         Stored hash : 91654fd25d317bd13a65e10d777ac021f4a1a4f6
Warning: The file properties have changed:
         File: /bin/dash
         Current hash: 23603f77da4ca37705146fd8a4ed951c8b037156
         Stored hash : 91654fd25d317bd13a65e10d777ac021f4a1a4f6
         Current inode: 180336    Stored inode: 180255
         Current file modification time: 1236603791
         Stored file modification time : 1213978027
Warning: The file properties have changed:
         File: /usr/bin/dpkg
         Current hash: 4e05d20a4f828c31eb5f6dd9cc5f04d1d6202d0a
         Stored hash : 09a5bbd0398cc9f02b52440e1241cd942e784a15
         Current inode: 248598    Stored inode: 246001
         Current size: 375340    Stored size: 371244
         Current file modification time: 1236595869
         Stored file modification time : 1220443410
Warning: The file properties have changed:
         File: /usr/bin/dpkg-query
         Current hash: ff8098920430d399933ee24245748983a0661869
         Stored hash : 4a1c1226cbe9dd2ddbec7b5652f1fa8aa0b15f09
         Current inode: 248600    Stored inode: 246003
         Current file modification time: 1236595869
         Stored file modification time : 1220443410
Warning: The file properties have changed:
         File: /usr/bin/file
         Current hash: 4ab93b21aaabb405f4bd2e90f16ee5e952aa746b
         Stored hash : 80dc1735091a4309d23e49ce542c58ddd16163dc
         Current inode: 245969    Stored inode: 246049
         Current file modification time: 1244193699
         Stored file modification time : 1215771733
Warning: The file properties have changed:
         File: /usr/bin/ldd
         Current inode: 248852    Stored inode: 246132
         Current file modification time: 1233224578
         Stored file modification time : 1222684817
Warning: The file properties have changed:
         File: /usr/bin/perl
         Current hash: 00d703e925eca6de0c8fc9bd9d4505db4b81ce33
         Stored hash : efb4a1a3d02798718b7f2bbfea6787dd0de79968
         Current inode: 245962    Stored inode: 246591
         Current file modification time: 1246045733
         Stored file modification time : 1216891204
Warning: The file properties have changed:
         File: /usr/bin/sudo
         Current hash: e649919d4bbc6ac78e38497ca94dc387cc2811a7
         Stored hash : 49e97774326fc9eb5f7cb680477c1d56f4e28921
         Current inode: 246543    Stored inode: 246747
         Current file modification time: 1234840625
         Stored file modification time : 1220275024
Warning: The file properties have changed:
         File: /usr/sbin/cron
         Current hash: 5efdffc9796731168fb7acc8688c5a02e0da42dd
         Stored hash : 04924b72b749e8179bb5839bac1a296c7acf93c4
         Current inode: 245910    Stored inode: 248315
         Current file modification time: 1242164811
         Stored file modification time : 1220989568

One or more warnings have been found while checking the system.
Please check the log file (/var/log/rkhunter.log)

I scanned /var/log/dpkg.log, to see if I had recently done an update, and I did not see anything. That little paranoid part of my brain started to wake up (those that know me might clarify that when I say “little”, I really mean “big”).

So briefly, someone had changed the following files on my server. How could I tell if they were the “real” ones or not?

  • /usr/sbin/cron
  • /usr/bin/sudo
  • /usr/bin/perl
  • /usr/bin/ldd
  • /usr/bin/file
  • /usr/bin/dpkg-query
  • /usr/bin/dpkg
  • /bin/dash
  • /bin/sh

What bothered me about this combination was that dpkg was in that list, so I could not use any of the apt/dpkg tools to verify the integrity of my packages against what is published on the Ubuntu mirrors.

So I had to take matters into my own hands. I went to the Ubuntu packages site and searched for the first package, ‘cron’. From there, I could click on the ‘i386’ link to download a local copy of the cron_3.0pl1-100ubuntu2.1_i386.deb file onto my laptop (not onto the suspect server). I extracted the contents using dpkg -x cron_3.0pl1-100ubuntu2.1_i386.deb .. From there, it was pretty easy to compare MD5 checksums of the files.

$ ls
cron_3.0pl1-100ubuntu2.1_i386.deb
$ dpkg -x cron_3.0pl1-100ubuntu2.1_i386.deb .
$ ls
cron_3.0pl1-100ubuntu2.1_i386.deb  etc  usr  var
$ md5sum usr/sbin/cron
c1d78d8a9a99b52df8ecba41517ab013  usr/sbin/cron
$

This checksum matched the one on my server. So that means my binary files were legitimate (this does not explain how they got updated without leaving a trail in the logs, but that is another issue).

Lather, rinse, and repeat for all of the suspected files.

I hope this little story helps someone else defuse that panicky feeling that sets in when your intrusion detection system sends you an unpleasant email.