{"id":46,"date":"2009-08-09T09:28:19","date_gmt":"2009-08-09T14:28:19","guid":{"rendered":"http:\/\/blog.alanporter.com\/2009-08-09\/compromised"},"modified":"2009-08-09T09:28:19","modified_gmt":"2009-08-09T14:28:19","slug":"compromised","status":"publish","type":"post","link":"https:\/\/alanporter.com\/blog\/2009\/08\/09\/compromised\/","title":{"rendered":"Compromised?"},"content":{"rendered":"<p>This morning, when I scanned my email, I ran across a report from <code>rkhunter<\/code>, a tool that runs on my web server that periodically checks to see if system files have changed, or if users have been added, what processes are listening on ports, and a litany of other tests to detect whether a system might be infected.  I get these reports fairly often, usually the day after I do an upgrade, or after I add a new user.<\/p>\n<p>However, today&#8217;s email was a little bit alarming.  It said that several files had been changed recently.  And these files all seemed to do with the same sort of things: running stuff behind the scenes, showing library dependencies, elevating privileges.  Basically, these were tools that you would want to modify if you wanted to cover your own tracks.<\/p>\n<pre>\nWarning: The file properties have changed:\n         File: \/bin\/sh\n         Current hash: 23603f77da4ca37705146fd8a4ed951c8b037156\n         Stored hash : 91654fd25d317bd13a65e10d777ac021f4a1a4f6\nWarning: The file properties have changed:\n         File: \/bin\/dash\n         Current hash: 23603f77da4ca37705146fd8a4ed951c8b037156\n         Stored hash : 91654fd25d317bd13a65e10d777ac021f4a1a4f6\n         Current inode: 180336    Stored inode: 180255\n         Current file modification time: 1236603791\n         Stored file modification time : 1213978027\nWarning: The file properties have changed:\n         File: \/usr\/bin\/dpkg\n         Current hash: 4e05d20a4f828c31eb5f6dd9cc5f04d1d6202d0a\n         Stored hash : 09a5bbd0398cc9f02b52440e1241cd942e784a15\n         Current inode: 248598    Stored inode: 246001\n         Current size: 375340    Stored size: 371244\n         Current file modification time: 1236595869\n         Stored file modification time : 1220443410\nWarning: The file properties have changed:\n         File: \/usr\/bin\/dpkg-query\n         Current hash: ff8098920430d399933ee24245748983a0661869\n         Stored hash : 4a1c1226cbe9dd2ddbec7b5652f1fa8aa0b15f09\n         Current inode: 248600    Stored inode: 246003\n         Current file modification time: 1236595869\n         Stored file modification time : 1220443410\nWarning: The file properties have changed:\n         File: \/usr\/bin\/file\n         Current hash: 4ab93b21aaabb405f4bd2e90f16ee5e952aa746b\n         Stored hash : 80dc1735091a4309d23e49ce542c58ddd16163dc\n         Current inode: 245969    Stored inode: 246049\n         Current file modification time: 1244193699\n         Stored file modification time : 1215771733\nWarning: The file properties have changed:\n         File: \/usr\/bin\/ldd\n         Current inode: 248852    Stored inode: 246132\n         Current file modification time: 1233224578\n         Stored file modification time : 1222684817\nWarning: The file properties have changed:\n         File: \/usr\/bin\/perl\n         Current hash: 00d703e925eca6de0c8fc9bd9d4505db4b81ce33\n         Stored hash : efb4a1a3d02798718b7f2bbfea6787dd0de79968\n         Current inode: 245962    Stored inode: 246591\n         Current file modification time: 1246045733\n         Stored file modification time : 1216891204\nWarning: The file properties have changed:\n         File: \/usr\/bin\/sudo\n         Current hash: e649919d4bbc6ac78e38497ca94dc387cc2811a7\n         Stored hash : 49e97774326fc9eb5f7cb680477c1d56f4e28921\n         Current inode: 246543    Stored inode: 246747\n         Current file modification time: 1234840625\n         Stored file modification time : 1220275024\nWarning: The file properties have changed:\n         File: \/usr\/sbin\/cron\n         Current hash: 5efdffc9796731168fb7acc8688c5a02e0da42dd\n         Stored hash : 04924b72b749e8179bb5839bac1a296c7acf93c4\n         Current inode: 245910    Stored inode: 248315\n         Current file modification time: 1242164811\n         Stored file modification time : 1220989568\n\nOne or more warnings have been found while checking the system.\nPlease check the log file (\/var\/log\/rkhunter.log)<\/pre>\n<p>I scanned \/var\/log\/dpkg.log, to see if I had recently done an update, and I did not see anything.  That little paranoid part of my brain started to wake up (those that know me might clarify that when I say &#8220;little&#8221;, I really mean &#8220;big&#8221;).<\/p>\n<p>So briefly, someone had changed the following files on my server.  How could I tell if they were the &#8220;real&#8221; ones or not?<\/p>\n<ul>\n<li>\/usr\/sbin\/cron<\/li>\n<li>\/usr\/bin\/sudo<\/li>\n<li>\/usr\/bin\/perl<\/li>\n<li>\/usr\/bin\/ldd<\/li>\n<li>\/usr\/bin\/file<\/li>\n<li>\/usr\/bin\/dpkg-query<\/li>\n<li>\/usr\/bin\/dpkg<\/li>\n<li>\/bin\/dash<\/li>\n<li>\/bin\/sh<\/li>\n<\/ul>\n<p>What bothered me about this combination was that <code>dpkg<\/code> was in that list, so I could not use any of the apt\/dpkg tools to verify the integrity of my packages against what is published on the Ubuntu mirrors.<\/p>\n<p>So I had to take matters into my own hands.  I went to the <a href=\"http:\/\/packages.ubuntu.com\">Ubuntu packages site<\/a> and searched for the first package, &#8216;cron&#8217;.  From there, I could click on the &#8216;i386&#8217; link to download a local copy of the <code>cron_3.0pl1-100ubuntu2.1_i386.deb<\/code> file onto my laptop (not onto the suspect server).  I extracted the contents using <code>dpkg -x cron_3.0pl1-100ubuntu2.1_i386.deb .<\/code>.  From there, it was pretty easy to compare MD5 checksums of the files.<\/p>\n<pre>\n$ ls\ncron_3.0pl1-100ubuntu2.1_i386.deb\n$ dpkg -x cron_3.0pl1-100ubuntu2.1_i386.deb .\n$ ls\ncron_3.0pl1-100ubuntu2.1_i386.deb  etc  usr  var\n$ md5sum usr\/sbin\/cron\nc1d78d8a9a99b52df8ecba41517ab013  usr\/sbin\/cron\n$<\/pre>\n<p>This checksum matched the one on my server.  So that means my binary files were legitimate (this does not explain how they got updated without leaving a trail in the logs, but that is another issue).<\/p>\n<p>Lather, rinse, and repeat for all of the suspected files.<\/p>\n<p>I hope this little story helps someone else defuse that panicky feeling that sets in when your intrusion detection system sends you an unpleasant email.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This morning, when I scanned my email, I ran across a report from rkhunter, a tool that runs on my web server that periodically checks to see if system files have changed, or if users have been added, what processes are listening on ports, and a litany of other tests to detect whether a system [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3,12],"tags":[],"class_list":["post-46","post","type-post","status-publish","format-standard","hentry","category-software","category-security","count-0","even alt","author-alan","last"],"_links":{"self":[{"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/posts\/46","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/comments?post=46"}],"version-history":[{"count":0,"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/posts\/46\/revisions"}],"wp:attachment":[{"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/media?parent=46"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/categories?post=46"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/tags?post=46"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}