free software
Linux, open source software, tips and tricks.
Compromised?
3This morning, when I scanned my email, I ran across a report from rkhunter, a tool that runs on my web server that periodically checks to see if system files have changed, or if users have been added, what processes are listening on ports, and a litany of other tests to detect whether a system might be infected. I get these reports fairly often, usually the day after I do an upgrade, or after I add a new user.
However, today’s email was a little bit alarming. It said that several files had been changed recently. And these files all seemed to do with the same sort of things: running stuff behind the scenes, showing library dependencies, elevating privileges. Basically, these were tools that you would want to modify if you wanted to cover your own tracks.
Warning: The file properties have changed:
File: /bin/sh
Current hash: 23603f77da4ca37705146fd8a4ed951c8b037156
Stored hash : 91654fd25d317bd13a65e10d777ac021f4a1a4f6
Warning: The file properties have changed:
File: /bin/dash
Current hash: 23603f77da4ca37705146fd8a4ed951c8b037156
Stored hash : 91654fd25d317bd13a65e10d777ac021f4a1a4f6
Current inode: 180336 Stored inode: 180255
Current file modification time: 1236603791
Stored file modification time : 1213978027
Warning: The file properties have changed:
File: /usr/bin/dpkg
Current hash: 4e05d20a4f828c31eb5f6dd9cc5f04d1d6202d0a
Stored hash : 09a5bbd0398cc9f02b52440e1241cd942e784a15
Current inode: 248598 Stored inode: 246001
Current size: 375340 Stored size: 371244
Current file modification time: 1236595869
Stored file modification time : 1220443410
Warning: The file properties have changed:
File: /usr/bin/dpkg-query
Current hash: ff8098920430d399933ee24245748983a0661869
Stored hash : 4a1c1226cbe9dd2ddbec7b5652f1fa8aa0b15f09
Current inode: 248600 Stored inode: 246003
Current file modification time: 1236595869
Stored file modification time : 1220443410
Warning: The file properties have changed:
File: /usr/bin/file
Current hash: 4ab93b21aaabb405f4bd2e90f16ee5e952aa746b
Stored hash : 80dc1735091a4309d23e49ce542c58ddd16163dc
Current inode: 245969 Stored inode: 246049
Current file modification time: 1244193699
Stored file modification time : 1215771733
Warning: The file properties have changed:
File: /usr/bin/ldd
Current inode: 248852 Stored inode: 246132
Current file modification time: 1233224578
Stored file modification time : 1222684817
Warning: The file properties have changed:
File: /usr/bin/perl
Current hash: 00d703e925eca6de0c8fc9bd9d4505db4b81ce33
Stored hash : efb4a1a3d02798718b7f2bbfea6787dd0de79968
Current inode: 245962 Stored inode: 246591
Current file modification time: 1246045733
Stored file modification time : 1216891204
Warning: The file properties have changed:
File: /usr/bin/sudo
Current hash: e649919d4bbc6ac78e38497ca94dc387cc2811a7
Stored hash : 49e97774326fc9eb5f7cb680477c1d56f4e28921
Current inode: 246543 Stored inode: 246747
Current file modification time: 1234840625
Stored file modification time : 1220275024
Warning: The file properties have changed:
File: /usr/sbin/cron
Current hash: 5efdffc9796731168fb7acc8688c5a02e0da42dd
Stored hash : 04924b72b749e8179bb5839bac1a296c7acf93c4
Current inode: 245910 Stored inode: 248315
Current file modification time: 1242164811
Stored file modification time : 1220989568
One or more warnings have been found while checking the system.
Please check the log file (/var/log/rkhunter.log)
I scanned /var/log/dpkg.log, to see if I had recently done an update, and I did not see anything. That little paranoid part of my brain started to wake up (those that know me might clarify that when I say “little”, I really mean “big”).
So briefly, someone had changed the following files on my server. How could I tell if they were the “real” ones or not?
- /usr/sbin/cron
- /usr/bin/sudo
- /usr/bin/perl
- /usr/bin/ldd
- /usr/bin/file
- /usr/bin/dpkg-query
- /usr/bin/dpkg
- /bin/dash
- /bin/sh
What bothered me about this combination was that dpkg was in that list, so I could not use any of the apt/dpkg tools to verify the integrity of my packages against what is published on the Ubuntu mirrors.
So I had to take matters into my own hands. I went to the Ubuntu packages site and searched for the first package, ‘cron’. From there, I could click on the ‘i386’ link to download a local copy of the cron_3.0pl1-100ubuntu2.1_i386.deb file onto my laptop (not onto the suspect server). I extracted the contents using dpkg -x cron_3.0pl1-100ubuntu2.1_i386.deb .. From there, it was pretty easy to compare MD5 checksums of the files.
$ ls cron_3.0pl1-100ubuntu2.1_i386.deb $ dpkg -x cron_3.0pl1-100ubuntu2.1_i386.deb . $ ls cron_3.0pl1-100ubuntu2.1_i386.deb etc usr var $ md5sum usr/sbin/cron c1d78d8a9a99b52df8ecba41517ab013 usr/sbin/cron $
This checksum matched the one on my server. So that means my binary files were legitimate (this does not explain how they got updated without leaving a trail in the logs, but that is another issue).
Lather, rinse, and repeat for all of the suspected files.
I hope this little story helps someone else defuse that panicky feeling that sets in when your intrusion detection system sends you an unpleasant email.
Plant my key
4Here’s a neat little script that I developed at “the oven place”. It “plants” my SSH key on a remote machine, so I can get in later without entering a password.
#!/bin/sh
user_at_machine=$1
# plant my SSH key on the target machine
cat $HOME/.ssh/id_rsa.pub | \
ssh $user_at_machine \
"if [ ! -d .ssh ] ; then mkdir -m 700 .ssh ; fi ; \
cat >> .ssh/authorized_keys"
All of the action takes place in one single line (which I broke up here so it would wrap OK). It takes your public key from the machine you’re on, and it pipes it into an SSH session, which you’ll have to type your password into. On the remote machine, it creates a .ssh directory if it needs to, and then it appends your public key to an authorized_keys file.
The next time you log in to that remote machine, you will not need to enter your password.
Counting down with ‘pv’
0Here’s a neat tool.
Ever start some long task, and wonder how much longer it has to go? There’s a small utility called pv (short for “pipe viewer”) that counts time and bytes through a pipe, and it shows a nice progress bar.
Here’s how I used it to watch a very large file being compressed (note the use of the most excellent lzma compression utility.
$ pv < winxp.vdi | lzma > winxp.vdi.lzma 1.79GB 0:21:50 [1.49MB/s] [===> ] 10% ETA 3:11:02
Note that if I had used a different pipe notation, pv would not have been able to read the input file size, and therefore it could not make estimates of remaining time. So instead, it shows you the “Knight Rider” scanning eye for progress.
$ cat winxp.vdi | pv | lzma > winxp.vdi.lzma 5.48MB 0:00:04 [1.27MB/s] [ <=> ]
Nice tool. Thanks, Andrew!
Save that thought…
1At work, our build process can produce thousands of lines of code, and sometimes the important bits can slide off of the terminal’s scrollback buffer.
So I created an alias that logs all output of the shell. It’s nothing fancy… it’s just a call to script with an argument to put the time and date in the filename. But it goes a step further, actually exec-ing the script command, so you do not have to exit twice when you’re done (once to exit the script, and once to exit the original shell).
alias log='exec script "/home/alan/logs/terminal-$(date +%Y%m%d-%H%M%S)-$$.log"'
Still, this requires you to think about logging before you start your work.
Since I normally launch my shell windows from a Gnome application launcher icon, I decided to modify that launcher so that EVERYTHING is logged.
On Gnome, right-click on the panel at the top and select “+ Add to panel…”. Then select “application launcher” and pick out the gnome-terminal from the menus. You’ll have a little terminal icon on the top Gnome panel.
Right-click on that terminal icon and select “properties”. It should say “gnome-terminal”. Change it to the following:
sh -c "exec gnome-terminal -x script $HOME/logs/terminal-$(date +%%Y%%m%%d-%%H%%M%%S)-$$.log"
The double-percents are so the launcher will not interpret them.
Now, when you click on that application launcher icon, a new terminal window will open, and all output to that terminal will be logged.
Firefox: invalid or unsupported form of compression
0Occasionally, I have been seeing the following error in Firefox (v3.0.11).
The page you are trying to view cannot be shown because it uses an invalid or unsupported form of compression.
It occurs frequently when I am using a SOCKS proxy. One web site which happens to exhibit the problem more than others is Bruce Schneier’s blog.
The work-around that seems to work for me is to do a “Tools / Clear Private Data…” and select Cache, Cookies and Offline Website Data.
There’s a lively discussion on the Mozilla support forum, but no one seems to know what the source of the problem is.
Two heads are better than one
0At work, I got a new PC with two monitors, and I have really enjoyed having the extra screen real estate.
When I worked from home this week, I decided that I needed to try hooking an external monitor to my HP laptop. I was happy that, unlike the last time I tried this a few years ago, it “Just Worked” ™ without any special calesthenics on my part.
I wondered why Gnome decided to put the panels (the bars at the top and the bottom) on the external monitor instead of on the laptop display. To me, it seemed more natural to have the “start” menu (actually, the “Applications, Places, System” tri-menu) on the external display instead of right in front of me.
It turns out that this is really easy to fix. Gnome has this configuration tool that is roughly equivalent to the Windows registry… yuck. You can access it using either gconf-editor or gconftool-2. Inside that deep mine of settings, there is one jewel that tells where the panels should go. So I wrote a quick script to move the panels from one monitor to the other and back.
#!/bin/bash
m=$1
if [[ ( ! -z "$m" ) && ( ( $m -eq 1 ) || ( $m -eq 0 ) ) ]]
then
menu="/apps/panel/toplevels"
panels=$(gconftool-2 --dump $menu | \
grep '<key>.*/monitor</key>' | \
sed -e 's/^.*<key>//g' -e 's/</key>.*$//g')
for p in $panels
do
gconftool-2 --set "$menu/$p" --type integer "$m"
done
else
echo "usage --> $0 [ 0 | 1 ]"
echo "moves gnome panels to monitor 0 (VGA) or 1 (LCD)"
fi
exit 0
The Quick Lounge
0In my line of work, I occasionally have the luxury of working from home. When that happens, I prefer to run all of my applications on my desktop PC at the office, and I use “NX” to connect remotely. Specifically, I use the NoMachine NX server on my Ubuntu PC at work (although I am considering trying the open source “freenx” server), and at home I use the “qtnx” client on my laptop, which also runs Ubuntu.
This works pretty well, except my desktop environment does not like the stress of changing resolutions back and forth. It seems that the “panels” in Gnome get confused about where to place the different applets and widgets, since at one time it may have a lot of room, and at another time it may be more cramped.
I got tired of scooting my quick-launch icons around one-by-one, and so I was looking for a container where I could keep them together and move them around as a group. I tried the “drawer” applet, which is available in Ubuntu by default, but that was not quite right. I don’t want to open and close the drawer every time I launch something… I just want the icons to always be there in a group.
I found an applet called “quick-lounge-applet” which really fits the bill. One wonders whether the author meant to say “quick launch”, but perhaps had a poor grasp of English. Either way, this little applet does a great job of keeping my quick-launch icons together in a group, and it can be moved around easily.
After installing the applet (using the normal apt-get install quick-lounge-applet, I found that it was not listed in the “+ Add to panel…” menu. Apparently, Gnome needs to be prodded before it recognises newly-installed applets. There is a simple work-around. Simply re-start the service that keeps track of that stuff: killall bonobo-activation-server. The service will re-start, and there will be a new entry in the “+ Add to panel…” menu called “Launchers List”.
Now, if you’ll excuse me, I am in a hurry. I have some quick lounging to do.
Firefox plug-in: SyncPlaces
0Early last year, I decided that my tired old HP laptop wanted to retire, and I started shopping for a new one. However, before I could find a suitable replacement, I discovered the Asus Eee PC, and I knew that I had to have one.
It did not make a lot of sense to buy a new laptop and a new Eee PC as well, so I held off buying a laptop. Over time, the Eee PC became my primary machine. Sometimes, I would plug in an external monitor and mouse and keyboard. And other times, I would just use it by itself. After a while, I migrated all of my old stuff off of the laptop and onto a mini- file server, and I eventually left the tired old laptop powered off.
I started using the HP laptop again when I started working from home, but I never really installed anything other than NX. On a whim, I installed the latest Ubuntu, Jaunty Jackalope (9.04), and that really breathed new life into the tired old laptop.
So now I find myself strattling the fence, sometimes using the tired old (but rejuvinated) HP laptop, and sometimes using the Eee PC. Since I keep most of my important stuff on an encrypted thumb drive, it was pretty easy to switch back and forth.
But there was one thing missing… my Firefox bookmarks.
I don’t like the idea of storing my stuff (tax records, email, bookmarks, or anything else) on a site like Google or xmarks (formerly foxmarks). So I went looking for a plug-in that would allow me to synchronize my bookmarks among multiple machines, but use my own server for storage.
SyncPlaces does a pretty good job of that.
It can sync using FTP (yuck) or https (yay) to a WebDAV-enabled server. It only took a few minutes to figure out WebDAV, and pretty soon I had the same bookmarks on the HP laptop and on the Eee PC.
Pidgin and Yahoo
2I ran into a strange bug with pidgin where I could not log into Yahoo. Strace did not shed any light on the problem:
gettimeofday({1245688276, 23774}, NULL) = 0
open("/home/alan/.gnome2/nautilus-sendto/spool", O_RDONLY...
fstat64(7, {st_mode=S_IFDIR|0755, st_size=4096, ...
getdents(7, /* 3 entries */, 4096) = 48
getdents(7, /* 0 entries */, 4096) = 0
close(7) = 0
read(3, 0x93c4508, 4096) = -1 EAGAIN ...
gettimeofday({1245688276, 24697}, NULL) = 0
poll([{fd=4, events=POLLIN}, {fd=3, events=POLLIN},...
Instead, the answer came from a blog post here.
Yahoo changed their login protocol. Pidgin released an update. Ubuntu did not propagate the fix, apparently because it was not a security bug, but new functionality.
The pidgin developers have released an Ubuntu package in their PPA (Personal Package Archive). See the details at the pidgin web site here.
South East Linux Fest (SELF)
0I spent the weekend in Clemson SC at the first South East Linux Fest.
It was a great chance to geek out with a bunch of Linux enthusiasts, as well as some of the movers and shakers in the industry.
We enjoyed presentations on topics that ranged from SQLite and the fsync() bug to Asterisk, Open Street Map and the Ubuntu kernel. These guys provided a clear perspective of how the open source world organizes and operates on a daily basis. And their projects just glowed with coolness, which created this viral kind of buzz around the entire event, whether you were a casual Linux user or a hardcore developer.
In the vendor area, we were wooed by several distros, a handful of businesses that sell services around open source software, and some content providers (that is, podcasters and “nerdcore” rappers). We were treated to several raffles… woot!
Mad props go out to the planners of the event, which seemed to go off without a hitch.