Archive for March, 2009

23 cubes in a bottle

Normally, I do not allow my children to have soft drinks. Occasionally, I will let them have a diet soft drink, just so it won’t be considered a forbidden fruit. But most of the time, they drink milk and water. Sometimes they have diluted fruit juice.

Today, they asked if they could use their own money to buy a soft drink from a vending machine. So that’s what we did. I was shocked to see how much sugar was in there.

The label says that there is 34 grams of sugar per cup of Sunkist. And then it goes on to say that there are 2.5 cups in the bottle. Apparently, you are supposed to share this drink with 1.5 friends.

So that means that one 20-ounce bottle contains 84 grams of sugar. That does not sound like much… after all, a gram is pretty small. You know, they measure stuff like medicine using grams, right?

So we got out the postage scale and started stacking up sugar cubes until we had 84 grams. How many cubes would you guess? I would not have guessed it… 23 sugar cubes!

23 cubes

Now imagine fixing yourself a cup of coffee and dropping 23 sugar cubes into the cup. Yuck!

TransUnion — free credit reports, or not?

This evening, something (I don’t know what) inspired me to check my credit report. I know that you are supposed to be able to get a free credit report from each of the “big three” (Equifax, Experian, and TransUnion) every year. So I started by going to the Federal Trade Commission’s web site. I know it’s easy to be scammed, so I wanted to start with the source.

They direct you to a site that sounds like a scam (, but is not. You can go through the government-designed (usability-be-damned) web interface that simply redirects you to the web sites of the big three. At some points in the process, my Firefox NoScript plug-in started complaining about potential cross-site scripting problems, and some scripts on a site called “”. But a few minutes later, I got reports from Experian and Equifax.

Transunion, however, gave me this lame excuse.

Unable to Confirm Identity
As a security precaution, we cannot provide online delivery

What’s happening…

We apologize for the inconvenience, but based on the
information provided, we have been unable to confirm
your identity.

Because the protection of your security and privacy
are of the utmost importance to us, we cannot provide
your Personal Credit Report online.

What you should do…

* Please return to to
order your Personal Credit Report by phone or mail.

This was after I entered a username and a password (generated and stored in 2007 using PasswordSafe).

In 2007, I did send a written request to TransUnion for the credit report by mail… I don’t recall ever receiving one from them.


I just spent the entire weekend re-building a server for the Triangle Linux Users Group.

We first noticed that something was wrong when the machine stopped responding over the network. A couple of our admins took a trip to the data center and noticed that we had a firehose of data on port 6667 (an IRC port), originating from a process owned by the “apache” user.

So we’d been pwned. Now what?

We figured the best way to proceed would be a complete re-install of the operating system. I happened to be free the next day, so I was volunteered to lead in the clean-up duty.

So I drove out to the data center to camp out in the cold air conditioning for a while. I saved away the old infected partitions (we use LVM) and I allocated new space for the fresh install. After I had the OS installed and responding over the network, I went home to finish. I worked frantically over the weekend to restore many of the services that we enjoyed. My priorities were clearly restoring our 250 user accounts and then getting email working (securely). In the process, I gave myself a crash course in LDAP, since that is what we use for user authentication.

Within about 48 hours, we had everything restored except our web pages. After all, we knew the break-in had allowed someone to create a rogue process owned by apache. So we must have had some problem with one of our web-based applications. We did not know whether it was our Drupal-based web page, our web mail client, our wiki, a user application, or something else.

I dug through the log files on the infected partitions, and soon it became apparent that there was a cron job set to run every minute, owned by the ‘apache’ user. The script simply looked to see if its IRC program was running, and if any part of it was damaged or deleted, it would reinstall a new copy of itself somewhere else on the disk… somewhere no one would look, like /var/tmp/.s/something.

Finally, the apache error logs showed what the problem was. It seems that we were running an unpatched version of “RoundCube“, a web-based IMAP e-mail client with a nice AJAX interface. There is a vulnerability in this package that allows a visitor to upload a package to your web server and then run their programs on your server.

Fortunately, the process runs as the “apache” user, and not as “root”. Otherwise, the rogue software would have had permission to do a lot more damage than it actually did. As it stands, the bot simply chatted with a lot of other infected machines. Thankfully, it did not seem interested in the files on our machine.

I learned a lot from this experience. As one admin said, the forced cleanup was a “much-needed enema”, something we had avoided for a long time. As a shared system, system administration was something that was handled by a loose group, and was handed off to new members every year. This break-in was enough to attract our attention, but it was not destructive. And it inspired us to simplify our existing system. And it inspired me to set up nightly backups.

Go to Top