CarolinaCon 8

Once again, it’s time for a weekend of frivolity and technology that we call “CarolinaCon”. While I was sure that doing so would put me on some secret list of persons-to-watch, I made plans to attend.

As usual, the Con spanned the weekend, from Friday evening until Sunday evening. Here’s my impressions.


  • 7:00pm – Identifying Cyber Warriors (Tom Holt / Professor Farnsworth)
    The professor always does such a good job of easing the crowd out of reality and into the “Con-zone”. His presentations are academic, but touch on hacker culture. This year, he gave questionnaires to college students to determine correlations between their attitudes on protesting and hactivism across two dimensions: domestic-vs-foreign and physical-vs-virtual.
  • 8:00pm – Bypassing Android Permissions (Georgia Weidman)
    Georgia explained how Android phone apps implement “intents” or services, and then how to inspect an Android app to see its contents. Using these two pieces, she was able to write unprivileged apps that use another app’s higher-priviledge services.
  • 9:00pm – Intro to Hacking Bluetooth (ronin)
    Ronin has assembled a wide array of open source Bluetooth tools into a Linux distribution that he calls “Blubuntu”. He showed the basic usage of some of these tools, with some good background info on how the protocols work. A good overview talk!
  • 10:00pm – F-ing the Friendly Skies (Deviant Ollam)
    The original plan was to end the Friday night session with a round of Hacker Trivia, but the schedule was abruptly changed to include this entertaining talk from Deviant Ollam, whose talks usually center on lock picking and air travel (and they usually include alcohol). This year’s presentation did not disappoint — the topic was the “Mile High Club”, with a complete survey of logistics, opportunity, and suggestions for flight timings and choice of aircraft.


  • 10:00am – DevHack: Pre-Product Exploitation (Snide)
    Once they cranked up the volume on Snide’s mic, he walked us through several ideas for planting malware into a software development environment, implanting your payload at the source.
  • 11:00am – Malware Retooled (Big-O)
    This talk discussed how we can watch what malware DOES rather than trying to match against signatures. Big-O showed some nice visualization tools, like thread graphs.
  • lunch break
  • 1:00pm – Inside Jobs: Stealing Sensitive Data and Intellectual Property (Vic Vandal and emwav)
    Vic and emwav enumerated several ways that companies and employees can escalate the arms race… employees can steal proprietary information, and employers can make it difficult. The bottom line for me was that this arms race is stacked in favor of the attacker.
  • 2:00pm – Project Byzantium: Improvisable Ad-Hoc Wireless Mesh Networking for Disaster Zones (Sitwon The Pirate and The Doctor)
    Apparently, someone told this group that CarolinaCon was a costume party. The pirate and the doctor win the prize for best costumes. But pay closer attention. These guys are smart — they have scoured the internet for cool mesh networking tools, and they have packaged them into a LiveCD distro called “Byzantium”. Imagine a disaster or political unrest scenario, where you would like to mesh together a group of users and share an internet connection. Keep an eye on this fledgling project.
  • 3:00pm – Hacking as an Act of War (G. Mark Hardy)
    Gmark has been keeping his eye on the geopolitical landscape and how different nations approach info security. He has insights into the capabilities and motivations of the different players, and he observes recent cyber-attacks that have shaped the new balance of powers. Where do we draw the line between hacking and warfare?
  • 4:00pm – Big Bang Theory: The Evolution of Pentesting High Security Environments (Joe McCray)
    Joe warns corporate America that we should stop focusing on the vulnerabilities, stop patting ourselves on the back when we’ve checked all of the patch check boxes, and instead focus on why an attacker would be interested in their companies to begin with, and prioritize based on the value of the assets (illustrated with a colorful analogy about driving through the ‘hood).
  • Dinner Break
  • 7:00pm – Spyometrics: New World of Biometric Surveillance (Dr. Noah Schiffman, aka Lo-Res)
    This talk had a lot of promise, because the subject matter is wide open for thought-provoking stories. But I felt like this talk left more loose ends than tied-up ones.
  • 8:00pm – Dr. Tran goes to Switzerland (Dr. Tran)
    One of the best talks of the show, and it was not even about security! Dr Tran recently moved to Zurich, and he recorded his impressions and shared them with the Con attendees.
  • 9:00pm – Hacker Trivia
    Much like hacker trivia in previous years, but I noticed three differences. The game was more lightly attended than at previous cons. Many of the questions went unanswered (meaning Vic had to take a drink). And most disappointing, Al did not emcee.


  • 10:00am – Attacking CAPTCHAs (Gursev Singh Kalra)
    This talk was canceled. Instead, we heard from some UNC-Charlotte students on the accomplishments of their hacking competition team.
  • 11:00am – Patch to Pwned: Exploiting Firmware Patching to Compromise MFP Devices (Deral Heiland)
    Deral Heiland has made a career out of keeping printer manufacturers on their toes. This time, he decomposes a firmware update package for Xerox printers, and he creates his own update that includes his “modifications”.
  • Lunch Break
  • 1:00pm – Hacking your Mind and Emotions (Branson Matheson)
    Branson shows how easily we can be socially engineered. It happens every day, from advertisers, authority figures and administrators. He shows us how to recognize when we’re being manipulated, and he encourages us to know our rights and responsibilities so we can limit our exposure.
  • 2:00pm – It’s 2012 and My Network Got Hacked (Omar Santos)
    Case studies of real-life compromises, in spite of the sophisticated defenses employed. Omar discussed the challenges that are encountered by large organizations with wide networks and hundreds of assets to manage.
  • 3:00pm – Declarative Web Security: DEP for the Web (Steve Pinkham)
    Modern browsers are starting to include advanced policy engines that allow web sites to declare rules, such as “only run javascript from this host”, in an attempt to restrict what potential malware can do.
  • 4:00pm – Raspberry Pi’s Impact on Hacking (DJ Palombo)
    OK, the RaspPi is a cheap small computer. We get that. DJ Palombo seems to think that the revolutionary concept is that its low profile and disposability make it a good “bug” or or hidden node for hacking.

Thanks to the organizers and presenters for another memorable Con!

The Pirate and the Doctor

The Pirate and the Doctor discuss Byzantium

CarolinaCon 7

CarolinaCon 7

For the fourth year in a row, I treated myself to a weekend of security training and adolescent tom-foolery called “CarolinaCon”. The event was sponsored by the local chapter of 2600. While the presentations covered a range of topics that would interest any security professional, the tone was that of a college party. It’s fun.

The tagline for this year’s event was “A weekend of brainstorming (and by brainstorming, we mean drinking)”.

The Con started on Friday after work, and it continued until supper time on Sunday. While the speakers were giving their presentations in the main room, there were side events going on, too: a “lock pick village”, a weekend-long game of capture-the-flag, and a crypto challenge. It was hard to decide what to do, but I stuck with the presentations.

Here’s my thoughts on the various presentations.


  • 7pm – Tales from the Crypto (G. Mark Hardy)
    Security conferences are known for having puzzle challenges in their admission badges, and Gmark is the twisted mind behind several of them. He gave us an overview of classical (pen-and-paper) cryptography techniques, and showed how these tricks were incorporated into past Con badges. Finally, he issued a challenge to crack the code buried in the CarolinaCon 7 badge.
  • 8pm – How to Own and Protect Your Office Space (Dr. Tran)
    Dr Tran, from the ToooL team (of lock-pickers) showed common weaknesses in office physical security.
  • 9pm – Serial Killers: USB as an Attack Vector (Nick Fury)
    Nick likes to build stuff, and this time he built a small USB device that claims to be a keyboard, or a mouse, or both.  And it can type stuff and click on stuff, just like a real mouse.
  • 10pm – Hacker Trivia (wxs, Vic Vandal, AlStrowger)
    Hacker Trivia is CarolinaCon’s version of Jeopardy, where the questions are about hacker topics, and the answers will win you prizes (or crap).  I won a lock pick set and a “vintage” (1990) T-shirt that says “Cray Ada 3.0”.


  • 10am – Music and Audio Production with FOSS (Adam Drew)
    Some of the audience members partied a little hard last night, so Adam woke them up with some noise/music that he produced using 100% free and open source software.  This was my favorite talk of the Con… but it had nothing to do with security.
  • 11am – PIG: Finding Truffles Without Leaving A Trace (Ryan Linn)
    Ryan has given talks before about using metasploit to find vulnerabilities in a network.  This time, he showed a plug-in that he calls “PIG”, which does Passive Information Gathering… just listening to what everyone else is willfully broadcasting.
  • 1pm – If You Own a Multi-Function Printer than I Own You (Deral Heiland)
    Every year, Deral brings some new way to crack an egg. This time, it was office printers with shoddy security. It’s as if the printer companies got their embedded software guys to do the web interfaces, too! [This, of course, describes MY day job pretty well – I should take notes.]
  • 2pm – Yara and Python: The Malware Detection Dynamic Duo (mjg = Michael Goffin)
    Yara is a tool that can generate a “signature” to help classify malware. I sort of faded out during this one. Hey, it was just after lunch.
  • 3pm – There’s An App For That: Pentesting Moble (Joe McCray)
    We always enjoy seeing Joe Mac, and so we were happy to learn that he could fill a last-minute vacancy with his talk on installing your favorite hacking tools on a mobile phone platform.
  • 4pm – Fun with SSH Honeypotting (Chris Teodorski)
    Best security talk of the Con, starting with a simple tool that looks like a vulnerable SSH shell account, but is really just a frustrating maze that records an intruder’s every move.  But not stopping there, Chris went on to profile his intruders by analyzing the rootkit they used, and then by chatting them up in their IRC forum.
  • 7pm – Why Your Password Policy Sucks (purehate = Martin Bos)
    Using statistics from databases of passwords that have previously been compromised, Martin narrows down the search space significantly, speeding up the time to guess new passwords. For example, a large percentage of passwords are in the form “(some word) + (one number digit)”.  Password policies that force you to use a digit only encourage users to choose one that follows this same template.
  • 8pm – Mackerel: A Progressive School of Cryptographic Thought (Justin Troutman)
    There’s a dud at every Con, and this one was it. At first, it looked as though Justin had promise, with his good-ole-boy charm and his trippy slides (done with prezi). But in the end, it was an hour of techno-babble with over-animated slide transitions and no practical substance, a rehash of an academic paper, and a marketing pitch for his consulting business. Boo! [1]
  • 9pm – TTL of a Penetration (Branson Matheson)
    Branson argues that it is more important to react quickly to the inevitable attack than to try to thwart attacks in the first place. Spoken like a real Windows user.
  • 10pm – Hacker Trivia (Vic Vandal, wxs, AlStrowger)
    The Saturday night episode of Hacker Trivia was a little harder-edged than Friday night’s game. The staff spent much more of their time enforcing (and changing) the rules, and generally arguing with each other. Al (the MC) commented that this was like a game of “CalvinBall”. The cheezy prizes were replaced by unique “CarolinaCon 7” shot glasses, awarded (full) to the correct answerers.


  • 10am – logstash: Open Source Log and Event Management (Jordan Sissel)
    Most of us just print logs to a file. Jordan Sissel manages logs like Sorcerer Mickey manages his broom minions. His open source tools consolidate, format and distribute log messages in a very flexible way. And he even has stickers with cartoon logs and beavers to boot!
  • 11am – Dissecting the Hack: Malware Analysis 101 (Gerry Brunelle)
    When a machine is hacked, many people want to just wipe it clean and reinstall the OS. Gerry tells us not to… save that image and study what the malware is doing. He showed several techniques for learning what a piece of malware is doing. This was primarily a Windows-based talk, and much of the work was done using debuggers that disassembled the object code.
  • 1pm – Security Lessons from Cracking Enigma (Lisa Lorenzin)
    Lisa walked us through the history of the German Enigma machine, and the extraordinary tale of how the code was broken by the British and the Allies.
  • 2pm – Hack from a Library with Katana (JP “ronin” Dunning)
    A pretty straightforward talk about a toolkit that can be installed on a USB flash drive, and about the many places where one might find an unattended PC (McDonald’s drive-thru??).
  • 3pm – The Art of (Cyber) War (wxs = Wes Shields)
    It’s time to put on our tin foil hats… Wes is about to tell us about the next generation of bad guys. While the media warns us of APT’s (Advanced Persistent Threats), Wes prefers to call them DHA’s (Determined Human Adversaries). They are organized, focused, funded, and take a long-term view of their attack. So determine what their motivation is, and how to counter them. Apologies for numerous Sun Tsu quotes.
  • 4pm – Pwning Capture the Flag: Step by Step (David Burt)
    David Burt set up the playing field for the weekend-long capture-the-flag game, and so as the Con came to a close, he spilled the beans.  Five machines were set up on a private network, two Linux machines and three Windows. Each had vulnerabilities and “flag files” hidden on them. Find the files, and cut-and-paste their contents into the scoreboard web page, and you pwn this game.

All in all, it was a very fun, but exhausting, weekend.  I learned a lot… some of it security-related, and some… well… Thanks to the organizers for another fun weekend.  I am glad that they only hold this thing once per year.

[1] I am also willing to accept an alternative explanation – that I am too dumb to understand what Justin was talking about.


Encrypting your entire hard disk (almost)

I have a small netbook that I use when I travel, one of the original Asus EeePC’s, the 900.  It has a 9″ screen and a 16GB flash drive.  It runs Linux, and it’s just about right for accessing email, some light surfing, and doing small tasks like writing blog posts and messing with my checkbook.  And since it runs Linux, I can do a lot of nice network stuff with it, like SSH tunneling, VPN’s, and I can even make it act like a wireless access point.

However, the idea of leaving my little PC in a hotel room while I am out having fun leaves me a little uneasy.  I am not concerned with the hardware… it’s not worth much.  But I am concerned about my files, and the temporary files like browser cookies and cache.  I’d hate for someone to walk away with my EeePC and also gain access to
countless other things with it.

So this week, I decided to encrypt the main flash drive.  Before, the entire flash device was allocated as one device:

partition 1  –  16GB  –  the whole enhilada

Here’s how I made my conversion.

(0) What you will need:

  • a 1GB or larger USB stick (to boot off of)
  • an SD card or USB drive big enough to back up your root partition

(1) Boot the system using a “live USB stick” (you can create one in Ubuntu by going to “System / Administration / Startup Disk Creator”.  Open up a terminal and do “sudo -i” to become root.

ubuntu@ubuntu:~$ sudo -i
root@ubuntu:~$ cd /

(2) Install some tools that you’ll need… they will be installed in the Live USB session in RAM, not on your computer.  We’ll install them on your computer later.

root@ubuntu:/$ apt-get install cryptsetup

(3) Insert an SD card and format it. I formatted the entire card.  Sometimes, you might want to make partitions on it and format one partition.

root@ubuntu:/$ mkfs.ext4 /dev/sdb
root@ubuntu:/$ mkdir /mnt/sd
root@ubuntu:/$ mount /dev/sdb /mnt/sd

(4) Back up the main disk onto the SD card. The “numeric-owner” option causes the actual owner and group numbers to be stored in the tar file, rather than trying to match the owner/group names to the names from /etc/passwd and /etc/group (remember, we booted from a live USB stick).

root@ubuntu:/$ tar --one-file-system --numeric-owner -zcf /mnt/sd/all.tar.gz .

(5) Re-partition the main disk. I chose 128MB for /boot.  The rest of the disk will be encrypted.  The new layout looks like this:

partition 1  –  128MB  –  /boot, must remain unencrypted
partition 2  –  15.8GB  –  everything else, encrypted

root@ubuntu:/$ fdisk -l

Disk /dev/sda: 16.1 GB, 16139354112 bytes
255 heads, 63 sectors/track, 1962 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0002d507

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        1          17      136521   83  Linux
/dev/sda2           18        1962    15623212+  83  Linux

(6) Make new filesystems on the newly-partitioned disk.

root@ubuntu:/$ mkfs.ext4 /dev/sda1
root@ubuntu:/$ mkfs.ext4 /dev/sda2

(7) Restore /boot to sda1. It will be restored into a “boot” subdirectory, because that’s the way it was on the original disk.  But since this is a stand-alone /boot partition, we need to move the files to that filesystem’s root.

root@ubuntu:/$ mkdir /mnt/sda1
root@ubuntu:/$ mount /dev/sda1 /mnt/sda1
root@ubuntu:/$ cd /mnt/sda1
root@ubuntu:/mnt/sda1$ tar --numeric-owner -zxf /mnt/sd/all.tar.gz ./boot
root@ubuntu:/mnt/sda1$ mv boot/* .
root@ubuntu:/mnt/sda1$ rmdir boot
root@ubuntu:/mnt/sda1$ cd /
root@ubuntu:/$ umount /mnt/sda1

(8) Make an encrypted filesystem on sda2. We will need a label, so I will call it “cryptoroot”.  You can choose anything here.

root@ubuntu:/$ cryptsetup luksFormat /dev/sda2

This will overwrite data on /dev/sda2 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase: ********
Verify passphrase: ********
root@ubuntu:/$ cryptsetup luksOpen /dev/sda2 cryptoroot
root@ubuntu:/$ mkfs.ext4 /dev/mapper/cryptoroot

(9) Restore the rest of the saved files to the encrypted filesystem that lives on sda2.  We can remove the extra files in /boot, since that will become the mount point for sda1.  We need to leave the empty /boot directory in place, though.

root@ubuntu:/$ mkdir /mnt/sda2
root@ubuntu:/$ mount /dev/mapper/cryptoroot /mnt/sda2
root@ubuntu:/$ cd /mnt/sda2
root@ubuntu:/mnt/sda2$ tar --numeric-owner -zxf /mnt/sd/all.tar.gz
root@ubuntu:/mnt/sda2$ rm -rf boot/*
root@ubuntu:/mnt/sda2$ cd /

(10) Determine the UUID’s of the sda2 device and the encrypted filesystem that sits on top of sda2.

root@ubuntu:/$ blkid
/dev/sda1: UUID="285c9798-1067-4f7f-bab0-4743b68d9f04" TYPE="ext4"
/dev/sda2: UUID="ddd60502-87f0-43c5-aa28-c911c35f9278" TYPE="crypto_LUKS"   << [UUID-LUKS]
/dev/mapper/root: UUID="a613df67-3179-441c-8ce5-a286c16aa053" TYPE="ext4"   << [UUID-ROOT]
/dev/sdb: UUID="41745452-3f89-44f9-b547-aca5a5306162" TYPE="ext3"

Notice that you’ll also see sda1 (/boot) and sdb (the SD card) as well as some others, like USB stick.  Below, I will refer to the actual UUID’s that we read here as [UUID-LUKS] and [UUID-ROOT].

(11) Do a “chroot” inside the target system. A chroot basically uses the kernel from the Live USB stick, but the filesystem from the main disk.  Notice that when you do this, the prompt changes to what you usually see when you boot that system.

root@ubuntu:/$ mount /dev/sda1       /mnt/sda2/boot
root@ubuntu:/$ mount --bind /proc    /mnt/sda2/proc
root@ubuntu:/$ mount --bind /dev     /mnt/sda2/dev
root@ubuntu:/$ mount --bind /dev/pts /mnt/sda2/dev/pts
root@ubuntu:/$ mount --bind /sys     /mnt/sda2/sys
root@ubuntu:/$ chroot /mnt/sda2

(12) Install cryptsetup on the target.

root@enigma:/$ apt-get install cryptsetup

(13) Change some of the config files on the encrypted drive’s /etc so it will know where to find the new root filesystem.

root@enigma:/$ cat /etc/crypttab
cryptoroot  UUID=[UUID-LUKS]  none  luks
root@enigma:/$ cat /etc/fstab
proc  /proc  proc  nodev,noexec,nosuid  0  0
# / was on /dev/sda1 during installation
# UUID=[OLD-UUID-OF-SDA1]  /  ext4  errors=remount-ro  0  1
UUID=[UUID-ROOT]  /  ext4  errors=remount-ro  0  1
/dev/sda1  /boot  ext4  defaults  0  0
# RAM disks
tmpfs   /tmp       tmpfs   defaults   0  0
tmpfs   /var/tmp   tmpfs   defaults   0  0
tmpfs   /var/log   tmpfs   defaults   0  0
tmpfs   /dev/shm   tmpfs   defaults   0  0

(14) Rebuild the GRUB bootloader, since the files have moved from sda1:/boot to sda1:/ .

root@enigma:/$ update-grub
root@enigma:/$ grub-install /dev/sda

(15) Update the initial RAM disk so it will know to prompt for the LUKS passphrase so it can mount the new encrypted root filesystem.

root@enigma:/$ update-initramfs -u -v

(16) Reboot.

root@enigma:/$ exit
root@ubuntu:/$ umount /mnt/sda2/sys
root@ubuntu:/$ umount /mnt/sda2/dev/pts
root@ubuntu:/$ umount /mnt/sda2/dev
root@ubuntu:/$ umount /mnt/sda2/proc
root@ubuntu:/$ umount /mnt/sda2/boot
root@ubuntu:/$ umount /mnt/sda2
root@ubuntu:/$ reboot

When it has shut down the Live USB system, you can remove the USB stick and let it boot the system normally.  If all went well, you will be prompted for the LUKS passphrase a few seconds into the bootup process.

CarolinaCon – Day 3

After Hacker Trivia last night, it was pretty hard to wind down and get to sleep. So 10am arrived quickly.

The Art of Software Destruction – Joshua Morin and Terron Williams

I missed this talk due to Daylight Saving Time… yeah, that’s it.

Apparenly, the topic was fuzzing, or throwing unexpected data at a system’s inputs to see how it handles them.

Why Linux is Bad For Business – Wesley Shields

Wes tried to raise the hackles of the Linux users in the audience with his provocative title and his confrontational style. However, his point was driven home very well. Many companies flock to Linux when they want to build on a base of a community-supported project. However, there are other alternatives that might be a much better fit to their development and deployment plans. FreeBSD has a very business-friendly license, which does not require re-contribution of a company’s changes (which may be their special sauce).

Wes makes a very compelling argument. While I appreciate his conclusion, I disagree with one of his premises: that anyone who is building an appliance will probably be making their changes to the kernel, and not in user space. That was true for him, since he was building a “networking appliance”, and the best place for fast networking logic is in kernel space. However, I have also developed a Linux-based “appliance“. But our secret sauce was in the application, and not in the networking or driver layers. So for us, the underlying kernel and support packages were just commodities.

Nits aside, Wesley’s talk was more thought-provoking than just provoking. And your author will certainly consider FreeBSD on his next project that requires an open source base.

Sorry, Wes. If you were trying to come across as a jerk, you failed. Great talk!

The Evolution of Social Engineering – Chris Silvers and Dawn Perry

These guys have entirely too much fun at their jobs. They are security consultants who specialize in penetration testing in the physical realm. That is, they break into office buildings. Well, that’s not really true. People let them in — they con their way into office buildings.

Chris and Dawn shared lots of stories about the many jobs they have been on, explaining along the way the rules of engagement, how they are hired by management, what they are trying to prove, and how far they’ll go to exploit the helpfulness of others.

One hour was simply not enough for these guys!

Metasploit – Ryan Linn

Man, I should have taped this talk.

Ryan gave us a very fast-paced hands-on demonstration of Metaspoit (as run from the Backtrack 4 Live CD), and the many ways that a target box can be probed and PWNED. He covered the msfconsole, meterpreter, automation of exploits, and generating malicious payloads.

This talk wins the “most informative” award from me. Very good stuff.

How the Droid Was Rooted – Michael Goffin

Michael shared his experience working with team that rooted the Motorola Droid phone (hint for developers, putting the phrase “this could be exploited” in the comments of your open-source code sort of acts like a neon sign that says “HACKERS WELCOME”).

There was a lot of good technical content, explaining how the Droid software is packaged and upgraded. But just as interesting was his account of the team dynamics. When one member decided to take the entire team’s marbles and go home, it really did not slow them down, because they were using a distributed source code control system (Mercurial). That meant that every team member had a complete copy of the source code repository. Lesson learned.

At the end of this talk, I wondered how long it would be before you could buy smart phones directly from the carrier that had root access, straight out of the box (after all, I have root access to my PC’s and PDA’s). Having worked for a cellular phone manufacturer, I would guess that we may never see that day. So, give a big thanks to Michael and his team for their hard work!

Protecting Systems Through Log Management and System Integrity – David Burt

This talk was, by far, the worst of the show.

David did not seem to have a core message… instead, he had hastily thrown together 86 slides worth of screen shots and raw data about logging tools. On the stage, he struggled to speed-read his way through the slides, speeding up even more when he hit the 5 minute warning. 75 minutes into his one-hour talk, though, David’s message started to shine through. He knows a lot about logging — and he is available for consulting work.


We wrapped up with some prize give-aways… youngest attendee, oldest attendee, drunkest attendee, winner of a rock-scissors-paper showdown, that guy who looks like some other famous guy, and anyone else who will take this junk.

And that’s it, the show is over.

Mad props to the CarolinaCon Group, organizers, sponsors and volunteers. I had a great time, and I learned a lot. And it looked like most of the other 176 attendees did as well.

Now let’s see how much trouble we can get into between now and CarolinaCon 7!

CarolinaCon – Day 2

The second day of CarolinaCon was packed from sunup to sundown — who am I kidding… hackers seldom rise before noon.  The festivities started at 10am.

Hacking with the iPhone – snide

No, not hacking the iPhone… but using the iPhone as a hacking tool. This talk was a good slide into the morning, a chance to let the coffee sink in. It could probably summarized with two main points:

  • Since the iPhone OS is a distant cousin of BSD Unix, many open source (Linux) networking tools can easily be ported to run on it, so a jailbroken iPhone makes a decent platform for network sniffing and the like.
  • A jailbroken iPhone provides a behind-the-scenes look at the user interface, and many things that are set on the main GUI can be changed by directly manipulating the underlying settings files.

Neither of these ideas is too surprising, and so this talk was nothing new. Still, for me, having never played with a jailbroken iPhone (honest), it was an eye-opening experience. Or maybe that was just the coffee kicking in.

We Don’t Need No Stinking Badges – Shawn Merdinger

Shawn has spent some time evaluating campus-oriented badge reader door locks from a company called S2 Security. He showed how they work, and how they are advertised to work — not necessarily the same thing. An interesting glimpse into the world of distributed security systems, with several take-home lessons about what not to do.

It’s a Feature, Not a Vulnerability – Deral Heiland

This is the third time that I have seen Deral present at CarolinaCon. In 2009, he showed us what a mistake it can be to “web-enable” your products, and in 2008, he showed us how he made friends at Symantec with “Format String Vulnerabilities 101”.

This time, he continued his endorsement of Symantec’s products by demonstrating how their AMS product conveniently allows very easy access to a machine’s resources. In fact, all it takes is a single packet to tell AMS to run any command on a target Windows box. That’s convenient! (PWNED)

Smart People, Stupid Emails – Margaret McDonald

Margaret came here all the way from Denver to tell us what we already knew… that otherwise intelligent people send the stupidest things in email. This was a lively discussion that we could all relate to… yet I have this sinking feeling that our inboxes will still be filled with garbage when we get back to work on Monday.

Mitigating Attacks with Existing Network Infrastructure – Omar Santos

Omar was cursed with the dreaded 3:00 time slot… just in time for the after-lunch sleepies. It did not help that his presentation was JAM-PACKED with very technical networking information. So, for the most part, I sort of zoned out during this very informative presentation.

I tried hard to stay awake by asking a question (about “bogons” — in this case, the newly-allocated and unfortunately-numbered address space). But it did not help.

Omar plans to give this same talk at “Hack in the Box” in Dubai later this year. So if I start feeling regrets that I missed something, I guess I can always book a flight.

OMG, The World Has Come To An End! – Felonious Fish

Hackers are usually prepared for anything… or are they? FF led a discussion on survival, what is needed when the rest of our infrastructure is gone. We might have food and water and shelter, but when my iPhone battery dies, it’s game over!

You Spent All That Money and You Still Got Owned – Joe McCray

Joe’s talk was one of the highlights of the Con… even Stevie Wonder could see that it was awesome. Joe told us his secret to success — he goes into companies, totally pwns them in short order, tells them how they suck, and then they pay him.

Apparently, corporate America makes Joe’s job very easy by following the worst practices. And on the odd chance that they have their operational act together, he can always solicit a security slip-up by sending them a carefully-crafted email (pwn), or if that fails, by leaving a CD with provocative title for some nosy employee to find (serious PWN).

What a life Joe leads — that “education” he got in prison has really paid off.

Locks: Past, Picking and Future – squ33k

The lovely and talented squ33k — 5th grade teacher by day, lock hacker by night — educated us on all things lock-related. With assistance from the TOOOL crew, she taught us how modern pin tumbler locks work, and how they can be picked. But being a full-time teacher, she made sure to frame her talk with some interesting background info on locks from as far back as 4000 years ago, and a glimpse into what locks may be like in the future.

I am so proud that our youngsters are learning their skills and attitudes from this woman. She’s a girl geek role model!

Hacker Trivia

What’s that? Al was spotted in parking lot? Someone allowed him back into the country? I thought that call to the TSA would be enough to keep him detained in the airport until the Con was over. I guess not. HE’S BACK!

Once again, Al Strowger took the stage and led us in a game of Hacker Trivia. Loosely based on Jeopardy!, this game quizzed the inebriated audience on the topics of: Movie Quotes, x86 instructions, other (hacker) conferences, math, 2009 tech, and ccTLD’s. John “Math for 400” Davis took home first prize, an iTunes gift card. Many other contestants won spot-prizes: hacking books, some new geek toys, donated “vintage” equipment, Vic Vandal’s old CarolinaCon 3 t-shirt, and lots of cupcakes.

Good night everybody. Sleep well, we’ll see you at 10am tomorrow morning!

CarolinaCon – Day 1

It’s that time of year again… time for the annual CarolinaCon security conference. This year promises to be bigger and better than last year — it has expanded from 1.5 days to 2.5 days, and it has moved from the somewhat undistinguished Holiday Inn in Chapel Hill to the somewhat less undistinguished Holiday Inn in Raleigh.

Notably missing was the “Master” of masters of ceremonies, Al Strowger. But Vic Vandal and his cohorts seemed to have the show in order. Personally, I can’t imagine a Con without the provocative charms of Al. But we’ll see how they do.

As usual, the Con started with a short after-work session on Friday night. There were three presentations to get the crowd warmed up.

Cybercrime and the Law Enforcement Response – Thomas Holt, a.k.a. Professor Farnsworth

The good professor never disappoints, and he really had a challenge this time, to warm up an otherwise un-primed crowd. He dove right in, with the not-so-statistically-significant results of a survey of state and local law enforcement officers, asking about their experience with computer crime. It was not surprising to find that most LEO’s were not very well versed in the specifics of computer-based crime, but they had a pretty good appreciation for the concepts. Thank you, CSI. Many trends and prejudices were revealed, and Dr Holt and members of the audience supposed several reasons why these might be so.

The Search for the Ultimate Handcuff Key – Deviant Ollam

If the crowd was not warmed up before Deviant Ollam took the stage, they certainly were after. He and the TOOOL team showed how handcuffs work, and how they can be defeated, sometimes with simple items like a piece of notebook paper!

But just as important as the actual material they presented, was the chosen format of their presentation. In true CarolinaCon fashion, they began by mixing a pitcher of their beverage of choice… tonight’s choice was a “Stone Fence” (one part Apple Jack, three parts hard apple cider, and a splash of bitters). Every time something in their presentation went unexpectedly, they would take a drink — this rule was strictly enforced by the audience.

In case that was not interesting enough, each live demonstration of handcuff picking techniques was accompanied by background music from a famous X-rated movie from the 1970’s and 80’s. In many cases, an audience member was able to “name that movie” before the lock-picker had freed himself, thus winning a prize.

Now this is the CarolinaCon that I came to see.

Microcontrollers 101 – Nick Fury

Finally, Nick showed the audience how to think small… he introduced the AVR microprocessor and the “Arduino” board and tools. Then he showed a few demos of what a small board like that can do. Certainly a tool that many hackers can add to their toolbox.

With these three presentations, the Con begins. We’re looking forward to Day 2, which brings a full day of hacking.  See you at 10am.

iPhone VPN

One of the coolest features of the iPhone is the way it uses the best data network that it can find. If you’re at home or at work, or even at a coffee shop, it will use the local wifi network. But if you’re out of range of any suitable wifi networks, it will use AT&T’s “3G” (UMTS) network. And if it can’t find a UMTS network, it’ll fall back to EDGE. Phone companies call this hybrid approach “ABC”, or “always best connection”.

Now that I have an internet device in my pocket, I find myself using public (or otherwise open) wifi connections quite a bit. And this carries with it some unintended consequences. That is… everything I type and everything I read is transmitted in the clear, unencrypted.

I try to make a habit of encrypting my data traffic whenever possible. My mail server is set up to only allow SSL connections. So no matter where I check my mail from, I am forced to use an encrypted connection. Similarly, banks and commerce web sites usually force you to switch to HTTPS before you start entering information. But there are a lot of applications on the iPhone that do not use encryption at all.

You might ask yourself why bother to encrypt your Twitter connection, since what you type is going to be blasted out to the world anyway. But the point is…

If you encrypt everything, then nothing is left to chance.

So I decided to explore a VPN option on the iPhone. It supports three flavors of VPN: L2TP, PPTP and IPSec. I was disappointed (but not surprised) that “openvpn” was not an option, since I already use this excellent open source SSL-based VPN package.

So I decided to give PPTP a try.

Setting up the PPTP server

On my Ubuntu 8.04 LTS server, I installed a PPTP server called, appropriately enough, “pptpd“. Configuration was very easy. Most of the setup was done for me after I did the standard apt-get install pptpd. I simply needed to pick a private subnet that would be used for my VPN clients, and an IP address in that subnet to use for the server. I chose the subnet and for the server (these addresses are part of a private network address space, defined by RFC 1918, just like 192.168.x.x and 10.x.x.x addresses).

My /etc/pptp.conf configuration file for the pptp daemon looks like this:

option /etc/ppp/pptpd-options

I also needed to tell the daemon to give out some DNS addresses when a client connects, so in the /etc/ppp/pptpd-options file, I added the two “ms-dns” lines below:

name pptpd
ms-dns  #
ms-dns  #

Finally, I needed to add an entry into the /etc/ppp/chap-secrets file that would contain my password. Mine looks like this:

alan pptpd MyHardToGuessPassword *

At this point, the PPTP server was completely configured, so I restarted it with service pptpd restart.

Setting up the iPhone

On the iPhone, I needed to set up a VPN client. This is very easy. On the settings screen, go to general / network / VPN and “Add VPN Configuration…”. Then just fill in the blanks.

  • choose “PPTP”
  • enter a description
  • your server’s IP address
  • the username (from above)
  • RSA SecurID=OFF
  • the password (from above)
  • encryption level = Auto
  • “Send All Traffic” = ON
  • Proxy = OFF

Click on “Save” and you will see a switch in the network tab and also in the main settings tab to turn the VPN on and off.

For now, I am leaving it off unless I am on a public network. I am not sure, but I think that keeping the VPN alive might use a lot of battery. So I do not use it unless I need it.


For me to get this VPN on the internet, I had to do two more things: punch a hole in my firewall for the PPTP traffic, and forward traffic from my VPN out to the rest of the world.

For my server, both of these tasks were handed by the same tool: shorewall.

I added a “masquerade” rule to /etc/shorewall/masq to NAT all of the traffic from 172.16.4.x out through my main network interface.

eth0    # OpenVPN and PPTP

And then I added two rules to /etc/shorewall/rules to allow the PPTP traffic in.

ACCEPT  net  fw   tcp  1723  # PPTP
ACCEPT  net  fw   gre        # PPTP

When shorewall starts, it will generate the iptables rules that are used by the kernel to filter packets. If you’re using hand-written iptables rules, then you will need some rules that look something like this:

# accept "gre" protocol traffic (PPTP tunnel traffic)
iptables -A INPUT -p gre -j ACCEPT
iptables -A OUTPUT -p gre -j ACCEPT
# accept PPTP control traffic to TCP port 1723
# (my server IP is
iptables -A INPUT -p tcp --sport 1723 -s -j ACCEPT
iptables -A OUTPUT -p tcp --dport 1723 -d -j ACCEPT
# masquerade/NAT internet traffic out of interface eth0
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
# turn on packet forwarding
echo "1" > /proc/sys/net/ipv4/ip_forward


Now, when I am in a coffee shop, I can turn on the VPN easily by flipping the switch in the iPhone’s main settings screen. It will make a TCP connection to my server, negotiate a few things, and then send all further network traffic through an encrypted tunnel directly to my server, which relays it out to the internet.

You can test this by going to a web site like If the VPN is working, it will show your server’s IP address. If not, it’ll show the coffee shop’s IP address.

Once I am using the VPN, anyone in the coffee shop who happens to be sniffing traffic on the wireless network will only see a single connection from my iPhone to my server, but the contents of this connection will be scrambled.


This morning, when I scanned my email, I ran across a report from rkhunter, a tool that runs on my web server that periodically checks to see if system files have changed, or if users have been added, what processes are listening on ports, and a litany of other tests to detect whether a system might be infected. I get these reports fairly often, usually the day after I do an upgrade, or after I add a new user.

However, today’s email was a little bit alarming. It said that several files had been changed recently. And these files all seemed to do with the same sort of things: running stuff behind the scenes, showing library dependencies, elevating privileges. Basically, these were tools that you would want to modify if you wanted to cover your own tracks.

Warning: The file properties have changed:
         File: /bin/sh
         Current hash: 23603f77da4ca37705146fd8a4ed951c8b037156
         Stored hash : 91654fd25d317bd13a65e10d777ac021f4a1a4f6
Warning: The file properties have changed:
         File: /bin/dash
         Current hash: 23603f77da4ca37705146fd8a4ed951c8b037156
         Stored hash : 91654fd25d317bd13a65e10d777ac021f4a1a4f6
         Current inode: 180336    Stored inode: 180255
         Current file modification time: 1236603791
         Stored file modification time : 1213978027
Warning: The file properties have changed:
         File: /usr/bin/dpkg
         Current hash: 4e05d20a4f828c31eb5f6dd9cc5f04d1d6202d0a
         Stored hash : 09a5bbd0398cc9f02b52440e1241cd942e784a15
         Current inode: 248598    Stored inode: 246001
         Current size: 375340    Stored size: 371244
         Current file modification time: 1236595869
         Stored file modification time : 1220443410
Warning: The file properties have changed:
         File: /usr/bin/dpkg-query
         Current hash: ff8098920430d399933ee24245748983a0661869
         Stored hash : 4a1c1226cbe9dd2ddbec7b5652f1fa8aa0b15f09
         Current inode: 248600    Stored inode: 246003
         Current file modification time: 1236595869
         Stored file modification time : 1220443410
Warning: The file properties have changed:
         File: /usr/bin/file
         Current hash: 4ab93b21aaabb405f4bd2e90f16ee5e952aa746b
         Stored hash : 80dc1735091a4309d23e49ce542c58ddd16163dc
         Current inode: 245969    Stored inode: 246049
         Current file modification time: 1244193699
         Stored file modification time : 1215771733
Warning: The file properties have changed:
         File: /usr/bin/ldd
         Current inode: 248852    Stored inode: 246132
         Current file modification time: 1233224578
         Stored file modification time : 1222684817
Warning: The file properties have changed:
         File: /usr/bin/perl
         Current hash: 00d703e925eca6de0c8fc9bd9d4505db4b81ce33
         Stored hash : efb4a1a3d02798718b7f2bbfea6787dd0de79968
         Current inode: 245962    Stored inode: 246591
         Current file modification time: 1246045733
         Stored file modification time : 1216891204
Warning: The file properties have changed:
         File: /usr/bin/sudo
         Current hash: e649919d4bbc6ac78e38497ca94dc387cc2811a7
         Stored hash : 49e97774326fc9eb5f7cb680477c1d56f4e28921
         Current inode: 246543    Stored inode: 246747
         Current file modification time: 1234840625
         Stored file modification time : 1220275024
Warning: The file properties have changed:
         File: /usr/sbin/cron
         Current hash: 5efdffc9796731168fb7acc8688c5a02e0da42dd
         Stored hash : 04924b72b749e8179bb5839bac1a296c7acf93c4
         Current inode: 245910    Stored inode: 248315
         Current file modification time: 1242164811
         Stored file modification time : 1220989568

One or more warnings have been found while checking the system.
Please check the log file (/var/log/rkhunter.log)

I scanned /var/log/dpkg.log, to see if I had recently done an update, and I did not see anything. That little paranoid part of my brain started to wake up (those that know me might clarify that when I say “little”, I really mean “big”).

So briefly, someone had changed the following files on my server. How could I tell if they were the “real” ones or not?

  • /usr/sbin/cron
  • /usr/bin/sudo
  • /usr/bin/perl
  • /usr/bin/ldd
  • /usr/bin/file
  • /usr/bin/dpkg-query
  • /usr/bin/dpkg
  • /bin/dash
  • /bin/sh

What bothered me about this combination was that dpkg was in that list, so I could not use any of the apt/dpkg tools to verify the integrity of my packages against what is published on the Ubuntu mirrors.

So I had to take matters into my own hands. I went to the Ubuntu packages site and searched for the first package, ‘cron’. From there, I could click on the ‘i386’ link to download a local copy of the cron_3.0pl1-100ubuntu2.1_i386.deb file onto my laptop (not onto the suspect server). I extracted the contents using dpkg -x cron_3.0pl1-100ubuntu2.1_i386.deb .. From there, it was pretty easy to compare MD5 checksums of the files.

$ ls
$ dpkg -x cron_3.0pl1-100ubuntu2.1_i386.deb .
$ ls
cron_3.0pl1-100ubuntu2.1_i386.deb  etc  usr  var
$ md5sum usr/sbin/cron
c1d78d8a9a99b52df8ecba41517ab013  usr/sbin/cron

This checksum matched the one on my server. So that means my binary files were legitimate (this does not explain how they got updated without leaving a trail in the logs, but that is another issue).

Lather, rinse, and repeat for all of the suspected files.

I hope this little story helps someone else defuse that panicky feeling that sets in when your intrusion detection system sends you an unpleasant email.


I just spent the entire weekend re-building a server for the Triangle Linux Users Group.

We first noticed that something was wrong when the machine stopped responding over the network. A couple of our admins took a trip to the data center and noticed that we had a firehose of data on port 6667 (an IRC port), originating from a process owned by the “apache” user.

So we’d been pwned. Now what?

We figured the best way to proceed would be a complete re-install of the operating system. I happened to be free the next day, so I was volunteered to lead in the clean-up duty.

So I drove out to the data center to camp out in the cold air conditioning for a while. I saved away the old infected partitions (we use LVM) and I allocated new space for the fresh install. After I had the OS installed and responding over the network, I went home to finish. I worked frantically over the weekend to restore many of the services that we enjoyed. My priorities were clearly restoring our 250 user accounts and then getting email working (securely). In the process, I gave myself a crash course in LDAP, since that is what we use for user authentication.

Within about 48 hours, we had everything restored except our web pages. After all, we knew the break-in had allowed someone to create a rogue process owned by apache. So we must have had some problem with one of our web-based applications. We did not know whether it was our Drupal-based web page, our web mail client, our wiki, a user application, or something else.

I dug through the log files on the infected partitions, and soon it became apparent that there was a cron job set to run every minute, owned by the ‘apache’ user. The script simply looked to see if its IRC program was running, and if any part of it was damaged or deleted, it would reinstall a new copy of itself somewhere else on the disk… somewhere no one would look, like /var/tmp/.s/something.

Finally, the apache error logs showed what the problem was. It seems that we were running an unpatched version of “RoundCube“, a web-based IMAP e-mail client with a nice AJAX interface. There is a vulnerability in this package that allows a visitor to upload a package to your web server and then run their programs on your server.

Fortunately, the process runs as the “apache” user, and not as “root”. Otherwise, the rogue software would have had permission to do a lot more damage than it actually did. As it stands, the bot simply chatted with a lot of other infected machines. Thankfully, it did not seem interested in the files on our machine.

I learned a lot from this experience. As one admin said, the forced cleanup was a “much-needed enema”, something we had avoided for a long time. As a shared system, system administration was something that was handled by a loose group, and was handed off to new members every year. This break-in was enough to attract our attention, but it was not destructive. And it inspired us to simplify our existing system. And it inspired me to set up nightly backups.

Go to Top