A few weeks ago we had a client of a client's machine get owned.

It was a RH7.3 box running an external mail server, NFS on their local network, and had a copy of Squirrelmail running for webmail access. And as you'd expect, it had a copy of sshd running.

Apparently they lost administrative access to the machine, and someone set up an account with the username admin and password admin.

It got owned in about 3 hours.

It was serving a Paypal phishing site, and they got notice from some US government department to take it down.

They rang in a bit of a panic, and I was tasked with working out what had happened.

So I logged into the box to check for all the basic things.

I'm fairly certain the box was owned by a script kiddie. The script they were using was half decent and erased all the logs, so I couldn't find out all that much on what had happened to the box since it was last rebooted 120+ days ago.

find(1) to the rescue! I was able to run a bunch of commands to see what had been created and modified over the last few days. I also ran rpm -V on all the packages installed to check if anything managed by RPM had been modified.

There wasn't anything particularly critical, except for a bunch of modified iptables kernel modules, but i'm pretty certain they weren't touched by the intruder.

I also checked out the phishing software itself, which consisted of a bunch of HTML and PHP scripts. Suprisingly well-written PHP, though. Most of scripts were functionised, there were humour-laden comments everywhere, and there were even credit card check regexs.

I had a look at the file the phishing site was logging details to, and there were about 5 people who had put their details in, with proper credit card and social security numbers.

After i'd finishing checking the box, I finally stopped httpd.

I got switched onto another task, but logged back in 30 minutes later to check on the system. I noticed the the script kiddie had logged back in, restarted httpd, and logged back out again, but they hadn't cleaned up after themselves.

So I stopped httpd again and replaced the httpd binary with a shell script that returned 0. I also fixed the init scripts so whenever you ran a service httpd start/stop/status/restart it would always visually return [ OK ] in big green letters.

The biggest problem was that I couldn't see what he was doing once he su'd to root, so I replaced the su binary with a shell script that wrapped sudo. I fixed up sudoers to auth the admin account using the root password, bypass syslog and log to a file, and notify via email when sudo was called.

So he'd log in as admin, su to root, do his thing, clean up root's .bash_history in, and log out.

Of course, because he'd "su'd to root" using sudo, all the logs were written out to /home/admin/.bash_history, so I had a full dump of everything he'd run.

It was relatively amusing. Lots of

service httpd stop

service httpd start

service httpd status

He also downloaded his root kit (p.jpg) off a server in Romania a number of times, and tried running it.

Curious, I downloaded his rootkit, untarred it, and poked around. There were two components - a binary (p), and the source (p.c). Curious, and fully expecting some horrible C, I opened up p.c.

This is what I found:

 * Linux kernel ptrace/kmod local root exploit
 * This code exploits a race condition in kernel/kmod.c, which creates
 * kernel thread in insecure manner. This bug allows to ptrace cloned
 * process, allowing to take control over privileged modprobe binary.
 * Should work under all current 2.2.x and 2.4.x kernels.
 * I discovered this stupid bug independently on January 25, 2003, that
 * is (almost) two month before it was fixed and published by Red Hat
 * and others.
 * Wojciech Purczynski 
 * (c) 2003 Copyright by iSEC Security Research


So now I knew how he got access.

He logged in a number of times over the next few days, tried to restart httpd, downloaded and ran the exploit, but didn't get any further.

Finally 3 days later, he logged in, ran the exploit again, decided to clean up the phishing site, and logged out again.

We subsequently reinstalled the box with a modern operating system and enforced a strong password policy.

So, what did we learn from this little escapade? The human element of security is always the weakest. Strong passwords are the first line of security.