AdmiNirvana

Almost 24 hours per day, The Linux Fix is inundated with FTP and SSH brute force attempts to our server farm.   This has compromised a few our our customer’s accounts from time to time, and I decided it was time to come up with a solution.

The problem is tricky–we must leave FTP and SSH open to the entire world, but at the same time be selective on what we black list.   How do you make that determination?  Strictly on bad login credentials?

We could, but that would mean that we’d inadvertently lock out real users.  A better solution we found has to do with timing connection attempts.   With IPTables, we can keep a counter based upon source IP–and track how many new socket attempts are made within a certain span of time.     For instance, if we detect the IP address 1.2.3.4 making 5 connection attempts within 60 seconds, there is a darn good chance it isn’t someone mistyping a password.

Here is how we did it, based upon another script we found out in the Internets:

#!/bin/bash
/sbin/iptables -N SSH
/sbin/iptables -N SSH_BLACKLIST
/sbin/iptables -A SSH_BLACKLIST -m recent --name SSH_COUNTER --set -j LOG --log-level warn --log-prefix "Blocked: "
/sbin/iptables -A SSH_BLACKLIST -j REJECT
/sbin/iptables -A SSH -m recent --name SSH_COUNTER --update --seconds 300 -j REJECT
/sbin/iptables -A SSH -m recent --name SSH --rcheck --seconds 60 --hitcount 5 -j SSH_BLACKLIST
/sbin/iptables -A SSH -m recent --name SSH --rcheck --seconds 2 -j LOG --log-level warn --log-prefix "Added: "
/sbin/iptables -A SSH -m recent --name SSH --update --seconds 2 -j REJECT
/sbin/iptables -A SSH -m recent --name SSH_COUNTER --remove -j LOG --log-level warn --log-prefix "Removed: "
/sbin/iptables -A SSH -m recent --name SSH --set -j ACCEPT
/sbin/iptables -A INPUT -m state --state NEW -p tcp -m tcp --dport 22 -j SSH

This creates two new tables, SSH and SSH_BLACKLIST.   Upon the intial connection attempt, the IP is added to the SSH_COUNTER counter.   If the same IP address is seen again within 60 seconds, it is duly noted–however no action is taken until the hitcount reaches 5.   In that case, the rules jump to the SSH_BLACKLIST table, it is logged, and subsequent connections from that IP are dropped for 5 minutes until things calm down.   In order to do this for FTP, just rename the targets as appropriate and change the target port to 21 on the last line.

The nice thing about this set up is that it is auto-cleaning.  After 5 minutes of no activity, the counter forgets about the IP address and things return to normal.   We’ve found that this is just enough protection to drastically reduce bruteforce attempts, yet not get in the way of normal usage by our customers.  Over time, this has become our favorite technique and we’ve begun to implement it on any Internet-facing machine with open SSH ports.

I usually don’t like to cross-post between blogs, but this problem seems prevalent enough that it needs more exposure.   We’ve solved it at TLF, and here’s how:

http://www.thelinuxfix.com/blog/2008/10/24/trixbox-vpns-and-the-20-second-issue/

Just got back on Sunday night from Detroit for the TLF rack upgrade.

Unfortunately our new rack didn’t make it there as fast as I did, so I ended up staging the equipment for deployment by the collocation facility.

While I was there however, I ended up racking equipment for three new customers–and now the existing TLF rack is completely full. We still have about 4U left, but are using 79% of both 30 amp circuits–which is too close for comfort (nominal states 80% maxiumum).

Bottom line though is that the time was right for a new rack purchase… now the goal is to fill that one up!

Here’s the picture I took right before I left on Saturday night.

I happened to run across this fella today, who also runs a very large VMware Server farm in a production environment. He makes a few mentions of his architecture which is slightly different than our approach, however it could prove handy to someone else building out such a thing.

Whew… after a few months of solid work on TLF stuff, we finally hit a big milestone–the opening of www.tlfhosting.com, which will eventually become the main jump for all things hosting related at the ‘Fix.Right now it’s pretty skeletal, but it’s a start.

A big component of this is the shared server hosting stuff which is now all handled under H-Sphere–an all encompassing control panel, distribution, and load-management system. The super cool thing is that it reduces management in a huge way allowing TLF to sell really nice shared sever hosting for super-cheap, like $4.50/mo. Go sign up!

TLF also been enlisted as an official domain registrar, as well as a reseller for Comodo–meaning TLF is now a one-stop shop. You just visit that control panel and you’re off and running.

The next phase is to get our VPS stuff finished up and migrate the rest of the internal TLF stuff to the new VMware server, freeing up the old one for customer VM’s.I can almost see the light at the end of the tunnel!