AdmiNirvana

Almost 24 hours per day, The Linux Fix is inundated with FTP and SSH brute force attempts to our server farm.   This has compromised a few our our customer’s accounts from time to time, and I decided it was time to come up with a solution.

The problem is tricky–we must leave FTP and SSH open to the entire world, but at the same time be selective on what we black list.   How do you make that determination?  Strictly on bad login credentials?

We could, but that would mean that we’d inadvertently lock out real users.  A better solution we found has to do with timing connection attempts.   With IPTables, we can keep a counter based upon source IP–and track how many new socket attempts are made within a certain span of time.     For instance, if we detect the IP address 1.2.3.4 making 5 connection attempts within 60 seconds, there is a darn good chance it isn’t someone mistyping a password.

Here is how we did it, based upon another script we found out in the Internets:

#!/bin/bash
/sbin/iptables -N SSH
/sbin/iptables -N SSH_BLACKLIST
/sbin/iptables -A SSH_BLACKLIST -m recent --name SSH_COUNTER --set -j LOG --log-level warn --log-prefix "Blocked: "
/sbin/iptables -A SSH_BLACKLIST -j REJECT
/sbin/iptables -A SSH -m recent --name SSH_COUNTER --update --seconds 300 -j REJECT
/sbin/iptables -A SSH -m recent --name SSH --rcheck --seconds 60 --hitcount 5 -j SSH_BLACKLIST
/sbin/iptables -A SSH -m recent --name SSH --rcheck --seconds 2 -j LOG --log-level warn --log-prefix "Added: "
/sbin/iptables -A SSH -m recent --name SSH --update --seconds 2 -j REJECT
/sbin/iptables -A SSH -m recent --name SSH_COUNTER --remove -j LOG --log-level warn --log-prefix "Removed: "
/sbin/iptables -A SSH -m recent --name SSH --set -j ACCEPT
/sbin/iptables -A INPUT -m state --state NEW -p tcp -m tcp --dport 22 -j SSH

This creates two new tables, SSH and SSH_BLACKLIST.   Upon the intial connection attempt, the IP is added to the SSH_COUNTER counter.   If the same IP address is seen again within 60 seconds, it is duly noted–however no action is taken until the hitcount reaches 5.   In that case, the rules jump to the SSH_BLACKLIST table, it is logged, and subsequent connections from that IP are dropped for 5 minutes until things calm down.   In order to do this for FTP, just rename the targets as appropriate and change the target port to 21 on the last line.

The nice thing about this set up is that it is auto-cleaning.  After 5 minutes of no activity, the counter forgets about the IP address and things return to normal.   We’ve found that this is just enough protection to drastically reduce bruteforce attempts, yet not get in the way of normal usage by our customers.  Over time, this has become our favorite technique and we’ve begun to implement it on any Internet-facing machine with open SSH ports.

This is one of those “no way, that can’t be true” type of events.

Using Solaris’ Dtrace functionality a systems engineer determines that noise is responsible for his intermittent disk latency problems.   Just wait for the last part of the video, it’s worth it.

http://blogs.sun.com/brendan/entry/unusual_disk_latency

Thanks to Greg for pointing that one out to me.

So, I finally had a chance to play with VMware ESXi.   It’s pretty much what I expected, a straight-up version of ESX.  Very, very nice… I’ll start moving more servers over from VMware Server 1.x and report back on my progress.

One of the things that annoyed me out of the gate is the lack of SSH support.   It’s there in the underlying operating system, just not enabled.   Here’s how to turn it on:

1. Get on the console of the ESXi server.
2. Press ALT-F1 to get to the OS system console
3. Type “unsupported”
4. Enter the root password at the password prompt.
5. Edit /etc/inetd.conf with vi, and uncomment the SSH line
6. Run:  kill -1 $(cat /var/run/inetd.pid)

And viola!  SSH to your ESX box.   Enjoy!

I have been itching to get Time Machine to work over a (unsupported) network-shared volume since we’re unwilling to drop the cash for a Time Capsule.  I’ve seen quite a few different posts on how to get it done, including the terminal trick to tell Time Machine to use unsupported volumes, but none of them quite worked.

The best I could get was Time Machine to begin to write the files to the share, but after a few seconds die with a vague “Backup disk could not be created error”.  This stumped me for a bit until I came across this.  Apparently sometime around 10.5.2, Apple introduced a new, undocumented “feature” to Time Machine that causes it to fail over network volumes when doing the initial backup.   However, once the files are created it will work fine.

So, the magical combination is as follows:

• CentOS 5 with Netatalk-2.0.3 compiled, installed, and configured per this post.  Note: I also had to modify etc/cnid_dbd/dbif.c with the same code change as specified there, but YMMV.
• OS X Leopard, patched to 10.5.4.
• Changes to Netatalk’s netatalk.conf file per this post at the Gentoo Wiki.
• Following the post linked above precisely.

Once that happened, Time Machine has begun to work great over AFP to our backup volume–even for multiple Macs connecting to the same share. Behold!

It probably doesn’t need saying; but this is clearly an unsupported way to use Time Machine.  It has been running this way for me only about a day.    If you’re concerned about having to troubleshoot problems that may pop up down the road, especially regarding backups; picking up a Time Capsule is probably a far better idea.

I think I’ve mentioned many times on this blog, but one of the most satisfying things regarding Linux and Unix are that you’re never done learning about it. A never-ending lesson in operating systems! Well, chalk up another lightbulb moment for me this morning.

Imagine a script wherein a process needs to be checked for proper exit. Let’s say “mysqldump”. Typically I’d do something like this, for example:


#!/bin/bash
STATUS=1
while [ ${STATUS} -ne 0 ]
do
mysqldump -uroot -psomepass --all-databases > sql-backup.sql
STATUS=${?}
done


exit 0


That’ll work just fine–the special reserved variable ${?} contains the exit code of the last run command. Mysqldump is kind enough to use non-zero ones on any kind of error, so if it doesn’t work in our script we’ll retry.

But for instance, let’s say our script looks like this:


#!/bin/bash
STATUS=1
while [ ${STATUS} -ne 0 ]
do
mysqldump -uroot -psomepass --all-databases | gzip > sql-backup.sql
STATUS=${?}
done


exit 0


The problem here is that ${?} now contains the exit code for gzip, not mysqldump! Will gzip respond properly if mysqldump doesn’t provide an input stream from the pipe? Maybe, maybe not. Bottom line is that it isn’t reliable, and not what I’d consider good shell programming.

Instead, check out this solution:


#!/bin/bash


STATUS=1
while [ ${STATUS} -ne 0 ]
do
mysqldump -uroot -psomepass --all-databases | gzip > sql-backup.sql
STATUS=${PIPESTATUS[0]}
done


exit 0


The BASH reserved array ${PIPESTAUTUS[x]} contains the exit codes for all programs in the array. In this example, ${PIPESTATUS[0]} is mysqldump, and ${PIPESTATUS[1]} is gzip.