The problem is tricky–we must leave FTP and SSH open to the entire world, but at the same time be selective on what we black list.   How do you make that determination?  Strictly on bad login credentials?

We could, but that would mean that we’d inadvertently lock out real users.  A better solution we found has to do with timing connection attempts.   With IPTables, we can keep a counter based upon source IP–and track how many new socket attempts are made within a certain span of time.     For instance, if we detect the IP address 1.2.3.4 making 5 connection attempts within 60 seconds, there is a darn good chance it isn’t someone mistyping a password.

Here is how we did it, based upon another script we found out in the Internets:

#!/bin/bash
/sbin/iptables -N SSH
/sbin/iptables -N SSH_BLACKLIST
/sbin/iptables -A SSH_BLACKLIST -m recent --name SSH_COUNTER --set -j LOG --log-level warn --log-prefix "Blocked: "
/sbin/iptables -A SSH_BLACKLIST -j REJECT
/sbin/iptables -A SSH -m recent --name SSH_COUNTER --update --seconds 300 -j REJECT
/sbin/iptables -A SSH -m recent --name SSH --rcheck --seconds 60 --hitcount 5 -j SSH_BLACKLIST
/sbin/iptables -A SSH -m recent --name SSH --rcheck --seconds 2 -j LOG --log-level warn --log-prefix "Added: "
/sbin/iptables -A SSH -m recent --name SSH --update --seconds 2 -j REJECT
/sbin/iptables -A SSH -m recent --name SSH_COUNTER --remove -j LOG --log-level warn --log-prefix "Removed: "
/sbin/iptables -A SSH -m recent --name SSH --set -j ACCEPT
/sbin/iptables -A INPUT -m state --state NEW -p tcp -m tcp --dport 22 -j SSH

This creates two new tables, SSH and SSH_BLACKLIST.   Upon the intial connection attempt, the IP is added to the SSH_COUNTER counter.   If the same IP address is seen again within 60 seconds, it is duly noted–however no action is taken until the hitcount reaches 5.   In that case, the rules jump to the SSH_BLACKLIST table, it is logged, and subsequent connections from that IP are dropped for 5 minutes until things calm down.   In order to do this for FTP, just rename the targets as appropriate and change the target port to 21 on the last line.

The nice thing about this set up is that it is auto-cleaning.  After 5 minutes of no activity, the counter forgets about the IP address and things return to normal.   We’ve found that this is just enough protection to drastically reduce bruteforce attempts, yet not get in the way of normal usage by our customers.  Over time, this has become our favorite technique and we’ve begun to implement it on any Internet-facing machine with open SSH ports.

http://www.thelinuxfix.com/blog/2008/10/24/trixbox-vpns-and-the-20-second-issue/

The best I could get was Time Machine to begin to write the files to the share, but after a few seconds die with a vague “Backup disk could not be created error”.  This stumped me for a bit until I came across this.  Apparently sometime around 10.5.2, Apple introduced a new, undocumented “feature” to Time Machine that causes it to fail over network volumes when doing the initial backup.   However, once the files are created it will work fine.

So, the magical combination is as follows:

• CentOS 5 with Netatalk-2.0.3 compiled, installed, and configured per this post.  Note: I also had to modify etc/cnid_dbd/dbif.c with the same code change as specified there, but YMMV.
• OS X Leopard, patched to 10.5.4.
• Changes to Netatalk’s netatalk.conf file per this post at the Gentoo Wiki.
• Following the post linked above precisely.

Once that happened, Time Machine has begun to work great over AFP to our backup volume–even for multiple Macs connecting to the same share. Behold!

It probably doesn’t need saying; but this is clearly an unsupported way to use Time Machine.  It has been running this way for me only about a day.    If you’re concerned about having to troubleshoot problems that may pop up down the road, especially regarding backups; picking up a Time Capsule is probably a far better idea.

So the time had come to build out a rig and see if this new idea would work. Worries about a failure of the test were unwarranted: If I ran into a major problem while testing, I would always be able to repurpose the equipment in another fashion.

Here’s a crude diagram of what I was trying to achieve.

As you probably notice, there is a blatant single point of failure–the ethernet switch. Normally, intuition says implement two switches and do failover. Problem is that would require more money in terms of a second switch and additional NICs for the client servers. In my experience the switch would be the least likely to fail, while the servers would be the most likely–so it was a calculated and designed-in risk. Besides, it would be trivial to add another switch and do it “right” at a later date.

I had no prior experience shopping for dedicated disk storage, so to get an idea of what was available I shot an email off to ePowerhouse PC, a customer of ours. I explained the goals we were trying to achieve as well as our need for low entry price.

Terry at ePC has been a great resource for getting The Linux Fix parts quickly, and uncommon ones at that. He made a suggestion of using arrays from a company called Infortrend, since ePC was already a reseller for that particular vendor. After doing some more research I found an Infortrend fiber-based array that used lower-priced (but still high performing) SATA-II hard disks for storage. After some specification digging and decision making, I chose the EonStor A24F-R2430, which provides dual-redundant SATA-to-3bgps Fiber controller modules with an eight-port integrated fiber switch. A nice, elegant, all-in-one solution. Did I mention it supports 24 hard drives? TLF’s storage needs would be solved for quite some time! It turned out to be a great choice, a perfect blend of value and performance for us.

For the head-end NFS cluster servers, I chose a pair of low-end Dell PowerEdge 860 1U servers. The PowerEdge 860 has an available PCIe slot for a fiber host bus adapter (HBA), as well as an optional quad-core Xeon. Being that these servers were going to be clustered, I was not overly concerned with individual redundancy and opted for a cheaper software-based RAID solution on each. However, the pricey Dell Remote Access Controller was necessity because of RedHat’s Cluster Suite. RHCS requires a “fencing” device in order to reboot a server remotely in the event of error–and the DRAC suffices for that task (there are shortcomings to this, do your own research!). As I had mentioned in the previous article, fiber equipment is insanely expensive; in our case the Emulex fiber HBA was 20% of the server price! However, the lower entry price of the PowerEdge 860 offset that and we still ended up with a bargain.

As for the ethernet switch, I fell back to my old reliable favorite, the Dell PowerConnect 5324. It’s worked well for TLF in the past, is reliable, and provides plenty of bang for the buck.

Finally, I purchased twenty four 250 gigabyte SATA-II drives. At the time, 250 gig hard disks were the price-to-value point, and depending on the type of RAID used on the array, we would end up with roughly 5 terabytes of usable space. An important detail that Terry mentioned to me was to ensure the hard drive model was included on Infortrend’s compatibility list, and that made the decision on which models to purchase easier.

Overall price tag for the equipment came out to approximately $16,000, and broke down about like this:

PowerEdge 860 Servers (w/HBA): $1,200/ea
PowerConnect 5234 GigE Switch: $800
EonStor A24F-R2430: $11,000
Seagate SATA HDs 250GB: $2800

Sounds expensive, but when pricing out equivalent storage options from a well-known vendor such as NetApp, $3.00/gb isn’t a bad deal at all. The only thing left was to ensure I was going to get a well performing, expandable storage network.

All that was left to do was wait for the equipment to arrive and being my tests, which will be the topics I cover in the next post. Stay tuned!

When starting out, local disk storage worked just fine for us. A few large SAS or SCSI drives behind a cached RAID controller offered plenty of performance. Most web hosts start with a single server, and there is really no need to look any further. In fact, that local storage sufficed for us quite a long time–three years–even with a few servers in the mix. When we started running short on other boxes we simply exported some via NFS from the server with space.

Of course this didn’t work for long. The time came when we needed more than one web server to host the same sites. NFS still worked for that, but the problem turned into an architectural one–a “spiderweb” of servers hosting data to each other, letting available storage dictate the network design.

Mixing application and storage boxes is just simply not a good idea. Though it may work at first, tuning and maintaining your installation becomes far more difficult than it needs to be–and that in turn can cause your customers grief. Stop right there! The time had come for us to invest in a central storage platform. This is the first serious dive into investing that The Linux Fix needed to make, and it had dollar signs floating like sugar plums in my head.

Researching storage options always lead to one of two solutions: Network based (NAS, usually over NFS) or direct-attached SAN. First, some background on the idea behind both.

A NAS device sits on the network with the rest of your servers and gets an IP address–just like any other server. Storage from the NAS is served up via the protocol of your choice (NFS for Linux/Unix) and usually doesn’t have a very large price tag. The downside, of course, is that the network is then taxed with your storage traffic and is often limited in expansion to the device or server select to perform this duty. Not much of an improvement over the setup of old other than delaying the inevitable.

Direct attached is basically just a NAS with the ability to plug servers directly into it, usually via fiber channel. This offers fantastic performance, but isn’t really an option unless there is money to burn. Entry level SAN disk arrays ($$$) only offer a few direct-attach points, for which you will need fiber-HBA cards for your servers ($$$). Then when running out of direct-attach points on your SAN, a fiber switch ($$$) is needed. After that is said and done, spending $10,000 building out a fiber-based network isn’t unrealistic–and we had not considered the cost of disks yet.

While pondering the situation, an idea hit. Why not build an ethernet-based SAN, instead of a fiber-based SAN? This would combine the best of both worlds: A dedicated gigabit ethernet network carrying NFS traffic. The final rendition would be an expandable NFS cluster, direct attached to a fiber disk array with redundant controllers and cache. The expensive part–fiber cards and disk array–would remain a fixed cost, and expansion could be done in terms of cheap methods, namely ethernet switches and NFS servers!

So a plan was in place, and it needed some testing before I was going to bet the business on it. Stay tuned, and I’ll touch on the steps taken to build out and test this idea in the next articles.