Jan 16 2008
Building the EtherSAN: Part 1
One of the first roadblocks I ran into growing The Linux Fix was disk space. Even a few years ago our primary storage file system was exceeding 450 gigabytes, and growing fast making a flexible storage a key point. After some trials (and errors!) the most recent and cost effective solution has become something we like to call the “Ether-SAN”.
When starting out, local disk storage worked just fine for us. A few large SAS or SCSI drives behind a cached RAID controller offered plenty of performance. Most web hosts start with a single server, and there is really no need to look any further. In fact, that local storage sufficed for us quite a long time–three years–even with a few servers in the mix. When we started running short on other boxes we simply exported some via NFS from the server with space.
Of course this didn’t work for long. The time came when we needed more than one web server to host the same sites. NFS still worked for that, but the problem turned into an architectural one–a “spiderweb” of servers hosting data to each other, letting available storage dictate the network design.
Mixing application and storage boxes is just simply not a good idea. Though it may work at first, tuning and maintaining your installation becomes far more difficult than it needs to be–and that in turn can cause your customers grief. Stop right there! The time had come for us to invest in a central storage platform. This is the first serious dive into investing that The Linux Fix needed to make, and it had dollar signs floating like sugar plums in my head.
Researching storage options always lead to one of two solutions: Network based (NAS, usually over NFS) or direct-attached SAN. First, some background on the idea behind both.
A NAS device sits on the network with the rest of your servers and gets an IP address–just like any other server. Storage from the NAS is served up via the protocol of your choice (NFS for Linux/Unix) and usually doesn’t have a very large price tag. The downside, of course, is that the network is then taxed with your storage traffic and is often limited in expansion to the device or server select to perform this duty. Not much of an improvement over the setup of old other than delaying the inevitable.
Direct attached is basically just a NAS with the ability to plug servers directly into it, usually via fiber channel. This offers fantastic performance, but isn’t really an option unless there is money to burn. Entry level SAN disk arrays ($$$) only offer a few direct-attach points, for which you will need fiber-HBA cards for your servers ($$$). Then when running out of direct-attach points on your SAN, a fiber switch ($$$) is needed. After that is said and done, spending $10,000 building out a fiber-based network isn’t unrealistic–and we had not considered the cost of disks yet.
While pondering the situation, an idea hit. Why not build an ethernet-based SAN, instead of a fiber-based SAN? This would combine the best of both worlds: A dedicated gigabit ethernet network carrying NFS traffic. The final rendition would be an expandable NFS cluster, direct attached to a fiber disk array with redundant controllers and cache. The expensive part–fiber cards and disk array–would remain a fixed cost, and expansion could be done in terms of cheap methods, namely ethernet switches and NFS servers!
So a plan was in place, and it needed some testing before I was going to bet the business on it. Stay tuned, and I’ll touch on the steps taken to build out and test this idea in the next articles.
