AdmiNirvana

I happened to run across this fella today, who also runs a very large VMware Server farm in a production environment. He makes a few mentions of his architecture which is slightly different than our approach, however it could prove handy to someone else building out such a thing.

Vim is a great editor. I don’t want to start an EMACS versus vim flame war here. It is just personal preference: I “get” vim, whereas EMACS confuses me.

Anyway, here’s a clever little tip for users of vim that often have to open both DOS and UNIX format files (you know, the old ^M line feed problem):

While in the editor you can write out either format.

:set ff=unix
:w

Sets the file to UNIX linefeeds, while

:set ff=dos
:w

Sets them to DOS. Enjoy!

In the first part of “Building the EtherSan”, I outlined The Linux Fix’s business needs and reasons for building out an ethernet-based storage area network (SAN), a relatively uncommon platform. In this part I discuss the design and products that were chosen to prove its viability before purchasing equipment for testing.

So the time had come to build out a rig and see if this new idea would work. Worries about a failure of the test were unwarranted: If I ran into a major problem while testing, I would always be able to repurpose the equipment in another fashion.

Here’s a crude diagram of what I was trying to achieve.

As you probably notice, there is a blatant single point of failure–the ethernet switch. Normally, intuition says implement two switches and do failover. Problem is that would require more money in terms of a second switch and additional NICs for the client servers. In my experience the switch would be the least likely to fail, while the servers would be the most likely–so it was a calculated and designed-in risk. Besides, it would be trivial to add another switch and do it “right” at a later date.

I had no prior experience shopping for dedicated disk storage, so to get an idea of what was available I shot an email off to ePowerhouse PC, a customer of ours. I explained the goals we were trying to achieve as well as our need for low entry price.

Terry at ePC has been a great resource for getting The Linux Fix parts quickly, and uncommon ones at that. He made a suggestion of using arrays from a company called Infortrend, since ePC was already a reseller for that particular vendor. After doing some more research I found an Infortrend fiber-based array that used lower-priced (but still high performing) SATA-II hard disks for storage. After some specification digging and decision making, I chose the EonStor A24F-R2430, which provides dual-redundant SATA-to-3bgps Fiber controller modules with an eight-port integrated fiber switch. A nice, elegant, all-in-one solution. Did I mention it supports 24 hard drives? TLF’s storage needs would be solved for quite some time! It turned out to be a great choice, a perfect blend of value and performance for us.

For the head-end NFS cluster servers, I chose a pair of low-end Dell PowerEdge 860 1U servers. The PowerEdge 860 has an available PCIe slot for a fiber host bus adapter (HBA), as well as an optional quad-core Xeon. Being that these servers were going to be clustered, I was not overly concerned with individual redundancy and opted for a cheaper software-based RAID solution on each. However, the pricey Dell Remote Access Controller was necessity because of RedHat’s Cluster Suite. RHCS requires a “fencing” device in order to reboot a server remotely in the event of error–and the DRAC suffices for that task (there are shortcomings to this, do your own research!). As I had mentioned in the previous article, fiber equipment is insanely expensive; in our case the Emulex fiber HBA was 20% of the server price! However, the lower entry price of the PowerEdge 860 offset that and we still ended up with a bargain.

As for the ethernet switch, I fell back to my old reliable favorite, the Dell PowerConnect 5324. It’s worked well for TLF in the past, is reliable, and provides plenty of bang for the buck.

Finally, I purchased twenty four 250 gigabyte SATA-II drives. At the time, 250 gig hard disks were the price-to-value point, and depending on the type of RAID used on the array, we would end up with roughly 5 terabytes of usable space. An important detail that Terry mentioned to me was to ensure the hard drive model was included on Infortrend’s compatibility list, and that made the decision on which models to purchase easier.

Overall price tag for the equipment came out to approximately $16,000, and broke down about like this:

PowerEdge 860 Servers (w/HBA): $1,200/ea
PowerConnect 5234 GigE Switch: $800
EonStor A24F-R2430: $11,000
Seagate SATA HDs 250GB: $2800

Sounds expensive, but when pricing out equivalent storage options from a well-known vendor such as NetApp, $3.00/gb isn’t a bad deal at all. The only thing left was to ensure I was going to get a well performing, expandable storage network.

All that was left to do was wait for the equipment to arrive and being my tests, which will be the topics I cover in the next post. Stay tuned!

Any true geek is impressed by a neat cabling job.  There’s something about the order that results from equally banded cables in conjunction with blinking lights that makes the heart warm.Here are a few snapshots of the cleanest cabling jobs I’ve seen.  Compare that with what I thought was a nice cabling job (sorry for the poor quality, cell phone shot) from one of the Linux Fix racks, and I feel like a slob!

A clever fellow I ran across used a dash of strace folded in with some shell functionality to give the ubiquitous `cp’ command a wget-like progress bar:

% cp_p /mnt/raid/pub/iso/debian/debian-2.2r4potato-i386-netinst.iso > /dev/null 

76% [===========================================>                ]

Granted, you could probably do the same with a few pipes and `pv‘, but this gets a medal for cleverness!

Full blog post here.