For now, visit the site and sign up for the mailing lists!   We’re still in the early phases, so be sure to suggest ideas and comments for the focus of the group.

One side of the coin, I’m an avid technologist.    I own an obscene number of computers, both Mac and PC.   I’ve got an iPhone, and owned one of each generation.   I cancelled my cable nearly a year ago and stream Netflix and local channels via Windows Media Center on a computer hooked up to my television.   I don’t own a landline; all my telephone service is wireless or VoIP.    Even my business is completely done on the web with Quickbooks Online and Google Sites, Gmail, and Docs.

Now flip that coin over:    I have three, yes three, apps on my iPhone.   I don’t use LinkedIn, Twitter, or Facebook.   I have no desire whatsoever for a tablet PC or an iPad.   I’m seriously considering purchasing an older (pre-1975) car to re-build as a daily driver and dumping my 2007 Aveo.   I still write checks, fill out forms with a ball point pen,  and do my bills via snail mail.    I’ll call someone instead of texting them.

So what gives?  Why the dual personalities?

I’ve been wondering that myself lately, and started keeping track of why I tend to do things a certain way versus another.   For instance, I’ll not think twice  in allowing Quickbooks Online (which is great, by the way) automatically post transactions in our business account directly from Chase, but minutes later I’ll be writing a check and affixing postage to an envelope addressed to the office insurance company.   Insanity perhaps?

Well, no, as a matter of fact.   You see my moment of clarification resulted from an airbag light and an accelerator pump.   Let’s back up six months, and put our gearhead hat (or welding visor, for you true hardcore grease monkeys) on for a few moments.

I’ve had a lingering problem with the Camaro that anyone with old-car experience has probably felt before.   When dropping the gas pedal to the floor, the car would stutter and hesitate for a moment (kind of embarassing in a muscle car!) and then finally pick up steam and roar ahead.   The reason for this is due to the carburetor’s throttle plates suddenly opening all the way, and all the vacuum in the engine’s intake system disappearing for a moment while the engine’s RPMs spool up.  For those not-quite-car-savvy readers out there, vacuum in the engine is what pulls the gas/air mix in and allows it to go ‘vroom’!  No vacuum, no vroom.

Over the decades carburetor designers figured out that if a small syringe-like pump–called an accelerator pump–was physically hooked up to the cabling coming from the gas pedal, they could preempt this pause with a small squirt of straight gasoline into the throat of the engine. This provided some temporary “oomph” for a few seconds while the vacuum caught up with the driver’s right foot.  Clever thinking, to be sure.

Now, let’s fast forward back to modern-day.   My 2007 Aveo has an Airbag light on.   Much to my chagrin, it is completely undiagnosable sitting in my garage.  You must take it to a service center with the appropriate code scanners to even know what’s wrong.

So here comes my epiphany.

As an engineer and a tinkerer, I want control.   However, I’m also lazy.  What I’ve found is that if something  just works, I’m willing to allow it to work behind a set of curtains–even if that means raising an eyebrow and surrendering a bit of control.   However,  if something is problematic nothing angers me more than not being able to fix it myself.  In fact, I’ll even stop using it or throw it away.

I own the 2007 Aveo.  I have the title, and it is lien-free.   However, I couldn’t even obtain the specifications on how to diagnose the airbag system on the car even if I wanted to.   Chevrolet doesn’t make them available, I’m sure because they’re afraid someone will set the airbags off by accident.    So the question is, do I really own it?   I’m still subservient to the manufacturer, even though I’ve paid for the product.

I also own my Camaro, and last weekend I finally fixed the hesitation problem by bending a piece of wire on the carburetor with a pair of pliers.

I’ve realized that this same disparity lack of control and trust is my issue with software and technology in general.   As software is getting more and more complex, and nearly everything is online 100% of the time, quality and control  is seemingly going south along with it.  I doubt this is intentional by software developers;  more likely its just the vast task of maintaining and QA’ing gigantic amount of source code.   I’ve recently noted that when software or a piece of digital tech fails me, I get very bitter and don’t often go back and use it again.   The same thing is happening with my 2007 Aveo, and I’m once again behaving irrationally about it.

So for all you software developers out there:   Take a lesson from the carburetor designers of an era past.   It’s cute to be clever, but simplicity and reliability are true genius. It seems software is going the wrong direction–placing features higher in importance over function and reliably.

Get back to to basics, and just write damn tight code.   Your user base will thank you.

I’ll keep the Haynes manual handy, if you need to borrow it.

The problem is tricky–we must leave FTP and SSH open to the entire world, but at the same time be selective on what we black list.   How do you make that determination?  Strictly on bad login credentials?

We could, but that would mean that we’d inadvertently lock out real users.  A better solution we found has to do with timing connection attempts.   With IPTables, we can keep a counter based upon source IP–and track how many new socket attempts are made within a certain span of time.     For instance, if we detect the IP address 1.2.3.4 making 5 connection attempts within 60 seconds, there is a darn good chance it isn’t someone mistyping a password.

Here is how we did it, based upon another script we found out in the Internets:

#!/bin/bash
/sbin/iptables -N SSH
/sbin/iptables -N SSH_BLACKLIST
/sbin/iptables -A SSH_BLACKLIST -m recent --name SSH_COUNTER --set -j LOG --log-level warn --log-prefix "Blocked: "
/sbin/iptables -A SSH_BLACKLIST -j REJECT
/sbin/iptables -A SSH -m recent --name SSH_COUNTER --update --seconds 300 -j REJECT
/sbin/iptables -A SSH -m recent --name SSH --rcheck --seconds 60 --hitcount 5 -j SSH_BLACKLIST
/sbin/iptables -A SSH -m recent --name SSH --rcheck --seconds 2 -j LOG --log-level warn --log-prefix "Added: "
/sbin/iptables -A SSH -m recent --name SSH --update --seconds 2 -j REJECT
/sbin/iptables -A SSH -m recent --name SSH_COUNTER --remove -j LOG --log-level warn --log-prefix "Removed: "
/sbin/iptables -A SSH -m recent --name SSH --set -j ACCEPT
/sbin/iptables -A INPUT -m state --state NEW -p tcp -m tcp --dport 22 -j SSH

This creates two new tables, SSH and SSH_BLACKLIST.   Upon the intial connection attempt, the IP is added to the SSH_COUNTER counter.   If the same IP address is seen again within 60 seconds, it is duly noted–however no action is taken until the hitcount reaches 5.   In that case, the rules jump to the SSH_BLACKLIST table, it is logged, and subsequent connections from that IP are dropped for 5 minutes until things calm down.   In order to do this for FTP, just rename the targets as appropriate and change the target port to 21 on the last line.

The nice thing about this set up is that it is auto-cleaning.  After 5 minutes of no activity, the counter forgets about the IP address and things return to normal.   We’ve found that this is just enough protection to drastically reduce bruteforce attempts, yet not get in the way of normal usage by our customers.  Over time, this has become our favorite technique and we’ve begun to implement it on any Internet-facing machine with open SSH ports.

Using Solaris’ Dtrace functionality a systems engineer determines that noise is responsible for his intermittent disk latency problems.   Just wait for the last part of the video, it’s worth it.

http://blogs.sun.com/brendan/entry/unusual_disk_latency

Thanks to Greg for pointing that one out to me.

One of the things that annoyed me out of the gate is the lack of SSH support.   It’s there in the underlying operating system, just not enabled.   Here’s how to turn it on:

1. Get on the console of the ESXi server.
2. Press ALT-F1 to get to the OS system console
3. Type “unsupported”
4. Enter the root password at the password prompt.
5. Edit /etc/inetd.conf with vi, and uncomment the SSH line
6. Run:  kill -1 $(cat /var/run/inetd.pid)

And viola!  SSH to your ESX box.   Enjoy!

The best I could get was Time Machine to begin to write the files to the share, but after a few seconds die with a vague “Backup disk could not be created error”.  This stumped me for a bit until I came across this.  Apparently sometime around 10.5.2, Apple introduced a new, undocumented “feature” to Time Machine that causes it to fail over network volumes when doing the initial backup.   However, once the files are created it will work fine.

So, the magical combination is as follows:

• CentOS 5 with Netatalk-2.0.3 compiled, installed, and configured per this post.  Note: I also had to modify etc/cnid_dbd/dbif.c with the same code change as specified there, but YMMV.
• OS X Leopard, patched to 10.5.4.
• Changes to Netatalk’s netatalk.conf file per this post at the Gentoo Wiki.
• Following the post linked above precisely.

Once that happened, Time Machine has begun to work great over AFP to our backup volume–even for multiple Macs connecting to the same share. Behold!

It probably doesn’t need saying; but this is clearly an unsupported way to use Time Machine.  It has been running this way for me only about a day.    If you’re concerned about having to troubleshoot problems that may pop up down the road, especially regarding backups; picking up a Time Capsule is probably a far better idea.

Imagine a script wherein a process needs to be checked for proper exit. Let’s say “mysqldump”. Typically I’d do something like this, for example:


#!/bin/bash
STATUS=1
while [ ${STATUS} -ne 0 ]
do
mysqldump -uroot -psomepass --all-databases > sql-backup.sql
STATUS=${?}
done


exit 0


That’ll work just fine–the special reserved variable ${?} contains the exit code of the last run command. Mysqldump is kind enough to use non-zero ones on any kind of error, so if it doesn’t work in our script we’ll retry.

But for instance, let’s say our script looks like this:


#!/bin/bash
STATUS=1
while [ ${STATUS} -ne 0 ]
do
mysqldump -uroot -psomepass --all-databases | gzip > sql-backup.sql
STATUS=${?}
done


exit 0


The problem here is that ${?} now contains the exit code for gzip, not mysqldump! Will gzip respond properly if mysqldump doesn’t provide an input stream from the pipe? Maybe, maybe not. Bottom line is that it isn’t reliable, and not what I’d consider good shell programming.

Instead, check out this solution:


#!/bin/bash


STATUS=1
while [ ${STATUS} -ne 0 ]
do
mysqldump -uroot -psomepass --all-databases | gzip > sql-backup.sql
STATUS=${PIPESTATUS[0]}
done


exit 0


The BASH reserved array ${PIPESTAUTUS[x]} contains the exit codes for all programs in the array. In this example, ${PIPESTATUS[0]} is mysqldump, and ${PIPESTATUS[1]} is gzip.

Tip #1 – Start and stop your VMs from the command line

If your VMware server is headless and gui-less (you didn’t install a GUI did you?) it’s handy to be able to start and stop your VM processes with a command line tool over ssh. Use the vmware-cmd tool for this:

vmware-cmd /path/to/vmxfile.vmx stop <hard|soft>

or

vmware-cmd /path/to/vmxfile.vmx start

The third option is the powerop mode. ‘soft’ uses the VMware tools within the guest OS, while ‘hard’ simply powers on and off the VM without the tools.

Tip #2 – Re-install your VM Tools quickly

After upgrading your kernel on Linux-based virtual machines, you’ll also have to re-compile vmware tools’ kernel modules. Upon initial installation, you probably executed the usual:

/usr/bin/vmware-config-tools.pl

But did you know you can speed up the process and make it automatic by using the default options? The next time you need to recompile your tools, use this instead

/usr/bin/vmware-config-tools.pl -default

Tip #3 – Fine-grain your VM’s priority

VMware Server does not provide the flexibility of ESX, but you can get it part-way there by using the Linux scheduler to prioritize your virtual machines. By default, VS gives all vmware-vmx processes a nice value of “-10″. In Linux, processes with “-20″ have the highest priority for system resources, and “20″ have the lowest. By adjusting your busy VMs to a higher negative number (e.g. -15) and your less-intensive VMs to a higher positive number (e.g 0) you can more finely tune your server’s performance and ensure timeslices on the host are more accurately granted.

To do this, use the `renice’ command. First, find the PIDs of your vmware-vmx processes, by using `ps’:

[root@tlfvm5 ~]# ps -ef | grep vmware-vmx

root      3374     1 13 Mar18 ?        2-20:03:36 /usr/lib/vmware/bin/vmware-vmx -C /vmware/tlfmonitor/tlfmonitor.vmx -@ ""

root      4833     1 15 Mar18 ?        3-04:09:11 /usr/lib/vmware/bin/vmware-vmx -C /vmware/DellMonitor/DellMonitor.vmx -@ ""

Then renice the appropriate PID. For example, to give the “tlfmonitor” a bit of a bump to “-12″:

renice -12 33

Like all good things, moderation is key. Start with smaller increments and note the change, then if needed bump it again. It should be noted that your reniced values will disappear as soon as the PID terminates. You can also give it a default higher priority via the .vmx file in the prority.grabbed and priority.ungrabbed directives (see http://sanbarrow.com/vmx/vmx-config-ini.html).

Tip #4 – Manage and extend your virtual disks

VMware Server comes with a tool to completely manage your .vmdk disks. The vmware-vdiskmanager tool can create, defrag, extend, and convert vmdks from one type to another. For example, to expand a vmdk from 10GB to 15GB, power off the VM and issue this command:

vmware-vdiskmanager -x 15Gb /path/to/vmdkfile.vmdk

Note that this extends the raw disk, but not the guest file system. For instance, after doing an extend in Linux on an ext3 file system, use “resize2fs” to adjust it accordingly. You may want to run the vmware-vdiskmanager command without arguments to see some help on the different options, as well as some examples.

Tip #5 – Install VMware tools from the command line

You don’t need to click “VM -> Install Vmware Tools…” on the Server Console to mount the virtual media. Do it from the command line!

vmrun installtools /path/to/vmxfile.vmx

This does precisely what clicking in the GUI does. Once this has been run from the host, go to your VM and mount up the /dev/cdrom device and find your tools RPM ready to go.

That’s it for now. Do you have any tips that are useful for other VMware Server administrators? If so, let me know!

Let us assume you need to perform an operation on a bunch of files. I typically see it done as so in the BASH interpretor:


for I in `ls *.sh`
do
something
done


Now, performing the backtick (`) subshell generally causes some hard to handle data mangling issues. Most commonly a space in the file name or special character will result in odd permutations of ${I}. For example in a file listing such as:


file1.sh
file2.sh
file three.sh


Your resulting loop will end up with:


[bdowney@tlfmgt1 ~]$ for I in `ls *.sh`; do echo ${I}; done
file1.sh
file2.sh
file
three.sh


The subshell returns strings to `for’ which interprets whitespace as a new loop iteration. This obviously causes issues for your loop logic, and I have seen people build in some pretty elaborate methods to handle it. But it’s all for not–since `for’ is a reserved shell function, it inherently understands file globbing. Thus:


[bdowney@tlfmgt1 ~]$ for I in *.sh; do echo ${I}; done
file1.sh
file2.sh
file three.sh


Is all you need to do and solve your problem. But what if one has a list of filenames in a file, or needs to pass more arguments to provide a better list of said files (for example, ls -t?). Not to worry, the BASH built-in `read’ to the rescue!


[bdowney@tlfmgt1 ~]$ ls -tr *.sh | while read I; do echo ${I}; done
file1.sh
file2.sh
file three.sh


And as you’d expect, works the same when redirecting STDIN:

[bdowney@tlfmgt1 ~]$while read I; do echo ${I}; done < input.file


So stop wasting those extra shell processes!

The basic specs are as follows:

• VMWare Host: Dell PowerEdge 1950 (dual quad core, 8 gigs RAM)
• NFS Cluster: Two Dell PowerEdge 860s (single quad core, 4 gigs RAM each)
• Networking: Dell PowerConnect 5324
• Centralized storage: EonStor A24F-R2224
• Internal DRAC on each node for cluster fencing (not ideal, read below).
• All the VMDKs are stored on an NFS mount from the cluster.

Through quite a bit of experimentation and trial and error, I have it running pretty solid. Some of the key points:

1. Used RedHat EL 5 all the way around, with the related GFS/RHCS packages
2. Mounted GFS on the NFS nodes with noatime,noquota for some minor speed improvements.
3. NFS4 and TCP for everything. Makes failover to the other node more reliable.
4. Use ‘hard’ mounts on the VMWare server, use timeo=600,retrans=2 in your options. This allows TCP to handle transmission delays during a failover versus NFS.
5. On the export side, craft your /etc/exports so that each has a matching ‘fsid=’ for every export. This gets around stale handles.
6. Use the GFS/shared storage/floating IP technique as documented in the Cluster NFS Cookbook versus managed NFS (read why below).
7. Bonded the NICs on the NFS nodes for higher throughput (in our case they will be exporting to more than one server when in production, so this was necessary).
8. Spanning tree algorithm delays on the PowerConnect can get you in trouble with a fencing loop in a two-node setup. During a reboot situation one of the nodes, the NICs come up quicker during Linux sysinit than they do on the switch. Thus, Linux thinks the interface should be reachable (when it’s not) and when fenced attempts to initalize, it cannot reach the other node and consequently fences that one. Solution is to either add “LINKDELAY” to /etc/sysconfig/network or just disable spanning tree on the switch.

I intially tried the managed NFS setup in Cluster Suite (check the cookbook), however there are two major problems. At this time, managed NFS appears to be set up to use NFSv2 and v3 only, as there is no opportunity to modify the export options via Cluster Suite. Also, there are timing delays with how Cluster Suite manages the NFS daemons…

Of course, during a failover speed is of the essence. So, when I had this rig configured for managed NFS failover, I was experiencing 12+ second delays in failover. Why?

Well, it turns out RedHat has a sleep command in /usr/share/cluster/ip.sh (the virtual IP management script) that adds 10 seconds to the failver so NFSD can clear its cache (!?). Pretty hackish, and results in an unacceptable delay during a failovers. Unfortunately if you’re running managed NFS, there’s no real way around this unless you want to risk corruption of NFSD going down without flushing its cache to disk.

I found this in the Cluster Project FAQ. With the ‘sleep 10′ command gone, failover is much, much quicker. As long as you’re doing the GFS thing versus the managed NFS setup this works quite nicely and fast enough that VMWare doesn’t seem to know the better of what is going on.

Performance-wise, it’s pretty darn good. I have a dedicated PowerConnect 5324 for use as an “ethernet SAN” to interconnect the NFS nodes, and VMWare Server. That being said, 20 concurrent lightly loaded VMs results in nothing abnormal in terms of performance or reliability. In fact, it’s hard to tell the difference from local disk–even during a failover. The NIC being used for “front-end” access to the VMWare Server Console even seems to experience more traffic than the NFS one according to the PowerConnect’s interface reports, though that leaves me a bit skeptical.

I would have been interesting to see if the TOE (TCP Offload Engine) equipped on the 1950′s NetExtremeII NICs would have made a performance improvment, but it works in Windows 2003 only. Bummer.

Another “gotcha” to watch for is using RAC cards on the servers for fencing purposes. In most cases it works fine however when power is lost to the entire server, the DRAC goes down with it and becomes unreachable. This leaves the surviving node stuck trying to fence the dead one, and failover never occurs. A better option would be to use a managable PDU (which we’ll do ultimately).

Bottom line it seems to work very well in almost all failure situations I’ve tested it in. The only time I was able to make it fail (badly) was to yank the power cord out of one of the cluster nodes, and have the entire cluster crunch to a halt, due to the problem I mentioned above. I did this while installing X Windows on 50% of the VMs to simulate a lot of NFS write activity.

After bringing the entire cluster back up manually, the only damage was a corrupt RPM DB on one of the VMs. The others came back up fine after a fsck on boot. Not bad!

After my testing, I’m confident this set up will work in a production environment. If you wish to do try the same, ensure your testing plan includes every possible outage situation you can fathom. Weird/odd stuff can come up (for example the spanning tree thing) and of course it is far better to nail those down in R&D than in production!

UPDATE 2011-02-22:

I couldn’t help but notice that this post garners a lot of search traffic, even to this day.   At this point in time RedHat no longer supports this type of NFS/GFS configuration.   This came up in a support call a while back and was confirmed by a high-level engineer at RedHat.

Since CentOS is essentially just RedHat de-branded, this holds true for CentOS as well.  You may experience success with this configuration, but be warned that in recent releases you may encounter data loss if specific events occur that cause cluster nodes to end servicing the cluster abnormally.