AdmiNirvana

This is one of those “no way, that can’t be true” type of events.

Using Solaris’ Dtrace functionality a systems engineer determines that noise is responsible for his intermittent disk latency problems.   Just wait for the last part of the video, it’s worth it.

http://blogs.sun.com/brendan/entry/unusual_disk_latency

Thanks to Greg for pointing that one out to me.

So, I finally had a chance to play with VMware ESXi.   It’s pretty much what I expected, a straight-up version of ESX.  Very, very nice… I’ll start moving more servers over from VMware Server 1.x and report back on my progress.

One of the things that annoyed me out of the gate is the lack of SSH support.   It’s there in the underlying operating system, just not enabled.   Here’s how to turn it on:

1. Get on the console of the ESXi server.
2. Press ALT-F1 to get to the OS system console
3. Type “unsupported”
4. Enter the root password at the password prompt.
5. Edit /etc/inetd.conf with vi, and uncomment the SSH line
6. Run:  kill -1 $(cat /var/run/inetd.pid)

And viola!  SSH to your ESX box.   Enjoy!

I have been itching to get Time Machine to work over a (unsupported) network-shared volume since we’re unwilling to drop the cash for a Time Capsule.  I’ve seen quite a few different posts on how to get it done, including the terminal trick to tell Time Machine to use unsupported volumes, but none of them quite worked.

The best I could get was Time Machine to begin to write the files to the share, but after a few seconds die with a vague “Backup disk could not be created error”.  This stumped me for a bit until I came across this.  Apparently sometime around 10.5.2, Apple introduced a new, undocumented “feature” to Time Machine that causes it to fail over network volumes when doing the initial backup.   However, once the files are created it will work fine.

So, the magical combination is as follows:

• CentOS 5 with Netatalk-2.0.3 compiled, installed, and configured per this post.  Note: I also had to modify etc/cnid_dbd/dbif.c with the same code change as specified there, but YMMV.
• OS X Leopard, patched to 10.5.4.
• Changes to Netatalk’s netatalk.conf file per this post at the Gentoo Wiki.
• Following the post linked above precisely.

Once that happened, Time Machine has begun to work great over AFP to our backup volume–even for multiple Macs connecting to the same share. Behold!

It probably doesn’t need saying; but this is clearly an unsupported way to use Time Machine.  It has been running this way for me only about a day.    If you’re concerned about having to troubleshoot problems that may pop up down the road, especially regarding backups; picking up a Time Capsule is probably a far better idea.

I think I’ve mentioned many times on this blog, but one of the most satisfying things regarding Linux and Unix are that you’re never done learning about it. A never-ending lesson in operating systems! Well, chalk up another lightbulb moment for me this morning.

Imagine a script wherein a process needs to be checked for proper exit. Let’s say “mysqldump”. Typically I’d do something like this, for example:


#!/bin/bash
STATUS=1
while [ ${STATUS} -ne 0 ]
do
mysqldump -uroot -psomepass –all-databases > sql-backup.sql
STATUS=${?}
done


exit 0


That’ll work just fine–the special reserved variable ${?} contains the exit code of the last run command. Mysqldump is kind enough to use non-zero ones on any kind of error, so if it doesn’t work in our script we’ll retry.

But for instance, let’s say our script looks like this:


#!/bin/bash
STATUS=1
while [ ${STATUS} -ne 0 ]
do
mysqldump -uroot -psomepass –all-databases | gzip > sql-backup.sql
STATUS=${?}
done


exit 0


The problem here is that ${?} now contains the exit code for gzip, not mysqldump! Will gzip respond properly if mysqldump doesn’t provide an input stream from the pipe? Maybe, maybe not. Bottom line is that it isn’t reliable, and not what I’d consider good shell programming.

Instead, check out this solution:


#!/bin/bash


STATUS=1
while [ ${STATUS} -ne 0 ]
do
mysqldump -uroot -psomepass –all-databases | gzip > sql-backup.sql
STATUS=${PIPESTATUS[0]}
done


exit 0


The BASH reserved array ${PIPESTAUTUS[x]} contains the exit codes for all programs in the array. In this example, ${PIPESTATUS[0]} is mysqldump, and ${PIPESTATUS[1]} is gzip.

As anyone whom reads this blog regularly knows, I’m a happy VMware Server user. In using it, I’ve come across some handy methods in administrating it and the virtual machines created with it. Without further ado, here they are!

Tip #1 - Start and stop your VMs from the command line

If your VMware server is headless and gui-less (you didn’t install a GUI did you?) it’s handy to be able to start and stop your VM processes with a command line tool over ssh. Use the vmware-cmd tool for this:

vmware-cmd /path/to/vmxfile.vmx stop <hard|soft>

or

vmware-cmd /path/to/vmxfile.vmx start

The third option is the powerop mode. ’soft’ uses the VMware tools within the guest OS, while ‘hard’ simply powers on and off the VM without the tools.

Tip #2 - Re-install your VM Tools quickly

After upgrading your kernel on Linux-based virtual machines, you’ll also have to re-compile vmware tools’ kernel modules. Upon initial installation, you probably executed the usual:

/usr/bin/vmware-config-tools.pl

But did you know you can speed up the process and make it automatic by using the default options? The next time you need to recompile your tools, use this instead

/usr/bin/vmware-config-tools.pl -default

Tip #3 - Fine-grain your VM’s priority

VMware Server does not provide the flexibility of ESX, but you can get it part-way there by using the Linux scheduler to prioritize your virtual machines. By default, VS gives all vmware-vmx processes a nice value of “-10″. In Linux, processes with “-20″ have the highest priority for system resources, and “20″ have the lowest. By adjusting your busy VMs to a higher negative number (e.g. -15) and your less-intensive VMs to a higher positive number (e.g 0) you can more finely tune your server’s performance and ensure timeslices on the host are more accurately granted.

To do this, use the `renice’ command. First, find the PIDs of your vmware-vmx processes, by using `ps’:

[root@tlfvm5 ~]# ps -ef | grep vmware-vmx

root      3374     1 13 Mar18 ?        2-20:03:36 /usr/lib/vmware/bin/vmware-vmx -C /vmware/tlfmonitor/tlfmonitor.vmx -@ ""

root      4833     1 15 Mar18 ?        3-04:09:11 /usr/lib/vmware/bin/vmware-vmx -C /vmware/DellMonitor/DellMonitor.vmx -@ ""

Then renice the appropriate PID. For example, to give the “tlfmonitor” a bit of a bump to “-12″:

renice -12 33

Like all good things, moderation is key. Start with smaller increments and note the change, then if needed bump it again. It should be noted that your reniced values will disappear as soon as the PID terminates. You can also give it a default higher priority via the .vmx file in the prority.grabbed and priority.ungrabbed directives (see http://sanbarrow.com/vmx/vmx-config-ini.html).

Tip #4 - Manage and extend your virtual disks

VMware Server comes with a tool to completely manage your .vmdk disks. The vmware-vdiskmanager tool can create, defrag, extend, and convert vmdks from one type to another. For example, to expand a vmdk from 10GB to 15GB, power off the VM and issue this command:

vmware-vdiskmanager -x 15Gb /path/to/vmdkfile.vmdk

Note that this extends the raw disk, but not the guest file system. For instance, after doing an extend in Linux on an ext3 file system, use “resize2fs” to adjust it accordingly. You may want to run the vmware-vdiskmanager command without arguments to see some help on the different options, as well as some examples.

Tip #5 - Install VMware tools from the command line

You don’t need to click “VM -> Install Vmware Tools…” on the Server Console to mount the virtual media. Do it from the command line!

vmrun installtools /path/to/vmxfile.vmx

This does precisely what clicking in the GUI does. Once this has been run from the host, go to your VM and mount up the /dev/cdrom device and find your tools RPM ready to go.

That’s it for now. Do you have any tips that are useful for other VMware Server administrators? If so, let me know!