Mar 30 2007
VMware Server in Production
After reading some of the Slashdot comments on an article regarding a VMWare Server web application performance test, I felt compelled to write up a “tips” post. I’m amazed how much misunderstanding (and assumption-making) there is regarding VMware and virtualization in general!
I’m an old hand at VMware. I began using it back in the late 90s when the company was still very young. Along the way, I’ve picked up quite a bit of knowledge of the product. Although VMware’s ESX Server offering is really “where it’s at”, when VMware Server was released early last year it brought true “server” virtualization within reach of small IT shops with very tiny budgets (like The Linux Fix!)
But first off, VMware Server is no replacement for ESX Server. ESX is a very robust part of the Virtual Infrastructure, within which you can do almost magical things. However if you have only a handful of VMs, or are running a small shop were you’d like to just set things up “proper” without ganging up all software onto one OS installation to keep things clean, VMware Server is a great way to make sure your getting the most of out the money you’ve invested into your server equipment. Especially since it doesn’t involve investing any more money–an especially great fit for us!
One misnomer is that VMware Server isn’t “stable” or enough of a “performer” for production use. This is absolutely wrong: VMware Server clunks along happily. For example, our environment regularly goes as much as six months without needing to take it down–and generally for some unrelated reason or to simply upgrade VMware itself. You cannot get the same performance numbers or quantity of VMs on a host as ESX, nor can you do the magic of balancing resources, but in most small environments that isn’t necessarily needed or even a problem.
What’s more important is that your entire setup is done properly from the beginning, and that mostly means hardware and the host operating system. Below are the things The Linux Fix has learned along the way, and if you follow them I’m sure you’ll experience the same success with VM-S that we have:
Tip #1 – The Host Machine
- For Pete’s sake: USE A REAL SERVER! Don’t hobble together a PC with some SATA drives and software RAID, planning on it running non-stop for years at a time. In our co-location, I see PC’s labeled “Production” all the time, and I just shake my head.Even the highest-end PC cannot give you the redundancy and sustained I/O that a properly designed server can. Using VMWare in any form makes the stability and reliability of your host a much more important factor. Do yourself a favor in the long term and *buy* a server from a reputable vendor. I’m talking Dell, Sun, or HP. Also get the (sometimes optional) “lights out” card, DRAC for the Dell, iLO for the HP, and SMC for Sun. Buy the optional redundant power supply. Opt for the better RAID controller with more cache. This will cost you more dough up front, but remember you’re making it back in server consolidation! In most calculations buying one “good” server is still cheaper than buying a bunch of cheap machines.You should also check the ESX Hardware Compatibility List and try to pick one off of there that suits your needs. The idea here is that if you *do* decide to step up to a full-blown ESX implementation later down the road, you won’t have to re-buy.
Buying “server” parts from SuperMicro or one of the others and putting it together yourself definitely doesn’t count.
Tip #2 – The Network
- Running more than a handful of virtual machines (over 10 moderately busy ones) on a server can stress the network port your server is plugged into. Even a gigabit link can get flooded with traffic–remember throughput doesn’t avoid congestion! Avoid some of this buy using port aggregation and trunking. Use a quality, manageable Layer-2 switch that has a high throughput-backbone and trunk 2, 3, or even 4 ports (if you have them) of your server’s NIC together in a load-balance & failover scheme. This will help spread out the load among multiple ports should avoid unintentionally DoS’ing your switch, and also provides a bit of redunancy at the network layer.
Tip #3 – The Host OS:
- Use CentOS with the “Minimal” installation option as a base for VMWare Server. Since it is based off of Red Hat Enterprise Linux, it’s very stable and just about guarantees compatibility with any vendor-specific server agents that you may (and should) run on your servers (see tip #4).Install CentOS, and use ‘yum update’ to update your system. Shutdown and disable the extra services: cupsd, NFS, smartd (your server’s management backplane should do the monitoring), pcmcia, sendmail, among the others. Try to set up CentOS so that there are no ports other than SSH listening on the network. Then do any necessary miscellaneous software installs (ntp, snmpd, etc.), install and configure VMWare Server, and *don’t touch it* from that point on. Don’t patch it, configure it, or fondle it in any way.Yes, I know–some admins would claim this as bad practice. However, this host’s copy of CentOS *should not* be facing the outside world or your clients. What you’re trying to do is emulate what ESX does: Make the host minimal and stable, remove the unecessary stuff and do not touch the rest.
Get around the security issues of this by securing your network. Again, disable any unnecessary network services. Use VLANs and VLAN-trunking to give your physical host a different network segment than your VMs. Set up a reputable hardware firewall (we like Fortigate) to control access between them . Make this VLAN accessible to those that must administer VMware itself.
Tip #4 – Use your server vendor’s management tools
- By all means, use your tools effectively. For example, Dell’s DRAC does far more than give you lights-out console access. It also provides a full management backplane, that in conjunction with the operating-system based server agents can provide information that use may use to proactively prevent down time. Funky DIMMs, bad spinning fans and disks, hot spots, and even CPU problems will all be detected by the hardware backplane far earlier than your software monitoring can. Set up SNMP and use it to forward traps to a good monitoring package, like Nagios. The goal here is to try and give you early warning on problems so you can fix them before they result in down time!
Tip #5 – Set up and tune your VMs carefully
- VMware Server doesn’t give you the robust resource scheduling that ESX does, but you can offset some of this by ensuring you only give your VMs the resources that they need, and that you manually load balance busy VMs on different physical hosts. Install the vmware-mui on your host, and use that to monitor how much your virtual machines are really using. Most decently busy Apache VMs can get away with as little as 192MB of RAM, but of course your mileage may vary. Monitor the OS in the virtual machine whenever you decrease allocated RAM to ensure your’re not swapping to virtual memory in the guest OS or doing too much buffering or caching. Giving it too much wastes resources, goes to disk cache in the VM, and requires extra work by the host to manage it all. Keep in mind: VM OS disk caching isn’t really important since a) you’re caching at the host OS already, and b) if you bought a server with a good RAID controller, it’s cached there too.Continually fine-tune your VMs so that they do not hog any more of the host’s resources than is absolutely necessary. DNS servers running djbdns usually do fine with 96MB of RAM. Don’t assign mutiple vCPUs in a VM unless you fully understand why that is good and bad (and most of the time, it’s bad).
Following those basic tips will easily allow 4-6 moderately loaded VM’s per physical core, which is right around what VMware claims VMware Server can provide. On a Sun X2200M I acquired for testing with four cores, I was able to concurrently run 25 VM’s in a light-to-medium load environment (network services) without over taxing the server at all. Quite an amazing feat for a free virtualization package!
All in all, The Linux Fix has approximately 100 virtual machines spread over four physical hosts. The money saved in power, rackspace, and management-related costs have easily saved us several thousands of dollars over the past year.
At some point, a return on ESX server’s performance makes it a better investment. Soon, we’ll have enough physical hosts to make the $6000 purchase a wise one: On ESX you can roughly double the amount of virtual machines any particular host can handle. At five physical machines running VMWare Server, we could technically reduce that to two, and sell the rest to recoup some of the licensing costs. Again, it requires you crunch your own numbers and see what works. The line at which that happens is different for everyone, but it always exists!
[...] tips for using VMware server in [...]
What do you do about disk storage and w/e for all of those VM’s?
I want to use VMServer in production but we are worried about disk I/O
DW:
Disk I/O is probably the first bottleneck you’ll run into.
Ensure you have a high-quality SAS/SCSI RAID controller with oodles of built-in cache. Dell’s PERC falls into this category.
Generally speaking, the only major difference between JBOD disk and SAN disk is that SANs have a large block of cache sitting in front of those disks. Loading up your box with plenty of spindles and onboard RAID cache helps lower the bar.
We’ve evaluated a few Sun X86 servers, but the problems we’ve seen is the low-quality RAID implementation on their boxes. The AMD Opteron setup on those boxes can handle plenty of VMs, but the disk subsystem (without a dedicated RAID controller and cache) is the first thing to crunch.
If you plan to “go big” on the cheap, look into setting up an Ethernet-based SAN using RHCS (CentOS has this rolled into it) and GNBD. But that’s probably a whole different post!
Wondering if you can provide some software advice (and maybe even a full spec). I’m a developer, not a hardware guy so I’m not quite sure about the optimal server to buy. I have several old (and I mean *OLD*) desktops in my home network that I’ve turned into network servers (DNS, DHCP) and file servers and I’d like to add additional servers for SVN, Trac, databases, etc. None of these will see huge traffic and don’t require massive resources. They’re probably best described as toys with a purpose for the moment.
I’m planning to buy one solid server to host most of this functionality on VMs. My plan is to run CentOS 5.0 on the server with dhcpd and named running on the host and at least 3 or 4 development-centric VMs (source control, build, web server, db server, etc). I’ll probably go with Dell since I’ve purchased from them in the past and feel comfortable with them.
Can you recommend a comprehensive and sufficient/optimal configuration?
Any suggestions would be greatly appreciated. As I mentioned, the hardware part isn’t really my thing.
I take exception to the remark “Buying server parts from SuperMicro or one of the others and putting it together yourself definitely doesn
George:
I never said there was anything *wrong* with building your own servers, just that if you want the greatest uptime go with someone who has already painstakingly done all of the work you concisely described–and backs up that work.
Here’s the gist: VMWare in any server rendition quite literally turns your x86 box into a mainframe. This is commonly held view among seasoned VMWare administrators. I have never met a mainframe administrator that has built their own mainframe.
Sure, it’s a bit unrealistic to expect someone to put a mainframe together. But in the same respect, I wouldn’t trust running 20-30 *mission critical* servers on a pile of parts I picked up from a wheel-and-deal vendor! The point is *not* whether it’s faster/better/cheaper, as I think you took the point as.
Instead, it’s about support and guarantee. If something goes wrong with your homebrew box–and it’s in a co-lo 2500 miles away, what are you going to do about it?
Me, I call Dell and they’re on the scene in under 4 hours. Problem solved, and I can get back to work.
[...] Need tips http://briandowney.net/blog/2007/03/30/vmware-server-in-production/ [...]
I agree with production environment stability, my production internet-server is running now for months without problem.
Use it with sarge 3.1, on cheap HP Server with 2 SATA 250gb Software Raid (easy installation with sarge), for a e-Larning Platform Moodle.
Very cheap.
Thanks for the article. I’m very new to the virtualization game and am looking for some advice on the CentOS install. I noticed you said choose a “minimal” install, could you please elaborate a bit for me? I’ve downloaded Cent OS 5 and there doesn’t appear to be a minimum option. I’ve put VMWare Server on a few Windows hosts and while things look pretty nice, I know that Windows is taking way too much memory. Any advice you may have would be greatly appreciated. Thank you.
Andrew:
CentOS5/RHEL5 has slimmed down the options a bit. Essentially all you really need to do for a ‘barebones’ install is uncheck GNOME in the Desktop category and Dial-up Networking support in the Base System category.
It’s not quite as slim as CentOS4, however it’s good enough. Make sure you still do the ‘yum update’ and update your installation after everything’s said and done.
Hello,
I’m testing with VMWare Server (free edition) but I like to know if it is possible to measure how much network traffic the VMs generate?
Kind regards
Maarten
Hi Maarten,
At this time threre’s no-built in method in Vmware Server to monitoring individual network activity. The closest thing you can do is run SNMP in each VM and monitor the transfer of data via the guest OS.
Thank you so much for re-adding this article. Can I ask if installing CENTOS without the gnome desktop strips it of any GUI what so ever? I am primarily a windows person and am a bit afraid of trying Linux again. I’ve had nightmares of a time to get it to operate with VMware in the past because my fundamental Linux knowledge is limited. I’ve configured about 30 Sco servers for application use but Linux seems to be very version specific and finding assistance when I hit a snag has been very difficult.
I am curious if you think I’ll be able to get the OS and VMWare running in one day or not. I saw this article and it’s for CENTOS 5.0 which I downloaded 5.1 and I am already seeing a possible differences in commands or software compatibility.
http://rootprompt.org/article.php3?article=10910
Dominic:
You should be able to get it installed in a day. Granted, I do this quite a bit–but I can usually have CentOS and VWS up and running in under 30 minutes–so a day should be realistic for a novice.
Removing the GNOME desktop does indeed strip CentOS of any GUI. But, you can always install it later vi the ‘yum’ package manager:
yum groupinstall “GNOME Desktop Environment”
There are a few things to install immediately over and above the “Bare Bones” install of CentOS. Vmware-server requires them:
* yum install gcc kernel-devel xinetd libXtst-devel libXrender-devel
Also, make sure to use the rpm version of the package from Vmware. Makes your life easier!
hmm… I’m not getting very far with regard to installation instructions for that RPM. Do you recommend that I FTP the software to the machine or save it to a CD and load it from there?
Any direction you can provide with getting solid command line instructions on installing the software would be great. I’d love to get this up and running and documented so I can do it over and over again.
One other thing I noticed is that like most Linux i’ve worked with I had problems finding a configuration tool or command like on SCO it was scoadmin or AIX I think it was smitty. Is there something like this for Linux normally?
Hi Brian,
I am at my ropes end with an issue I am having hopefully you can help me with sort out the fact from fiction, and guide me in the right direction.
I am running VMware Server 1.04 on a Dell Poweredge 2900 box with Win2K3 R2 Standard with 4Gb ram. I have Netware 6.5 with Groupwise7 running on one VM(1Gb mem assigned, SpamTitan on another VM(512Mb), and BES server running on another VM(512Mb).
My users are complaining of latency issues or hangs when opening emails in Outlook 2003. If they use the GW client, the emails open instaneously.
The bosses called in a consultant to look at the setup. He told them the setup of NW and GW is perfect. The only issue he has is with VMware.
He said this:
1)VMware server was never meant to be put in production. He recommends going to ESX.
2)There is not enough memory on the box, that we should add 4 to 8Gb more.
Can Vmware server be put in production? It replaced the GSX product and it WAS made to run in a production environment…
As for the memory, I would have to install the 64bit version of Windows to take advantage of the extra memory…
What are your thoughts or suggestions?
Any input is greatly appreciated. No one seems to be able to give me a straight answer. I spoke to 2 people at Vmware- one said yes, it could be run in production. Another said no, to go with ESX.
Thanks,
Rick
Hi Rick.
First, a bit of a disclaimer–I’ve never used Vmware Server on a Windows Host, only Linux ones. So YMMV.
My instant guess based on what you told me is disk latency.
But first of all, if you haven’t already–install the Vmware Server MUI on your machine, and take a look at it. You’ll probably notice that the VM’s that you’ve allocated a full 512 megs to aren’t actually using anything close to that. I very rarely see a VM actually /need/ 512M of RAM allocated. Adjust them accordingly.
Here’s the reason this is important: On Vmware (of any rendition), your single biggest performance hurdle will always be I/O wait on disk operations. But on Vmware Server, tuning your VM’s to allocate approximately the memory they’re actually using allows your Host OS use more of that RAM for disk cache, which is a good thing considering the size of the .vmdk files usually involved.
That would be my initial suggestion–see how your RAM is allocated, and free up what you can back to the Host OS. 1GB free isn’t a whole lot for both Windows and a disk cache.
Additionally, it would be helpful to know the setup on the 2900. If you haven’t already, mirror the drives and use a PERC that has built-in cache on it.
But as far as Vmware Server not being used in production, it’s hogwash. We’ve got 4 years of it under our belt at TLF–and *yet* have had it flake out. But sure, ESX is preferable of course, if you can actually afford it.
interesting article!
I’ve been messing around a lot with vmware server and centos 5.1 over the last few days and I have a problem… Maybe you have ideas about it.
configuration is
host: centos 5.1 64 bits on a dell 1950 (2 cpu with 2 core each)
guest: centos 5.1 64 bits smp (2 vcpu)
The bottom line is that I’m able to get the load average very high inside the guest, but the the load average on the host doesn’t go over 2… any idea why?
The real problem is that performance inside the vm are way lower than it would directly on the host… (like 10 times less work done) And this is not a memory issue.
thanks for any advice.
Great article, Brian. That’s the re-assurance I was looking for as I finish prepping a Dell PE2900 for VMware Server (with Ubuntu 6.06 as host OS, as opposed to CentOS) and a couple of slim virtual machines (primarily for ZCS, DNS, and Apache2). I was scouring blogs for some user-detailed examples of successful implementation and the TLF project sounded great! Much appreciated.
Hi Brian
Can you please point me to an article on how to configure mainframe on vmware ?
My host operating system is windows XP Professional.
Brian,
Excellent article. Thank you for sharing.
My experience: DOS 3.2 –> Win2k3, the last 10 years as a server admin at the corporate enterprise level. Beginner in Linux (very green)
My last two engagements were building large ESX farms attached to fiber and IP SANs. I now find myself doing some work for a small company and I have little budget. Thus I’ve built the following:
- Dell PowerEdge R900, 32 Gig RAM, 8x 146 Gig SAS (all RAID 10, single container) purchased through Dell Outlet for under $7K (2K cheaper than a white box and half the cost of a “new” R900)
- Ubuntu 8.04.1 64 Server (free)
- VMware server 1.0.6.9 (free)
Never heard of CentOS until this post.
Here’s my question: Within ESX I can trunk multiple ports and have my VMs run their own IPs (DHCP or static) accross this. Can I do this within VMware Server?
Please reply to my email address as I rarely visit these posting boards.
Your time and effort are greatly appreciated.
Thanks,
David
To previous poster, yes I think you can do any arbitrary configuration using VMware Server, but ESXi is now free, so I would use that if your hardware supports if for lots of reasons.
Whoever says that,”VMware Server is not fit for production” has surely never ever worked with well enough to understand what is going on and last but not least,has no iota of what virtualization/vmware is all about and the way a virtualization solution is put in place.
I have been experimenting,designing & implementing VMWare’s ESX,ESXi & Server editions for some major,small & startup companies with real good results.Putting a virtualization or vmware solution in place is a whole different ball game.Its all about careful & meticulous design keeping that various performance requirements,bottlenecks,purpose of the VMs,number of users,available hardware,bandwidth,etc,etc….Execution or deployment is just 15% and whereas design & system planning phase is 85%.That says it all.I know a case of one huge financial institution which screwed up its virtualization project because of hurrying up the design/system planning phase and directly jumping into implementing it.Ultimately it found itself in a huge mess with huge performance issues and blaming it all on VMware,saying it was not yet ready for mainstream.Later some sense dawned on this company and realized it was their incapability and not VMWare’s.Now,they are spending huge amounts on undoing everything,redoing the design and implementation all over again from square#1.
In short,what I am trying to say is either VMware Server or ESX/ESXi,they are all awesome technologies in their own sense.But,if you dont get the design& system planning basics right while putting a Vmware Server or ESX server ,no matter how big or great hardware & resources you have,they are all a huge waste and nothing more than a expensive 10k$ paperweight.
I have been personally running VMware Server 2.0 on CentOS 5.2(Final),patched and updated without any sort of issues,at my home.This has 6gb of RAM,2 x 640GB sata2 drives and a AMD Phenom Quad core CPU.In the Admin world,this machine would be compared to a midlevel gaming pc.But,you will be surprised to know that this is running 11 VMs at this point and all are up & running without any performance issues.6 are linux,2 Solaris 10,1 Vista,1 Windows server 2008 enterprise and all are running 64bit OS,since this physical cpu has AMD-V enabled.I am also running SAP Netweaver 2004s in one of the linux machines and Oracle 10g RAC simulation on 2 of the VMs,10gR2 on Solaris 10.
Coming to the way,I setup the filesystems or drives or CentOS installation.Like, Brian rightly pointed out,I am running a minimal install(although I have GNOME installed)but running it in run level 3 and disabled un-needed services and only 3-4 services are running.I have no onboard hardware raid as this is not a high end pc or entry level server.So,I am using software raid available in Centos and setup the RAID partitions according to my performance or redundancy needs.For example,VMs which can afford to go down even if one of the datastore filesystem(s) goes down,but need speed over redundancy,I chose to use RAID 0 with an EXT2 formatting.For VMs,which need redundancy over speed and cannot afford to be lost for a filesystem corruption or disk crash,I chose to go with RAID1 with EXT3.My SAP and Win 2008 server have their own dedicated datastores/filesystems and I chose speed over redundancy in both their cases as SAP is pretty IO intensive.So,I used RAID 0 with EXT2 with a bigger chunk size on the RAID level & bigger block sizes on the filesystem level.
Moreover,I make sure I allocate all the disk space upfront while I am creating a VM to give it a performance boost and at the same time avoid defragmentation,which indirectly results in inefficient IO seeks & burden on the kernel ultimately affecting end performance which will have a domino effect on all the VMs in that filesystem.Before,I create my first VM,I make sure the kernel & system parameters,swap ,shared mem params are tuned to requirements & to optimal values to reduce the possibilities of any bottlenecks in the first place.In short,better to be PROACTIVE rather than be REACTIVE & regret later.
This is just a tip of the iceberg,this small setup itself can be tuned,designed in multiple ways depending upon your needs and requirements.Its all about understanding the requirements clearly.That’s the first step for a successful vmware solution.And,trust me VMWare Server 1.x/2.x is a peach of a product,especially when you see its capabilities with the right design/hardware and absolutely at no licensing cost.It may be free of licensing costs,but that doesnt mean its inferior or cannot handle production.One shouldnt compare it with ESX/ESXi server,as that a whole different ball game when you really get into it.You will obviously getfor a 15grand a cutting/bleeding edge technology which can run on bare metal hardware with absolutely negligible overhead & footprint.Whereas,VMware server is running on top of a OS as an application.Now,the beauty is,inspite of that with careful but simple design ,one can churn out immense power,performance with it and can peacefully run handful of decent production VMs.So,compare apples to apples and not apples to oranges,which is not totally fair.
In the first place,the very reason vmware server is free with not licensing strings attached is,VMWARE wants people or companies(Big/Small) to have a taste of what it could achieve and the whole objective is to entice the companies to buy the REAL DEAL (ESX Server) once they tasted the power of vmware server.Any company would think,”If a free product like vmware server can give such a good server consolidation ratio & performance,then imagine what ESX could do which was designed from group up keeping performance,availability,resource scheduling & flexibility in mind”….VMware server is just a starting point…..Sky is the limit from there on….
Hope this info has helped you folks!!
Do feel free to email me@(fuseboxvirt@gmail.com) for any queries,suggestions,ideas,assistance in setting up,designing,tuning VMware Server Or ESX at your homes or organizations.Would be glad to assist you.
Thanks.Adios!!
[...] VMWare Server in Production by Brian Downey and [...]
[...] Algunas recomendaciones para usar el VMware Server 2 en una instalación mínima de CentOS: http://www.briandowney.net/blog/2007/03/30/vmware-server-in-production/ [...]