Archive
VMware vCenter Server 5 Upgrade and Service Accounts
When upgrading our vCenter Server instances, I encountered an annoying quirk in the vCenter Server installer, which may or may not depend on the way the database backend is configured. At least in our case, with the transition from a local SQL Express instance to a separate SQL database server, the installer didn’t allow to choose the account to be used to run vpxd and tomcat. Obviously, running the service under the personal account of the user who happened to be installing the service is a bad idea. So here’s a quick recipe for changing the service account:
- Create the database per http://www.vmware.com/files/pdf/techpaper/vSphere-5-Upgrade-Best-Practices-Guide.pdf, pg.20. Run SQL Management Studio under the AD account that you will later use to install vCenter Server, make sure first to give it sufficient privileges in SQL Server to create the database (you can even make it a sysadmin – it’s temporary, so you can drop the privileges later). When creating the database, leave the owner default.
- On the system where you’ll install vCenter Server, create a DSN (see same document), make sure to specify Windows Authentication.
- Run vCenter Server installer.
- After the installation is finished, install vCenter Client, confirm that everything works.
- Stop VirtualCenter Server and Web Management services.
- Go into these services’ properties and specify a special account you want to use for this purpose on the Log On tab.
- Go to SQL Management Studio again. Create a regular user associated with the same AD account, then execute the following query on VCDB: “ALTER AUTHORIZATION ON DATABASE::VCDB TO <useraccount>”. This will change the database’s owner to this user.
- Start the two vCenter services.
It seems there’s also a similar issue with VUM: its service gets installed with local SYSTEM account, but your default DSN configuration will most likely use Windows Authentication and therefore require a proper AD account. So, go to the service’s properties, change its account to the same one you used for vCenter, and restart the service. This issue is sufficiently widespread to have earned itself a KB article: http://kb.vmware.com/kb/1011858.
Update 10/19/12 (something that wasn’t a problem until upgrade to 5.1 where we had to start using vSphere Web Client): use the same account to run Inventory Service and Profile-Driven Storage Service.
Fixing VM-based pfSense CARP announcement echoes when using teamed network adapters
A few people who have tried to run pfSense as a virtual appliance in an ESX(i) host have found that CARP may refuse to work. Both pfSense nodes remain in “Backup” state – none of them is willing to take the Master role and start serving the VIP. This problem can be observed only if the VIP belongs to a virtual network interface that has multiple underlying physical adapters in a teamed configuration.
The best clue to the solution can be found in pfSense logs. There, one can see that the primary pfSense node actually tries to become a Master, but every time it receives a CARP announcement with the same advertisement frequency as its own, which makes it drop to Backup state (a router must see only lower frequency advertisements to remain Master). In fact, those advertisements are its own.
So, the real question is: why does the router VM receive IP packets that it has just sent? To answer it, we need to remember that, first, CARP advertisements are multicast, and, second, the typical ESX teaming setup uses the “Route based on the originating virtual switch port ID” option. This setting means that any given vNIC will consistently use the same pNIC, unless a hardware failure occurs. When this setting is used by the host, the physical switch, which the host’s pNICs are connected to, has all its corresponding ports configured by default, with no link aggregation.
Now, what happens when a CARP advertisement is sent? It exits the host on one pNIC, travels to the switch, where, being a multicast, it’s sent to other switch ports, including the other pNICs in the same team as the originating pNIC. The multicast comes back into the host, where it’s sent to all VMs on the same vSwitch, including the originating router. Oops.
We can argue whether or not this ESX behavior is correct, but the important fact is that VMware doesn’t seem to be interested in changing this behavior (the problem existed in 3.5 and still exists in 4.0). Instead, but there has been no VMware solution until 4.0U2. If you’re on this version, you can use the new Net.ReversePathFwdCheckPromisc option (refer to 4.0U2 release notes). Or we can fix it by ourselves, in a very simple way. We need to make the switch aware of the teamed nature of the pNICs involved. This way the switch won’t send the multicast packets back to the host.
I will let you figure out the correct setting for the switch (different vendors use different names for the same thing: Cisco has EtherChannel, Nortel calls it MultiLink Trunking, etc.). As for the host side, change the load balancing algorithm from the default to “Route based on IP hash”. Just keep in mind that until you have made the changes on both ends, that connection may not work, so make sure you’re not transferring anything important to/from the VMs on the same vSwitch while you’re making the changes. (I’m assuming your management network is on a different vSwitch; otherwise you’re on your own.)
Update: Thank you to Anne Jan Elsinga for pointing out that 4.0U2 provides a new option that can be used to solve the problem. I’ve modified the post to reflect this.
IT Infrastructure - The Infinite Source of Puzzles 