Archive

Archive for December, 2011

Linux, EMC SANs, and TCP Delayed ACKs

December 21, 2011 4 comments

One of  relatively well-known issues when using EMC (and some other vendors’) SANs over iSCSI is the SANs’ dislike for TCP delayed ACKs. The reasons for the dislike are best described in this VMware KB article. EMC also has several articles discussing delayed ACKs on Primus, but overall the issue is confusing. With this post, I’ll try to clear up the confusion.

(Since you can’t deep-link to EMC Powerlink pages, I’m just going to give article numbers.) Out of many articles returned by Powerlink search on “delayed ack”, we can consider emc245445 to be the starting point, since it discusses general best practices for improving Clariion iSCSI performance and provides references to the articles covering Windows and ESX hosts specifically. About Linux hosts, it has this strange “may also apply” statement with no further explanation or instructions. Looking at the Windows article (emc150702), we see very detailed instructions on tweaking TCP stack settings; the articles for ESX (emc191777 and emc273003) lead to the VMware article mentioned above, which tells us where to find the magic checkbox that disables delayed ACKs. But the best Powerlink can do about Linux is emc264171, which reads like an exam answer of a mediocre student, who remembers some stuff from lectures, but can’t put it all together into a cohesive response. So, what’s going on?

The issue is not trivial and was probably never researched by EMC deeply enough to produce a useful Primus article. In Linux, delayed ACKs are disabled by setting the TCP_NODELAY option on the socket. It’s done by the application that wants to use the socket and is not a simple system-wide setting like in Windows or ESX. Therefore, in Linux, there are many different places where this setting may be specified.

If the Linux server is using software iSCSI, it is the responsibility of the iSCSI initiator code to set this option. Fortunately, open-iscsi, which is used by most (all?) modern Linuxes, does the right thing: TCP_NODELAY is hardcoded in the function that handles TCP connections.

In the case of hardware iSCSI initiators (proper HBA or iSCSI offload provided by Broadcom NICs and the like), things get more complicated. Since these adapters implement their own TCP stacks, delayed ACKs need to be disabled through the driver. And, of course, every driver will have its own setting for that. For example, Broadcom’s bnx2i will take “options bnx2i en_tcp_dack=0” in modprobe.conf. For other iSCSI implementations, you will need to consult the documentation or contact the vendor (or just Google it).

Categories: Operating Systems, Storage

VMware vCenter Server 5 Upgrade and Service Accounts

December 20, 2011 Leave a comment

When upgrading our vCenter Server instances, I encountered an annoying quirk in the vCenter Server installer, which may or may not depend on the way the database backend is configured. At least in our case, with the transition from a local SQL Express instance to a separate SQL database server, the installer didn’t allow to choose the account to be used to run vpxd and tomcat. Obviously, running the service under the personal account of the user who happened to be installing the service is a bad idea. So here’s a quick recipe for changing the service account:

  • Create the database per http://www.vmware.com/files/pdf/techpaper/vSphere-5-Upgrade-Best-Practices-Guide.pdf, pg.20. Run SQL Management Studio under the AD account that you will later use to install vCenter Server, make sure first to give it sufficient privileges in SQL Server to create the database (you can even make it a sysadmin – it’s temporary, so you can drop the privileges later). When creating the database, leave the owner default.
  • On the system where you’ll install vCenter Server, create a DSN (see same document), make sure to specify Windows Authentication.
  • Run vCenter Server installer.
  • After the installation is finished, install vCenter Client, confirm that everything works.
  • Stop VirtualCenter Server and Web Management services.
  • Go into these services’ properties and specify a special account you want to use for this purpose on the Log On tab.
  • Go to SQL Management Studio again. Create a regular user associated with the same AD account, then execute the following query on VCDB: “ALTER AUTHORIZATION ON DATABASE::VCDB TO <useraccount>”. This will change the database’s owner to this user.
  • Start the two vCenter services.

It seems there’s also a similar issue with VUM: its service gets installed with local SYSTEM account, but your default DSN configuration will most likely use Windows Authentication and therefore require a proper AD account. So, go to the service’s properties, change its account to the same one you used for vCenter, and restart the service. This issue is sufficiently widespread to have earned itself a KB article: http://kb.vmware.com/kb/1011858.

Update 10/19/12 (something that wasn’t a problem until upgrade to 5.1 where we had to start using vSphere Web Client): use the same account to run Inventory Service and Profile-Driven Storage Service.

Categories: Virtualization