Home > Operating Systems, Storage > Linux, EMC SANs, and TCP Delayed ACKs

Linux, EMC SANs, and TCP Delayed ACKs

One of  relatively well-known issues when using EMC (and some other vendors’) SANs over iSCSI is the SANs’ dislike for TCP delayed ACKs. The reasons for the dislike are best described in this VMware KB article. EMC also has several articles discussing delayed ACKs on Primus, but overall the issue is confusing. With this post, I’ll try to clear up the confusion.

(Since you can’t deep-link to EMC Powerlink pages, I’m just going to give article numbers.) Out of many articles returned by Powerlink search on “delayed ack”, we can consider emc245445 to be the starting point, since it discusses general best practices for improving Clariion iSCSI performance and provides references to the articles covering Windows and ESX hosts specifically. About Linux hosts, it has this strange “may also apply” statement with no further explanation or instructions. Looking at the Windows article (emc150702), we see very detailed instructions on tweaking TCP stack settings; the articles for ESX (emc191777 and emc273003) lead to the VMware article mentioned above, which tells us where to find the magic checkbox that disables delayed ACKs. But the best Powerlink can do about Linux is emc264171, which reads like an exam answer of a mediocre student, who remembers some stuff from lectures, but can’t put it all together into a cohesive response. So, what’s going on?

The issue is not trivial and was probably never researched by EMC deeply enough to produce a useful Primus article. In Linux, delayed ACKs are disabled by setting the TCP_NODELAY option on the socket. It’s done by the application that wants to use the socket and is not a simple system-wide setting like in Windows or ESX. Therefore, in Linux, there are many different places where this setting may be specified.

If the Linux server is using software iSCSI, it is the responsibility of the iSCSI initiator code to set this option. Fortunately, open-iscsi, which is used by most (all?) modern Linuxes, does the right thing: TCP_NODELAY is hardcoded in the function that handles TCP connections.

In the case of hardware iSCSI initiators (proper HBA or iSCSI offload provided by Broadcom NICs and the like), things get more complicated. Since these adapters implement their own TCP stacks, delayed ACKs need to be disabled through the driver. And, of course, every driver will have its own setting for that. For example, Broadcom’s bnx2i will take “options bnx2i en_tcp_dack=0” in modprobe.conf. For other iSCSI implementations, you will need to consult the documentation or contact the vendor (or just Google it).

Categories: Operating Systems, Storage
  1. Glen Kelley
    June 18, 2015 at 12:48

    Just found this article – thanks for the additional information about delayed ACK and Linux. I’ve modified the EMC article you referenced, emc264171, to link to this article. You are correct, I’m not a Linux expert and finding information about how Linux works with delayed ACK was not easy and I apologize for the muddle of information about this.

    Always welcome suggestions for improving our knowledgebase articles and this was very helpful.

    If you are an EMC user, you can visit our support forum at the link below. Your comments would be appreciated.

    glen Kelley EMC

  2. Beverly Tookey
    June 18, 2015 at 14:18

    question I have – does the NODELAY disable Nagles or delayedACK?

  3. Glen Kelley
    June 18, 2015 at 16:56

    Not being a Linux expert I think the NODELAY only disables the Nagles

  4. Beverly Tookey
    June 19, 2015 at 08:55

    That’s what I thought – we have a customer with the NODELAY set and the SAN still has the 40ms delay.

  1. No trackbacks yet.

Leave a Reply to Beverly Tookey Cancel reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: