Executive summary: NOT STABLE. Not OK for production use.
Those who have one of Hewlett Packard Proliant servers probably noticed that in their quickspecs (an HP term used to describe technical specification of an HP product) it is stated, that the servers support iSCSI offload and TCP offload features.
iSCSI is (was?) very promising technology to connect disk arrays to servers over ordinary Ethernet/IP networks instead of using proprietary/expensive fibre-channel infrastructure.
This common HP and Broadcom page describes exactly what we need. Yet that never worked stable in practice. 1Gbps speed of Ethernet is still good speed to access disk, and also the number of applications increases where 10Gbps connection needed. The fibre-channel connections increased the speed gradually over time (1-2-4 and now 8Gbps) , but Ethernet historically progressed in the steps of 10, so its either 1Gbps or 10Gbps. As 10Gbps switches becoming mainstream (like HP ProCurve 6600-24XG 24 port 10Gig rack switch that uses only 350W of power (15W per 10Gbps port) and has back to front cooling), why to maintain fibre-channel if we can make all the networks IP based. Also, why to maintain the alternative FCoE technology, that tries to modify Ethernet to incorporate fibre-channel features (invest in more complex switches etc.). We even managed to fit Voice into IP networks, why not use the same IP for the disk connections.
As processing of iSCSI and TCP requires lot of computing power (the simple rule is that 1Gbps of traffic needs about 1GHz of modern processor), to reduce load on main processor network chips were created to handle this task (this was called hardware offload). iSCSI is a protocol to access disks over TCP protocol. So, good network adapters can offload TCP processing, and best ones can offload all the iSCSI processing so computer access the disks as the ordinary SCSI disks with no extra processing used. This is good even on non-heavily loaded processors as in modern servers we save the power if main processor is less loaded.
The worlds best selling servers - HP Proliant DL380/385 - have iSCSI support built-in according to their QuickSpecs, and now new servers will have it free-of-charge, according to this Press Release. But does iSCSI technology, including hardware offload, work properly under latest versions of Windows Server?
The purpose of this blog will be to monitor the situation with iSCSI and TCP offload on HP Proliant servers, that use built-in Broadcom chips for the iSCSI and TCP offload, on latest version of Windows Server x64 operating system (currently Windows 2008 x64 SP2). Both 1G and 10G situation will be monitored. I will erase this blog when it all works properly. Availability of TCP offload on older versions of Windows (like Windows 2003) has no any value to the customers as this feature is not properly supported by old version of Windows OS and was officially disabled by Microsoft via Windows Update.
The good information how to enable/disable TCP offload under Windows 2008 Server and how to check if it is working, can be found on this blog.
The test platform is the following one (all equipment from HP to avoid cross vendor blames):
- Servers: DL385 G5p
- Blades based on C7000 enclosure: BL465c G5, BL495c G5
- iSCSI disk arrays: MSA2000i (also alternative iSCSI disk array Promise M500i was tested)
- Everything connected via ProCurve 5406 switch with 1G and 10G connections. Blade interconnects are Virtual Connect 1/10G for BL465c G5, and BL495c G5 interconnect is Flex-10 Virtual Connect.
- Operating system is Windows 2008 x64, SP2 (latest OS, latest SP).
- The test applications to generate the TCP and disk loads are FTP server (built-in OS), IOmeter (industry standart program to measure disk performance), and Microsoft Data Protection Server 2007 SP1.
- Latest HP network drivers from HP web site. Latest firmwares for servers/disk arrays.
As of 2009-June-05:
Latest driver pack tested is "HP ProLiant Networking Software v.9.65 (28-May-2009).
iSCSI status: NOT STABLE.
TCP offload status: NOT STABLE.
Particular driver versions (part of 9.65 release, latest ones):
NC532i: 4.8.101.2
NC373i, NC382T: 4.6.112.0 (B)
Ability to turn on hardware offloads in the drivers:
- 1Gbps Broadcom chips (5706, 5708, 5709 based, network adapter names are NC370, NC371, NC373, NC380, NC382, both blade and PCI versions) :
TCP offload - available
iSCSI offload - available
- 10Gbps Broadcom chips (57711 based, network adapter name is NC532, only build-in blade and mezzanine versions available):
TCP offload - available
iSCSI offload - not available.
This chip looks very promising IF properly working drivers will be written. Unfortunatelly not yet possible to buy the same chip as 10Gbps PCI-e card with SFP+ connectors for use in ProLiant DL series (recently NC522 10Gbps blade adapter appeared as alternative to Broadcom, this one is based on NetXen chip, recently acquired by QLogic, but its a different architecture, and only supports TCP offload, no iSCSI).
The results of the stability tests under heavy load:
1. The test for hardware TCP offload: NOT STABLE.
The public FTP server stops seeing network cards after several days of heavy load with TCP offload enabled.
2. The test for hardware iSCSI offload: NOT STABLE.
Timeout errors occur under heavy load, OS looses the disks and the data on disks is corrupted. This is tested under 1G because iSCSI offload drivers for 10G are not yet available.
3. The test for iSCSI disks with built-in Windows 2008 x64 SP2 OS iSCSI initiator (software initiator): NOT STABLE.
Timeout errors occur under heavy load, OS looses the disks and the data is corrupted. Not stable both under 1G and 10G Broadcom adapters.
We suspect the reason for software initiator problems may be in Windows 2008 TCP/IP stack, which was rewritten for this version of Windows. The reason for hardware initiator problems may be either in Broadcom drivers/firmware (repackaged and supplied by HP) or in Windows TCP/IP stack. The reason that hardware iSCSI offload is not yet available for Broadcom 57711 is that Broadcom is late to release the drivers (as always - drivers for 1Gbps network adapters were released more than a year late). According to Broadcom site, the 10G Broadcom chip supports everyting, including TCP offload and iSCSI. Just no working drivers, for years.
HP is working on the problem for several months now, but no solution yet. This may be the reason why in the QuickSpecs of MSA2000i Windows 2008 is still not included, despite that HP in February released a separate note that Windows 2008 is supported.
We will monitor every new release of HP Proliant networking software with the hope that the problems are fixed. My proposal for HP is to put all efforts now to ensure that Windows 2008 R2 works both with all offloads in the Proliant servers and with MSA2000i (or other) arrays without problems under heavy load. With Windows 2008 it all failed.
Currently, if you want to use iSCSI technology to connect disk arrays like MSA2000i, the only option is to use 1Gbps speeds (2Gbps may be achieved with 2 ports, etc.) and pricey QLogic iSCSI offload adapters. For blades, HP sell them as QMH4062 iSCSI adapter (and you will need to pay for 2 extra Virtual Connects+for all the external connection ports), for Proliant DL servers there is no offer from HP and probably the only option is to buy this iSCSI card directly from QLogic. QLogic was the only option were no iSCSI problems appear in any application. But only 1Gbps speeds are available.
Unfortunatelly, Flex-10 infrastructure from HP cannot be utilized currently for iSCSI access to disks using Windows 2008 OS. The idea of this product is very good (unified 10Gbps network), but unfortunatelly no working drivers yet. To have a perfect Flex-10 based datacenter servers+disks solution, two pieces are missing - MSA2000i (next version) with 10Gbps SPF+ connectors , and stable iSCSI and TCP offload drivers for Broadcom 57711 chip. Offload is required at 10Gbps speeds.
Of course if the loads are very light the iSCSI will appear work and maybe the errors will not happen for simple file access. But I cannot imagine who would want to use the technology knowing that the data may be lost if the load to disks will inrease, no matter if it is Microsoft Exchange or Data Protection (backup) type of load.
By the way, the iSCSI solution with the above listed hardware configuration is stable under Open SuSE Linux 11, but no any offloads are supported under Linux with Broadcom chips. My investment was into HP BL495c+ HP Flex10+ HP MSA2000i + MS Windows2008 + MS DataProtectionServer backup solution (HP hardware, MS software), and I am unable to use it for 6 months for now due to unstable iSCSI connection to disks from Windows 2008.
In the comming weeks I will continue to add updated information to this article, as new versions of drivers will appear. But with the progress like it was until now, it must be said "in the comming years, I will continue to test new versions of drivers to find that they do not work". We need the drivers now, not after 3 years when servers will be decommisioned.
HP - please, fix the situation. You are the biggest integrator, low quality deliveries must be your biggest problem. Push the Broadcom and Microsoft properly. Test it in your labs, escalate this! Forget Windows 2008, it failed already. Do all the efforts to release stable solution with Windows7. Please contact me if any co-operation is needed.
