iSCSI works fully stable on ProLiant Gen9 servers (Windows Server 2012R2/2016 tested).
Time flies, but Moore's law still a law. Processor speeds continue to increase. For example, there is no need to offload the clock function from our mobile phones to separate dedicated hardware because the impact to phone performance is very low - yet first Nokia mobile phones were without the clock at all, because that required separate clock HW.
Similar is happening with iSCSI. There are no longer much benefits to offload the protocols to dedicated HW on NIC, even at 10Gbps speeds. Most NIC manufacturers no longer support stateful HW offloads, because iSCSI can just be used on any quality network card with software only initiator (built into OS), utilizing stateless accelerations on NIC, but via standard TCP stack. Why increase complexity with extra technology/drivers on server if everything works well without it, by just buying a few more cores in server processor or faster processor, and also because even encryption used for iSCSI is now accelerated in the processors. No surprise, that Intel explained already in 2010 why they will not implement iSCSI HW offloads in their NICs.
The last remaining card from HP that supports full iSCSI hardware offload on Gen9 servers is Emulex based HPE FlexFabric 10Gb 2-port 556FLR-SFP+ Adapter.
Check the description in standard features for "Hardware iSCSI or FCoE with Full Offload": The 556FLR-SFP+ is the only network option for HPE Gen9 rack servers that supports hardware FCoE or iSCSI with full protocol offload.
Emulex has the proven high quality offload drivers and works stable with offloads (Windows OS work stable starting from 2018-R2 version, I not tested if VMWare works properly). So inserting 556FLR into Gen9 server is the last chance to save on server processor utilization by TCP and iSCSI HW offload technologies.
Therefore, if you own G7/G8/Gen9 HP server today, have 10Gbps Emulex adapter, and access disks via iSCSI, or just use long data transmissions at very high bitrates with just a few TCP sessions (this is not for short TCP sessions like serving www pages), you can turn offloads on in Windows: for TCP, check this post, for iSCSI offload - you select the offloaded/HW initiator in GUI of OS "iSCSI initiator" program.
As situation develops now, it seems general processor speed increase is winning over HW offload technologies, as massively tested OS TCP stacks work well with most NICs, but for TCP/iSCSI offload you need high quality/stable offload drivers from high quality NIC vendor - currently only Emulex delivers it for HP ProLiant's, via 556FLR supported NIC card, in FLR form factor for inserting into special networking slot on Gen9.
2014-01-10
iSCSI works properly in HP ProLiant G7 and G8 servers with Emulex adapters
Summary: I confirm iSCSI works properly with HP ProLiant servers starting from G7 series (tested BL465G7 blades), and G8 servers (tested blades and DL series, both OK) with Emulex based adapters. Also, iSCSI works stable with Windows Server versions starting from 2008R2. Previous ProLiant or Windows versions do not perform or are unstable.
HP changed Broadcom to Emulex with G7 blades, and no problems were detected with Emulex chips.
Here is the measurement with BL465G7 for those who followed the test schemas in previous posts.
with processors getting faster, SW vs HW performance differences are not so high anymore. Additional benefit is the lower processor utilisation while iSCSI HW offload is used. Offload now only matters for 10G speeds.
To say it short - G7 blade servers and iSCSI are OK. Problematic BL495G5 with Broadcom based 10G chips were partially improved by driver updates during these years but never fully fixed, still small block writes do not work properly. But its time to refurbish such an old hardware, as it is a waste of electricity to use them novadays.
With HP ProLiant G8 servers its a new level of choice. You can select from a lot of HP network adapters, based on Broadcom, Emulex, QLogic and other chips, read about chip and wattages at the end of quickspecs sheet. If no particular requirements just select the inexpensive model, for example 554 that will do HW iSCSI if needed without extra licensing. Higher number in the name does not always mean better adapter, just a different chip manufacturer.
HP with G8 series of ProLiant's has several formats for network adapters:
FLR - special slot on rackmount G8 servers,
FLB - special slot on blade G8 servers.
SFP+ - usual PCI-Express cards.
FLR and FLB meaning is as follows:
Flexible Lan on motherboard, and then last letter Rack or Blade. FRL - FlexLOM Rack. FLB - FlexLOM Blade. It is easy to distinguish by last letter is the particular model for Rackmount or Blade servers.
Its no more possible to test them all :) . I will share here only what I tested and using without problems:
The conlusion is that particular application will perform best on particular adapter. Performances, features, processor utilization and costs vary.
This "broken iSCSI" saga is now over, iSCSI is stable and safe to use in production with HP ProLiant G7 or G8 servers. I will delete this blog after some time so people do not waste time reading old information. More use would be to open new blog like about creating terabit bandwidth and petabytes sized video delivery platform on ProLiants, with all the strange things happening on SmartArrays, for example if two disks fail under RAID6, replacement disks are inserted in the middle of rebuilding to spare, etc. Lot of illogical happens, but cases are at HP support and seems are handled properly.
Thanks for all, including the HP people who managed to eliminate these problems from G7 and later ProLiant server generations.
HP changed Broadcom to Emulex with G7 blades, and no problems were detected with Emulex chips.
Here is the measurement with BL465G7 for those who followed the test schemas in previous posts.
with processors getting faster, SW vs HW performance differences are not so high anymore. Additional benefit is the lower processor utilisation while iSCSI HW offload is used. Offload now only matters for 10G speeds.
To say it short - G7 blade servers and iSCSI are OK. Problematic BL495G5 with Broadcom based 10G chips were partially improved by driver updates during these years but never fully fixed, still small block writes do not work properly. But its time to refurbish such an old hardware, as it is a waste of electricity to use them novadays.
With HP ProLiant G8 servers its a new level of choice. You can select from a lot of HP network adapters, based on Broadcom, Emulex, QLogic and other chips, read about chip and wattages at the end of quickspecs sheet. If no particular requirements just select the inexpensive model, for example 554 that will do HW iSCSI if needed without extra licensing. Higher number in the name does not always mean better adapter, just a different chip manufacturer.
HP with G8 series of ProLiant's has several formats for network adapters:
FLR - special slot on rackmount G8 servers,
FLB - special slot on blade G8 servers.
SFP+ - usual PCI-Express cards.
FLR and FLB meaning is as follows:
Flexible Lan on motherboard, and then last letter Rack or Blade. FRL - FlexLOM Rack. FLB - FlexLOM Blade. It is easy to distinguish by last letter is the particular model for Rackmount or Blade servers.
Its no more possible to test them all :) . I will share here only what I tested and using without problems:
- Emulex 554FLR, 554FLB - work properly with HW iSCSI under Windows 2008R2, 2012 and 2012R2. Wire speed at 10GBbps for HW iSCSI. The systems under my responsibility currently standardised on Emulex for iSCSI applications. I do not know if QLogic and new Broadcom based adapters are working or not - not tested, and no reason to invest into testing as their price is the same.
- Intel 560FLR, 560FLB or 560SFP+ - these Intel network adapters do not have iSCSI HW offload but they have highest possible UDP packet send-out performance with lowest processor utilisation. I am able to reach under RedHat 6.4 full wire speed (10G) with UDP packets and no processor core exceeding 60%. 60Gbps can be served out of server if you have good VOD application. With Emulex 554, for example, with UDP only traffic the processor core that serves adapter reach 100% at about 7Gbps and it is impossible to reach wire speed. This was tested only for outgoing UDP traffic so if you have Video on demand servers serving lot of outgoing UDP packets with video, 560 is better choice than 554.
The conlusion is that particular application will perform best on particular adapter. Performances, features, processor utilization and costs vary.
This "broken iSCSI" saga is now over, iSCSI is stable and safe to use in production with HP ProLiant G7 or G8 servers. I will delete this blog after some time so people do not waste time reading old information. More use would be to open new blog like about creating terabit bandwidth and petabytes sized video delivery platform on ProLiants, with all the strange things happening on SmartArrays, for example if two disks fail under RAID6, replacement disks are inserted in the middle of rebuilding to spare, etc. Lot of illogical happens, but cases are at HP support and seems are handled properly.
Thanks for all, including the HP people who managed to eliminate these problems from G7 and later ProLiant server generations.
2010-02-21
TCP offload and iSCSI offload does not work under Windows 2008 R2 with Broadcom multifunction adapters
Executive summary for Windows 2008 R2 and Broadcom multifunction adapters:
1. TCP offload - problems. Not OK for production use.
2. iSCSI offoad - 10G - still not supported, 1G - problems. Not OK for production use.
Lot of time passed since the last blog. Still, no good news about Windows 2008 but today that is an OS of the past. After new Windows server version release, in-depth testing of TCP offload and iSCSI offload was done on Windows 2008 R2 with the latest drivers of the Broadcom multifunction chips. QLogic add-on iSCSI adapters were used for comparison. Processor performance impact was not measured - of course the more offloads the less loaded processor, but the goal of this test was to measure througputs and stability.
Equipment was the same as used for W2008 tests described in previous post.
•BL495 G5, tested:
–Broadcom (LOM) network interfaces (FLEX-10 VC)
(custom throughput rates 9gbps, 5gbps,2gbps and 1gbps)
(Separate tests with TCP offload enabled and iSCSI offload enabled)
–QLOGIC network interfaces (separate tests with 2 and 4 interfaces)
–QLOGIC network interfaces with ISCSI offload enabled (separate tests with 2 and 4 interfaces)
•BL465G5, tested:
–Broadcom (LOM) network interfaces (1Gbps, Virtual Connect 1/10)
–(separate tests with TCP offload enabled and iSCSI offload enabled)
•DL385 G5, tested:
–Broadcom LOM NIC
–QLOGIC network interfaces (separate tests with 2 and 4 interfaces), used as network adapters –QLOGIC network interfaces with iSCSI offload enabled (separate tests with 2 and 4 interfaces)
Test environment:
•Windows 2008 R2 Enterprise
•In all tests the same dual-controller MSA was used (MSA 2324i)
•QLogic QMH4062, two cards placed in BL495G5 (4 1Gbps ports total)
•QLogic QLE4062C used in DL385G5
• Latest drivers from hp.com used at the time of tests (5.0.32 for 10G, 5.0.16 for 1G)
•Duplicate HP Procurve 2910 switches are used to connect iSCSI target and initiator Network interfaces.
•Jumbo frames were NOT used in the test. We did several tests with jumbo frames enabled but no noticeable difference (packet/second possibility with one MSA connected is far below PPS that switches can achieve)
•During the tests, switches, MSA and servers were not used in any other environment and were not loaded with any other tasks except iSCSI Performance tests.
•10 LUNS (2 VDISKS, 24 X SAS-S, 73,4GB,15k RPM disks)
•IOMETER.org 2006.07.27, all tests 100% sequential (as the purpose was to test iSCSI performance). Processor load was low during all the tests, the purpose of this test was not to measure processor impact of iSCSI offload (that is a separate question)
•When testing with 4 adapters, one iSCSI session was opened from each server side port to each MSA2324i port (4x1Gbps total). When tested with 2 adapters, two iSCSI sessions were opened from each server side adapter to two separate physical ports of MSA2324i. N/O means NO Offloads. TCP offload means OS initiator + TCP offload ON. iSCSI offload means OS initiator was not used (except for configuration) and disks to OS presented by iSCSI card drivers.The lab test environment configuration is here (click for full sized picture):
Test No.1. The BL495G5 Flex-10 test results were as follows (click to see full slide):
The findings:
Improvements over W2008:
• TCP offload gives noticeable speed increases in Flex10 configuration (in cases where it works).
• No stability problems (iSCSI time-outs and dropped disks) with W2008 R2. With 2008, iSCSI from LOM is not stable, and iSCSI disks are lost from OS under heavy traffic, data is corrupted if under heavy writes. Customers must be adviced to use W2008 R2 or W2003, otherwise they will loose their data. Only Qlogic are stable under W2008.
Problems:
• iSCSI offload is still not available in Flex-10. In the FAQ it is stated that this is a Flex-10 limitation that is not yet addressed in firmware. This needs to be solved. Can be classified as the missing feature, but customers were waiting years for Broadcom iSCSI offload firmware/drivers to be fixed. NO RESULT YET, this severely hurts Flex-10 architecture as such, and hurts all iSCSI ecosystem in general.
• In all BL495G5+Flex10 tests write operation performance was hard-limited at 126MBps, regardless of I/O request size or NIC interface speed. This is a performance problem that needs to be solved.
• Problem: with TCP offload ON, write traffic FAILS with I/O request sizes larger than 124 (ok at 124, goes to nearly zero at IOMETER size 125 and above). This is not a problem of IOMETER. Microsoft Data Protection server 2007 for example never finishes backup jobs in this configuration as it probably uses blocks greater than 124.
Note: Flex10 traffic limiter configured at 1Gbps allows at least 1.2Gbps read traffic with low request sizes. Not a problem at all for production, just a note.
Test No.2. Broadcom LOM versus QLogic iSCSI performance (click for full size slide):
The findings:
Improvements over W2008:
• No stability (iSCSI time-outs and dropped disks) problems with Windows 2008 R2. With W2008+Broadcom, iSCSI is not stable, and iSCSI disks are lost from OS under heavy traffic, data is corrupted if under heavy writes.
Notes:
• Virtual connect 1/10 gives only a sligh decrease in performance in low I/O sizes, and very small increase in performance for big I/O sizes. = No impact to performance. Good.
Problems:
• 1Gbps Broadcom LOM read performance becomes unstable and worse with TCP offload ON, and substantially worse with iSCSI offload ON (performance decreases >2 times). Server cannot be a problem as Broadcom adapters behave exactly the same in BL465 or in DL385. Traffic becomes very unstable. Network utilization charts look like seismograph chart under heavy earth-quake. Traffic from Qlogic adapters is always even (very stable at all times).
• With TCP offload ON, write traffic FAILS with I/O request sizes larger than 124kB (still ok at 124, goes to nearly zero at IOMETER request size 125 and above). This is not a problem of IOMETER. Microsoft DP server 2007 for example never finishes backup jobs in this configuration as it probably uses blocks greater than 124. Same bug as with 10Gbps adapters under Flex-10.
Conclusions:
Executive summary: Broadcom multifunction adapter drivers and/or firmwares on ProLiant G5 servers are LOW QUALITY and DO NOT WORK PROPERLY. HP needs to either fix the firmware/drivers or replace Broadcom with different vendor for new HP servers.
Unsolved problems:
• Write performance problems with Flex-10. Traffic from 10Gbps port is limited at ~126Mbps with no TCP offload, and fails at I/O sizes over 124kB with TCP offload.
• TCP oflload cannot be used in production environment both for 10G and 1G Broadcom multifunction adapters. Write operations fail under Flex10 and Read operations fail under Blade/Virtual Connect 1/10 or DL385G5/ProCurve. Its Broadcom chip/driver problem.
• iSCSI offload cannot be used in production enviroment. Not possible on 10G where it would be most relevant due to Flex-10 limitations, and 3 times less read performance with both blade and non blade 1Gbps LOM.
Expectations to HP from the customer:
Priority No.1: fix the Flex-10 problems with new firmware/driver version. Due to write performance problems Flex-10 cannot be used for iSCSI (even with no any offloads).
Priority No.2: fix the write operation failures in TCP offload Broadcom driver if I/O size is larger than 124kB (both 10G and 1G multifunction adapters).
Priority No.3: fix the TCP offload and iSCSI offload READ problems. Until they are fixed, support for TCP offload and iSC SI offload must be withdrawn from all Quickspecs. ON is the current default in drivers.
Priority No.4: enable iSCSI offload in Flex-10.
In overal, iSCSI and TCP offloads on Proliant via Broadcom based LOM or PCI cards were never stable and were not fixed, for years. The only properly working offloaded iSCSI solution at 1Gbps is to purchase expensive QLogic cards, HP resells them for blades. Low (1Gbps) speed, high price - but stable. But as Broadcom's can, starting with Windows 2008 R2, be used as simple NIC with OS iSCSI initiator properly, no reason to invest into QLogic. No iSCSI offload solution at all for 10G. Seems HP will try to abandon this good disk access technology and try to rush to another new technology - FCoE, convergent adapters, and so on.
Additional unneded complexity being pushed out to the customers. Focus is shifting from existing and still not fixed - to new, more complex, and untested. Hard to believe such a big company like HP cannot properly update drivers for their multifunction network adapters. For years. Situation is better with W2008 R2, at least we can use 1Gbps LOM adapters with OS iSCSI initiator, but still on Flex-10 there is no properly working any settings combination, and TCP and iSCSI offloads do not work properly for all Broadcom adapters.
Are they really testing "thorougly", as it is said on the QuickSpecs, before selling?
We put a ban on purchases of HP blades, Flex-10, and "malfunction" Broadcom network adapters, until HP will fix the problems. 2 years waiting - and still low quality drivers, with problems.
1. TCP offload - problems. Not OK for production use.
2. iSCSI offoad - 10G - still not supported, 1G - problems. Not OK for production use.
Lot of time passed since the last blog. Still, no good news about Windows 2008 but today that is an OS of the past. After new Windows server version release, in-depth testing of TCP offload and iSCSI offload was done on Windows 2008 R2 with the latest drivers of the Broadcom multifunction chips. QLogic add-on iSCSI adapters were used for comparison. Processor performance impact was not measured - of course the more offloads the less loaded processor, but the goal of this test was to measure througputs and stability.
Equipment was the same as used for W2008 tests described in previous post.
•BL495 G5, tested:
–Broadcom (LOM) network interfaces (FLEX-10 VC)
(custom throughput rates 9gbps, 5gbps,2gbps and 1gbps)
(Separate tests with TCP offload enabled and iSCSI offload enabled)
–QLOGIC network interfaces (separate tests with 2 and 4 interfaces)
–QLOGIC network interfaces with ISCSI offload enabled (separate tests with 2 and 4 interfaces)
•BL465G5, tested:
–Broadcom (LOM) network interfaces (1Gbps, Virtual Connect 1/10)
–(separate tests with TCP offload enabled and iSCSI offload enabled)
•DL385 G5, tested:
–Broadcom LOM NIC
–QLOGIC network interfaces (separate tests with 2 and 4 interfaces), used as network adapters –QLOGIC network interfaces with iSCSI offload enabled (separate tests with 2 and 4 interfaces)
Test environment:
•Windows 2008 R2 Enterprise
•In all tests the same dual-controller MSA was used (MSA 2324i)
•QLogic QMH4062, two cards placed in BL495G5 (4 1Gbps ports total)
•QLogic QLE4062C used in DL385G5
• Latest drivers from hp.com used at the time of tests (5.0.32 for 10G, 5.0.16 for 1G)
•Duplicate HP Procurve 2910 switches are used to connect iSCSI target and initiator Network interfaces.
•Jumbo frames were NOT used in the test. We did several tests with jumbo frames enabled but no noticeable difference (packet/second possibility with one MSA connected is far below PPS that switches can achieve)
•During the tests, switches, MSA and servers were not used in any other environment and were not loaded with any other tasks except iSCSI Performance tests.
•10 LUNS (2 VDISKS, 24 X SAS-S, 73,4GB,15k RPM disks)
•IOMETER.org 2006.07.27, all tests 100% sequential (as the purpose was to test iSCSI performance). Processor load was low during all the tests, the purpose of this test was not to measure processor impact of iSCSI offload (that is a separate question)
•When testing with 4 adapters, one iSCSI session was opened from each server side port to each MSA2324i port (4x1Gbps total). When tested with 2 adapters, two iSCSI sessions were opened from each server side adapter to two separate physical ports of MSA2324i. N/O means NO Offloads. TCP offload means OS initiator + TCP offload ON. iSCSI offload means OS initiator was not used (except for configuration) and disks to OS presented by iSCSI card drivers.The lab test environment configuration is here (click for full sized picture):
Test No.1. The BL495G5 Flex-10 test results were as follows (click to see full slide):
The findings:
Improvements over W2008:
• TCP offload gives noticeable speed increases in Flex10 configuration (in cases where it works).
• No stability problems (iSCSI time-outs and dropped disks) with W2008 R2. With 2008, iSCSI from LOM is not stable, and iSCSI disks are lost from OS under heavy traffic, data is corrupted if under heavy writes. Customers must be adviced to use W2008 R2 or W2003, otherwise they will loose their data. Only Qlogic are stable under W2008.
Problems:
• iSCSI offload is still not available in Flex-10. In the FAQ it is stated that this is a Flex-10 limitation that is not yet addressed in firmware. This needs to be solved. Can be classified as the missing feature, but customers were waiting years for Broadcom iSCSI offload firmware/drivers to be fixed. NO RESULT YET, this severely hurts Flex-10 architecture as such, and hurts all iSCSI ecosystem in general.
• In all BL495G5+Flex10 tests write operation performance was hard-limited at 126MBps, regardless of I/O request size or NIC interface speed. This is a performance problem that needs to be solved.
• Problem: with TCP offload ON, write traffic FAILS with I/O request sizes larger than 124 (ok at 124, goes to nearly zero at IOMETER size 125 and above). This is not a problem of IOMETER. Microsoft Data Protection server 2007 for example never finishes backup jobs in this configuration as it probably uses blocks greater than 124.
Note: Flex10 traffic limiter configured at 1Gbps allows at least 1.2Gbps read traffic with low request sizes. Not a problem at all for production, just a note.
Test No.2. Broadcom LOM versus QLogic iSCSI performance (click for full size slide):
The findings:
Improvements over W2008:
• No stability (iSCSI time-outs and dropped disks) problems with Windows 2008 R2. With W2008+Broadcom, iSCSI is not stable, and iSCSI disks are lost from OS under heavy traffic, data is corrupted if under heavy writes.
Notes:
• Virtual connect 1/10 gives only a sligh decrease in performance in low I/O sizes, and very small increase in performance for big I/O sizes. = No impact to performance. Good.
Problems:
• 1Gbps Broadcom LOM read performance becomes unstable and worse with TCP offload ON, and substantially worse with iSCSI offload ON (performance decreases >2 times). Server cannot be a problem as Broadcom adapters behave exactly the same in BL465 or in DL385. Traffic becomes very unstable. Network utilization charts look like seismograph chart under heavy earth-quake. Traffic from Qlogic adapters is always even (very stable at all times).
• With TCP offload ON, write traffic FAILS with I/O request sizes larger than 124kB (still ok at 124, goes to nearly zero at IOMETER request size 125 and above). This is not a problem of IOMETER. Microsoft DP server 2007 for example never finishes backup jobs in this configuration as it probably uses blocks greater than 124. Same bug as with 10Gbps adapters under Flex-10.
Conclusions:
Executive summary: Broadcom multifunction adapter drivers and/or firmwares on ProLiant G5 servers are LOW QUALITY and DO NOT WORK PROPERLY. HP needs to either fix the firmware/drivers or replace Broadcom with different vendor for new HP servers.
Unsolved problems:
• Write performance problems with Flex-10. Traffic from 10Gbps port is limited at ~126Mbps with no TCP offload, and fails at I/O sizes over 124kB with TCP offload.
• TCP oflload cannot be used in production environment both for 10G and 1G Broadcom multifunction adapters. Write operations fail under Flex10 and Read operations fail under Blade/Virtual Connect 1/10 or DL385G5/ProCurve. Its Broadcom chip/driver problem.
• iSCSI offload cannot be used in production enviroment. Not possible on 10G where it would be most relevant due to Flex-10 limitations, and 3 times less read performance with both blade and non blade 1Gbps LOM.
Expectations to HP from the customer:
Priority No.1: fix the Flex-10 problems with new firmware/driver version. Due to write performance problems Flex-10 cannot be used for iSCSI (even with no any offloads).
Priority No.2: fix the write operation failures in TCP offload Broadcom driver if I/O size is larger than 124kB (both 10G and 1G multifunction adapters).
Priority No.3: fix the TCP offload and iSCSI offload READ problems. Until they are fixed, support for TCP offload and iSC SI offload must be withdrawn from all Quickspecs. ON is the current default in drivers.
Priority No.4: enable iSCSI offload in Flex-10.
In overal, iSCSI and TCP offloads on Proliant via Broadcom based LOM or PCI cards were never stable and were not fixed, for years. The only properly working offloaded iSCSI solution at 1Gbps is to purchase expensive QLogic cards, HP resells them for blades. Low (1Gbps) speed, high price - but stable. But as Broadcom's can, starting with Windows 2008 R2, be used as simple NIC with OS iSCSI initiator properly, no reason to invest into QLogic. No iSCSI offload solution at all for 10G. Seems HP will try to abandon this good disk access technology and try to rush to another new technology - FCoE, convergent adapters, and so on.
Additional unneded complexity being pushed out to the customers. Focus is shifting from existing and still not fixed - to new, more complex, and untested. Hard to believe such a big company like HP cannot properly update drivers for their multifunction network adapters. For years. Situation is better with W2008 R2, at least we can use 1Gbps LOM adapters with OS iSCSI initiator, but still on Flex-10 there is no properly working any settings combination, and TCP and iSCSI offloads do not work properly for all Broadcom adapters.
Are they really testing "thorougly", as it is said on the QuickSpecs, before selling?
We put a ban on purchases of HP blades, Flex-10, and "malfunction" Broadcom network adapters, until HP will fix the problems. 2 years waiting - and still low quality drivers, with problems.
2009-05-23
TCP offload and iSCSI offload does not work in Windows Server 2008 X64 with Broadcom chips
Updated: 2009.06.07
Executive summary: NOT STABLE. Not OK for production use.
Those who have one of Hewlett Packard Proliant servers probably noticed that in their quickspecs (an HP term used to describe technical specification of an HP product) it is stated, that the servers support iSCSI offload and TCP offload features.
iSCSI is (was?) very promising technology to connect disk arrays to servers over ordinary Ethernet/IP networks instead of using proprietary/expensive fibre-channel infrastructure.
This common HP and Broadcom page describes exactly what we need. Yet that never worked stable in practice. 1Gbps speed of Ethernet is still good speed to access disk, and also the number of applications increases where 10Gbps connection needed. The fibre-channel connections increased the speed gradually over time (1-2-4 and now 8Gbps) , but Ethernet historically progressed in the steps of 10, so its either 1Gbps or 10Gbps. As 10Gbps switches becoming mainstream (like HP ProCurve 6600-24XG 24 port 10Gig rack switch that uses only 350W of power (15W per 10Gbps port) and has back to front cooling), why to maintain fibre-channel if we can make all the networks IP based. Also, why to maintain the alternative FCoE technology, that tries to modify Ethernet to incorporate fibre-channel features (invest in more complex switches etc.). We even managed to fit Voice into IP networks, why not use the same IP for the disk connections.
As processing of iSCSI and TCP requires lot of computing power (the simple rule is that 1Gbps of traffic needs about 1GHz of modern processor), to reduce load on main processor network chips were created to handle this task (this was called hardware offload). iSCSI is a protocol to access disks over TCP protocol. So, good network adapters can offload TCP processing, and best ones can offload all the iSCSI processing so computer access the disks as the ordinary SCSI disks with no extra processing used. This is good even on non-heavily loaded processors as in modern servers we save the power if main processor is less loaded.
The worlds best selling servers - HP Proliant DL380/385 - have iSCSI support built-in according to their QuickSpecs, and now new servers will have it free-of-charge, according to this Press Release. But does iSCSI technology, including hardware offload, work properly under latest versions of Windows Server?
The purpose of this blog will be to monitor the situation with iSCSI and TCP offload on HP Proliant servers, that use built-in Broadcom chips for the iSCSI and TCP offload, on latest version of Windows Server x64 operating system (currently Windows 2008 x64 SP2). Both 1G and 10G situation will be monitored. I will erase this blog when it all works properly. Availability of TCP offload on older versions of Windows (like Windows 2003) has no any value to the customers as this feature is not properly supported by old version of Windows OS and was officially disabled by Microsoft via Windows Update.
The good information how to enable/disable TCP offload under Windows 2008 Server and how to check if it is working, can be found on this blog.
The test platform is the following one (all equipment from HP to avoid cross vendor blames):
As of 2009-June-05:
Latest driver pack tested is "HP ProLiant Networking Software v.9.65 (28-May-2009).
iSCSI status: NOT STABLE.
TCP offload status: NOT STABLE.
Particular driver versions (part of 9.65 release, latest ones):
NC532i: 4.8.101.2
NC373i, NC382T: 4.6.112.0 (B)
Ability to turn on hardware offloads in the drivers:
- 1Gbps Broadcom chips (5706, 5708, 5709 based, network adapter names are NC370, NC371, NC373, NC380, NC382, both blade and PCI versions) :
TCP offload - available
iSCSI offload - available
- 10Gbps Broadcom chips (57711 based, network adapter name is NC532, only build-in blade and mezzanine versions available):
TCP offload - available
iSCSI offload - not available.
This chip looks very promising IF properly working drivers will be written. Unfortunatelly not yet possible to buy the same chip as 10Gbps PCI-e card with SFP+ connectors for use in ProLiant DL series (recently NC522 10Gbps blade adapter appeared as alternative to Broadcom, this one is based on NetXen chip, recently acquired by QLogic, but its a different architecture, and only supports TCP offload, no iSCSI).
The results of the stability tests under heavy load:
1. The test for hardware TCP offload: NOT STABLE.
The public FTP server stops seeing network cards after several days of heavy load with TCP offload enabled.
2. The test for hardware iSCSI offload: NOT STABLE.
Timeout errors occur under heavy load, OS looses the disks and the data on disks is corrupted. This is tested under 1G because iSCSI offload drivers for 10G are not yet available.
3. The test for iSCSI disks with built-in Windows 2008 x64 SP2 OS iSCSI initiator (software initiator): NOT STABLE.
Timeout errors occur under heavy load, OS looses the disks and the data is corrupted. Not stable both under 1G and 10G Broadcom adapters.
We suspect the reason for software initiator problems may be in Windows 2008 TCP/IP stack, which was rewritten for this version of Windows. The reason for hardware initiator problems may be either in Broadcom drivers/firmware (repackaged and supplied by HP) or in Windows TCP/IP stack. The reason that hardware iSCSI offload is not yet available for Broadcom 57711 is that Broadcom is late to release the drivers (as always - drivers for 1Gbps network adapters were released more than a year late). According to Broadcom site, the 10G Broadcom chip supports everyting, including TCP offload and iSCSI. Just no working drivers, for years.
HP is working on the problem for several months now, but no solution yet. This may be the reason why in the QuickSpecs of MSA2000i Windows 2008 is still not included, despite that HP in February released a separate note that Windows 2008 is supported.
We will monitor every new release of HP Proliant networking software with the hope that the problems are fixed. My proposal for HP is to put all efforts now to ensure that Windows 2008 R2 works both with all offloads in the Proliant servers and with MSA2000i (or other) arrays without problems under heavy load. With Windows 2008 it all failed.
Currently, if you want to use iSCSI technology to connect disk arrays like MSA2000i, the only option is to use 1Gbps speeds (2Gbps may be achieved with 2 ports, etc.) and pricey QLogic iSCSI offload adapters. For blades, HP sell them as QMH4062 iSCSI adapter (and you will need to pay for 2 extra Virtual Connects+for all the external connection ports), for Proliant DL servers there is no offer from HP and probably the only option is to buy this iSCSI card directly from QLogic. QLogic was the only option were no iSCSI problems appear in any application. But only 1Gbps speeds are available.
Unfortunatelly, Flex-10 infrastructure from HP cannot be utilized currently for iSCSI access to disks using Windows 2008 OS. The idea of this product is very good (unified 10Gbps network), but unfortunatelly no working drivers yet. To have a perfect Flex-10 based datacenter servers+disks solution, two pieces are missing - MSA2000i (next version) with 10Gbps SPF+ connectors , and stable iSCSI and TCP offload drivers for Broadcom 57711 chip. Offload is required at 10Gbps speeds.
Of course if the loads are very light the iSCSI will appear work and maybe the errors will not happen for simple file access. But I cannot imagine who would want to use the technology knowing that the data may be lost if the load to disks will inrease, no matter if it is Microsoft Exchange or Data Protection (backup) type of load.
By the way, the iSCSI solution with the above listed hardware configuration is stable under Open SuSE Linux 11, but no any offloads are supported under Linux with Broadcom chips. My investment was into HP BL495c+ HP Flex10+ HP MSA2000i + MS Windows2008 + MS DataProtectionServer backup solution (HP hardware, MS software), and I am unable to use it for 6 months for now due to unstable iSCSI connection to disks from Windows 2008.
In the comming weeks I will continue to add updated information to this article, as new versions of drivers will appear. But with the progress like it was until now, it must be said "in the comming years, I will continue to test new versions of drivers to find that they do not work". We need the drivers now, not after 3 years when servers will be decommisioned.
HP - please, fix the situation. You are the biggest integrator, low quality deliveries must be your biggest problem. Push the Broadcom and Microsoft properly. Test it in your labs, escalate this! Forget Windows 2008, it failed already. Do all the efforts to release stable solution with Windows7. Please contact me if any co-operation is needed.
Executive summary: NOT STABLE. Not OK for production use.
Those who have one of Hewlett Packard Proliant servers probably noticed that in their quickspecs (an HP term used to describe technical specification of an HP product) it is stated, that the servers support iSCSI offload and TCP offload features.
iSCSI is (was?) very promising technology to connect disk arrays to servers over ordinary Ethernet/IP networks instead of using proprietary/expensive fibre-channel infrastructure.
This common HP and Broadcom page describes exactly what we need. Yet that never worked stable in practice. 1Gbps speed of Ethernet is still good speed to access disk, and also the number of applications increases where 10Gbps connection needed. The fibre-channel connections increased the speed gradually over time (1-2-4 and now 8Gbps) , but Ethernet historically progressed in the steps of 10, so its either 1Gbps or 10Gbps. As 10Gbps switches becoming mainstream (like HP ProCurve 6600-24XG 24 port 10Gig rack switch that uses only 350W of power (15W per 10Gbps port) and has back to front cooling), why to maintain fibre-channel if we can make all the networks IP based. Also, why to maintain the alternative FCoE technology, that tries to modify Ethernet to incorporate fibre-channel features (invest in more complex switches etc.). We even managed to fit Voice into IP networks, why not use the same IP for the disk connections.
As processing of iSCSI and TCP requires lot of computing power (the simple rule is that 1Gbps of traffic needs about 1GHz of modern processor), to reduce load on main processor network chips were created to handle this task (this was called hardware offload). iSCSI is a protocol to access disks over TCP protocol. So, good network adapters can offload TCP processing, and best ones can offload all the iSCSI processing so computer access the disks as the ordinary SCSI disks with no extra processing used. This is good even on non-heavily loaded processors as in modern servers we save the power if main processor is less loaded.
The worlds best selling servers - HP Proliant DL380/385 - have iSCSI support built-in according to their QuickSpecs, and now new servers will have it free-of-charge, according to this Press Release. But does iSCSI technology, including hardware offload, work properly under latest versions of Windows Server?
The purpose of this blog will be to monitor the situation with iSCSI and TCP offload on HP Proliant servers, that use built-in Broadcom chips for the iSCSI and TCP offload, on latest version of Windows Server x64 operating system (currently Windows 2008 x64 SP2). Both 1G and 10G situation will be monitored. I will erase this blog when it all works properly. Availability of TCP offload on older versions of Windows (like Windows 2003) has no any value to the customers as this feature is not properly supported by old version of Windows OS and was officially disabled by Microsoft via Windows Update.
The good information how to enable/disable TCP offload under Windows 2008 Server and how to check if it is working, can be found on this blog.
The test platform is the following one (all equipment from HP to avoid cross vendor blames):
- Servers: DL385 G5p
- Blades based on C7000 enclosure: BL465c G5, BL495c G5
- iSCSI disk arrays: MSA2000i (also alternative iSCSI disk array Promise M500i was tested)
- Everything connected via ProCurve 5406 switch with 1G and 10G connections. Blade interconnects are Virtual Connect 1/10G for BL465c G5, and BL495c G5 interconnect is Flex-10 Virtual Connect.
- Operating system is Windows 2008 x64, SP2 (latest OS, latest SP).
- The test applications to generate the TCP and disk loads are FTP server (built-in OS), IOmeter (industry standart program to measure disk performance), and Microsoft Data Protection Server 2007 SP1.
- Latest HP network drivers from HP web site. Latest firmwares for servers/disk arrays.
As of 2009-June-05:
Latest driver pack tested is "HP ProLiant Networking Software v.9.65 (28-May-2009).
iSCSI status: NOT STABLE.
TCP offload status: NOT STABLE.
Particular driver versions (part of 9.65 release, latest ones):
NC532i: 4.8.101.2
NC373i, NC382T: 4.6.112.0 (B)
Ability to turn on hardware offloads in the drivers:
- 1Gbps Broadcom chips (5706, 5708, 5709 based, network adapter names are NC370, NC371, NC373, NC380, NC382, both blade and PCI versions) :
TCP offload - available
iSCSI offload - available
- 10Gbps Broadcom chips (57711 based, network adapter name is NC532, only build-in blade and mezzanine versions available):
TCP offload - available
iSCSI offload - not available.
This chip looks very promising IF properly working drivers will be written. Unfortunatelly not yet possible to buy the same chip as 10Gbps PCI-e card with SFP+ connectors for use in ProLiant DL series (recently NC522 10Gbps blade adapter appeared as alternative to Broadcom, this one is based on NetXen chip, recently acquired by QLogic, but its a different architecture, and only supports TCP offload, no iSCSI).
The results of the stability tests under heavy load:
1. The test for hardware TCP offload: NOT STABLE.
The public FTP server stops seeing network cards after several days of heavy load with TCP offload enabled.
2. The test for hardware iSCSI offload: NOT STABLE.
Timeout errors occur under heavy load, OS looses the disks and the data on disks is corrupted. This is tested under 1G because iSCSI offload drivers for 10G are not yet available.
3. The test for iSCSI disks with built-in Windows 2008 x64 SP2 OS iSCSI initiator (software initiator): NOT STABLE.
Timeout errors occur under heavy load, OS looses the disks and the data is corrupted. Not stable both under 1G and 10G Broadcom adapters.
We suspect the reason for software initiator problems may be in Windows 2008 TCP/IP stack, which was rewritten for this version of Windows. The reason for hardware initiator problems may be either in Broadcom drivers/firmware (repackaged and supplied by HP) or in Windows TCP/IP stack. The reason that hardware iSCSI offload is not yet available for Broadcom 57711 is that Broadcom is late to release the drivers (as always - drivers for 1Gbps network adapters were released more than a year late). According to Broadcom site, the 10G Broadcom chip supports everyting, including TCP offload and iSCSI. Just no working drivers, for years.
HP is working on the problem for several months now, but no solution yet. This may be the reason why in the QuickSpecs of MSA2000i Windows 2008 is still not included, despite that HP in February released a separate note that Windows 2008 is supported.
We will monitor every new release of HP Proliant networking software with the hope that the problems are fixed. My proposal for HP is to put all efforts now to ensure that Windows 2008 R2 works both with all offloads in the Proliant servers and with MSA2000i (or other) arrays without problems under heavy load. With Windows 2008 it all failed.
Currently, if you want to use iSCSI technology to connect disk arrays like MSA2000i, the only option is to use 1Gbps speeds (2Gbps may be achieved with 2 ports, etc.) and pricey QLogic iSCSI offload adapters. For blades, HP sell them as QMH4062 iSCSI adapter (and you will need to pay for 2 extra Virtual Connects+for all the external connection ports), for Proliant DL servers there is no offer from HP and probably the only option is to buy this iSCSI card directly from QLogic. QLogic was the only option were no iSCSI problems appear in any application. But only 1Gbps speeds are available.
Unfortunatelly, Flex-10 infrastructure from HP cannot be utilized currently for iSCSI access to disks using Windows 2008 OS. The idea of this product is very good (unified 10Gbps network), but unfortunatelly no working drivers yet. To have a perfect Flex-10 based datacenter servers+disks solution, two pieces are missing - MSA2000i (next version) with 10Gbps SPF+ connectors , and stable iSCSI and TCP offload drivers for Broadcom 57711 chip. Offload is required at 10Gbps speeds.
Of course if the loads are very light the iSCSI will appear work and maybe the errors will not happen for simple file access. But I cannot imagine who would want to use the technology knowing that the data may be lost if the load to disks will inrease, no matter if it is Microsoft Exchange or Data Protection (backup) type of load.
By the way, the iSCSI solution with the above listed hardware configuration is stable under Open SuSE Linux 11, but no any offloads are supported under Linux with Broadcom chips. My investment was into HP BL495c+ HP Flex10+ HP MSA2000i + MS Windows2008 + MS DataProtectionServer backup solution (HP hardware, MS software), and I am unable to use it for 6 months for now due to unstable iSCSI connection to disks from Windows 2008.
In the comming weeks I will continue to add updated information to this article, as new versions of drivers will appear. But with the progress like it was until now, it must be said "in the comming years, I will continue to test new versions of drivers to find that they do not work". We need the drivers now, not after 3 years when servers will be decommisioned.
HP - please, fix the situation. You are the biggest integrator, low quality deliveries must be your biggest problem. Push the Broadcom and Microsoft properly. Test it in your labs, escalate this! Forget Windows 2008, it failed already. Do all the efforts to release stable solution with Windows7. Please contact me if any co-operation is needed.
Subscribe to:
Posts (Atom)