Wherefore art thou IP packet? Make haste Windows 2008 R2

How camest thou hither, tell me, and wherefore?
Lest the bard fray more, the topic is of PGM haste in the homogeneous environment, and the unfortunate absence of said haste. We take performance readings of PGM across multiple hosts and present a visual heat map of latency to provide insight to the actual performance. Testing entails transmission of a message onto a single LAN segment, the message is received by a listening application which immediately re-broadcasts the message, when the message is received back at the source the round-trip-time is calculated using a single high precision clock source.

Performance testing configuration with a sender maintaining a reference clock to calculate message round-trip-time (RTT).

The baseline reading is taken from Linux to Linux, the reference hardware is an IBM BladeCentre HS20 with Broadcom BCM5704S gigabit Ethernet adapters and the networking infrastructure is provided by a BNT fibre gigabit Ethernet switch.

Latency in microseconds from Linux to Linux at 10,000 packets-per-second one-way.

The numbers themselves are of minor consequence, for explanation at 10,000 packets-per-second (pps) there is a marked grouping at 200μs round-trip-time (RTT) latency.  The marketing version would be 20,000pps, as we consider 10,000pps being transmitted, and 10,000pps being received simultaneously, with a one-way latency of 100μs.  Also note that the packet reflection is implemented at the application layer much like any end-developer written software using the OpenPGM BSD socket API, compare this with alternative testing configurations that may operate at the network layer and bypass the effective full latency of the networking stack and yield to disingenuous figures.

20,000 feet (6,096 meters) Sir Hillary
Onward and upward we must go, with an IFG of 96ns the line capacity of a gigabit network is 81,274pps leading to the test potential limit of 40,000pps one-way with a little safety room above.

Latency in microseconds from Linux to Linux at 20,000 packets-per-second one-way.

At 20,000pps we start to see a spread of outliers but notice the grouping remains at 200μs.

Latency in microseconds from Linux to Linux at 30,000 packets-per-second one-way.

At 30,000pps outlier latency jumps to 1ms.

Latency in microseconds from Linux to Linux at 40,000 packets-per-second on-way.

At 40,000pps you are starting to see everything start to break down with the majority of packets from 200-600μs.  Above 40,000pps the network is saturated and packet loss starts to occur, packet loss initiating PGM reliability and consuming more bandwidth than is available for full speed operation.

Windows 2008 R2 performance

Here comes Windows.  Windows!

Latency in microseconds from Windows to Linux at 10,000 packets-per-second one-way.

Non-blocking sockets at 10,000pps show a grouping just as Linux at 200μs but the spread of outliers reaches as far as 2ms. This is highlights the artifacts of a low scheduling granularity and an inefficient IP stack.

Spot the difference

A common call to arms on Windows IP networking hoists the flag of Input/Output Completion Ports or IOCP as a more efficient design to event handling as it depends upon blocking sockets, reducing the number of system calls to send and receive packets, and permits zero-copy memory handling.

Latency in microseconds from Windows to Linux at 10,000 packets-per-second one-way using IOCP.

The only difference is a slightly lighter line at ~220μs, the spread of latencies is still rather broad.

Increasing the socket buffer sizes permits the test to run at 20,000pps but with heavy packet loss requiring the PGM reliability engine yielding 1-2 seconds average latency.

High performance Windows datagrams

All the popular test utilities use blocking sockets to yield remotely reasonable figures. These include netperf, iperf, ttcp, and ntttcp - Microsoft's port of ttcp for Windows sockets. These sometimes yield higher raw numbers than iperf on Linux which considering the previous results is unexpected.

• iperf on Linux yields ~70,000pps.
• PCATTCP on Windows yields ~90,000pps.
• ntttcp on Windows yields ~190,000pps.
• iperf on Windows yields ~20,000pps.

There appears to be either a significant driver flaw or severe Windows limitation in transmit interrupt coalescing as the resultant bandwidths from testing yield ~800mbs to a Windows host, but only ~400mbs from a Windows host. Drivers for the Broadcom Ethernet adapter have undergone many revisions from 2001 through to present, all consistently show weak transmit performance even with TCP transports.

Windows registry settings

To achieve these high performance Windows results the following changes were applied.

• Disable the multimedia network throttling scheduler. By default Windows limits streams to 10,000pps when a multimedia application is running.

Under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WindowsNT\CurrentVersion\Multimedia\SystemProfile set NetworkThrottlingIndex, type REG_DWORD32, value set to 0xffffffff.

• Two settings define an IP stack path for datagrams, by default datagrams above 1024 go through a slow locked double buffer, increase this to the network MTU size, i.e. 1500 bytes.

Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters:

• FastSendDatagramThreshold, type REG_DWORD32, value set to 1500. 

• FastCopyReceiveThreshold, type REG_DWORD32, value set to 1500.

• Without hardware acceleration for high resolution time stamping incoming packets will be tagged with the receipt time at expense of processing time.  Disable the time stamps on both sender and receiver. This is performed by means of the following command:

netsh int tcp set global timestamps=disabled

• A firewall will intercept all packets causing increased latency and processing time, disable the filtering and firewall services to ensure direct network access.  Disable the following services:

• Base Filtering Engine (BFE) 

• Windows Firewall (MpsSvc)

Additional notes
The test hardware nodes are single core Xeons and so Receive Side Scaling (RSS) does not assist performance.  Also, the Ethernet adapters do not support Direct Cache Access (DCA) also known as NetDMA 2.0 which should improve performance by reducing system time required to send or receive packets.

To significantly increase the default socket buffer size you can set a multiplier number under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters:
BufferMultiplier, type REG_DWORD32, value set to 0x400.

Strawberries and a Python through the Looking Glass

How do we get OpenPGM on Windows?  This is an obvious question popular with new developers.  The stumbling blocks tend to be that OpenPGM is written to the ANSI C99 specification and Microsoft's Visual C++ 2010 compiler only supports ANSI C89 or C++ 2003.

Convexing Cross-Compiling of C

Microsoft hasn't the only C compiler vendor targeting Windows, and it also isn't necessary that the compiler actually runs on Windows, by a process called cross-compiling the Windows platform can be targeted from another.  This also tends to be scary for many Windows developers as they need a second operating system, learn how to use a new compiler, and finally you have additional compiler dependencies when bringing back the library on Windows.

Priceless Princely Patch for Progress

As it so happens cross-compilation has its pros and cons, on the plus side it means one code base and one build system; on the negative side it means the resultant library brings in additional libraries for the cross-compiler and its compatibility layers, and debugging is made rather inconvenient.

So the benefits of a native build are strong but how much effort is required to modify C99 code to get it working in a C89 compiler and how difficult is a native build system?  Well the differences from one compiler to the next can be managed with a patch cluster.  The build system is similar to Automake and SConscript build components as the Autoconf comparables are constant to Windows XP SP3.  We choose a build system that can target agnostic to Visual Studio compiler version for greatest compatibility to minimize maintenance overhead.

CMake Make Make

With Microsoft Visual C++ 2010 installed we need to install some extra software to start, first the build system called CMake which has a simple Win32 installer with minimal options, it is recommended to allow it to update your system PATH so you can use immediately.  Next are two critical build dependencies for OpenPGM, we need both the Perl and Python 2 scripting languages both of which offer x86 and x64 builds depending on your platform architecture.

Extract the source for the latest OpenPGM archive into somewhere convenient, for example C:\>

CMake generating Makefile.

By default CMake is creating a Makefile with debugging enabled, you can use cmake-gui to update the parameter for a release build.

CMake-GUI updating configuration parameters and regenerating Makefile.

With a successfully generated Makefile we can start the build process using Microsoft's standard command line tools.

Generating C89-compatible source files.

The build process continues automatically by first patching all the source files for C89 compatibility,  the C89-compatible source files are fed into the compiler, finally the object files are linked together into the native library.

Generating object files and final libpgm library.

Now complete we can start to use the library to build the example applications and start development.  An additional feature of the CMake system is called CPack which allows us to create a Windows installer we can use to install OpenPGM binaries on other systems.

First download the latest NSIS package and install with full options, then simply run cmake package to produce the Windows installer.

Building an OpenPGM Windows installer with CPack and NSIS.

What is PGM? What is OpenPGM?

Well lets start by saying PGM is the name of a internet, lower-case "i", communications protocol, and OpenPGM is an implementation of that protocol.  A communications protocol being the definition of how two or more computers or electronic devices communicate with each other.  For example you just enjoyed a good slice of cherry pie and damn fine cup of coffee and you wanted to tweet it to your friends, you tap the message into your phone and hit a button, then by magic the message appears on your microblog.  Between your phone's Twitter app' and the Twitter website your tweet message has been sent following a predefined communications protocol.

There is a subtle difference between the Internet, upper-case "I", and an internet, lower-case "i", the former is the name for the public network where we can access Facebook and Twitter, the latter is the generic term for a group of interconnected networks.  This means we could find PGM on the Internet or we could find it on other non-public networks such as within a stock exchange's trading system or communicating physics of the Higgs Boson at the Large Hadron Collider.

So what is special about PGM?

The erhu is a Chinese two-stringed bowed musical instrument and not commonly found in Western music.  If you wanted to listen to a recital of Sanmen Capriccio a convenient choice might be to watch a performance by Yang Ying on YouTube.  For a live performance you might tune in the radio and listen alongside other fans of huqin.  On the Internet you may find an Internet radio station to listen on but there is fundamental limitation on how many fans can be plugging in.  Say Girls Generation announced a online concert tonight the stampede from eager fans may overload the site and crash the network as the concert site has to stream a duplicate copy of the event to each and every listener.  The PGM protocol allows one sent stream of data to be received by multiple recipients, the fan-out of the stream to each recipient being managed by the network instead of say the host concert site.

Why should I choose OpenPGM?

So PGM is the protocol on paper, digitally at least, you need an actual implementation for something to start looking like it will work.  There are a selection of vendors providing commercial solutions, including Microsoft, IBM, and TIBCO,  and a variety of open source projects as long as your preferred platform is FreeBSD.  So an open source implementation that works on multiple platforms, is compatible with existing vendor solutions, and is even faster and more flexible sounds like a good choice, then have a look at OpenPGM.