Earlier this week I came across some interesting ACM papers discussing high-frequency trading (HFT). There's some food for thought here for more, shall we say, "mundane" enterprise network optimization and big data applications.
If a typical enterprise network is a Mercedes or BMW, HFT network engineering is NASCAR. Highly optimized designs to shave milliseconds and microseconds from automated stock trading transactions. Every cycle counts, with computer scientists looking to simplify, speed up and optimize everything. Profits are made by reaping fractions of a cent across millions of transactions per day.
HFT started life in 2003 with a mundane regulatory change requiring stock exchanges to quote prices in decimals instead of fractions. Financial firms specializing in HFT started with off-the-shelf hardware that could execute trades in under 10 milliseconds.
By 2005, time per trade was down to one millisecond and lead firms moving into modifying and/or running real-time kernels on servers, followed by custom code on switches, and an influx of data mining talent. Improvements in trading models were fleeting, with model optimization good for anywhere from three to six months before new models had to be rolled out.
Today, traders locate HFT systems directly within or next to stock exchanges, paying princely sums of money. A cabinet of 23 servers can suck up 17 kilowatts of power and cost $14,000 per month. Fiber cable lengths are precisely measured so each individual rack and equipment cage has exactly the same length of fiber, so no company gains unfair advantage with a shorter cable run.
WAN links are engineered in two forms: high-volume and high-speed. Volume is run via point-to-point fiber at 10 Gig Ethernet while high-speed links wireless links that provide lower latency. Caching is used to keep data flowing swiftly to subscribers.
Over longer distances, fiber optic routes have been deployed to shave milliseconds from transactions. Spread Networks operates a fiber route between Chicago and New York of under 13 milliseconds round trip with the right gear, shaving three milliseconds off existing routes. At least two companies are offering microwave networks to cut round trip time between the two cities to 8.5 to 9 milliseconds.
The quest for speed between cities has extended to international routes as well. Hibernia Atlantic's $300 million Project Express is deploying new fiber between London and New York. The route, supporting 40 Gbp per channel and ultimately 100 Gbps, will enable round trip speeds of 59.6 milliseconds, down from a 64.8 milliseconds on existing routes. Other firms are raising money to pull fiber over the Arctic Circle to establish a high-speed route between Tokyo and London, cutting speeds from 230 milliseconds to a range between 155 to 168 milliseconds.
Other optimizations occur at trading exchanges, with trade-offs between optimization and security. Unscrupulous traders might want to gain unfair advantage by reading packets from other participants or introducing heavy loading in one part of the network in order to create a "free path" for their transactions.
But how do high-speed optimization tricks by the HFT world translate to lessons learned for enterprise applications? For Big Data applications housed within a large data center, applying HFT techniques across server racks and farms should result in faster run time. Optimized kernel stacks to squeeze more out of server I/O along with a hard look at switches and other data exchange mechanisms are also worthy of examination. Dedicated/optimized servers for some applications may also be applicable - so long as everything stays under one roof.
Needless to say, one of the major keys for server-intensive Big Data applications is a fiber network with lots of speed. Shortening fiber runs between/among servers sounds like a pain in the neck, but any large scale application dependent upon shuffling bits back and forth on the LAN will benefit.
Moving beyond the data center and HFT practices become much more expensive. Shaving milliseconds off of WAN links between data centers is likely not to be a cost-effective solution compared to existing fiber connectivity, especially given the very customized and limited network access points necessary to implement least-distance fiber and microwave. There may be some utility in tapping into optimized undersea fiber if it is cost-competitive with existing routes; I suspect there will be excess capacity above and beyond HFT demands for general use - just don't expect the same sort of SLAs as the traders.