Measuring Latency is the First Step to Reducing it

Posted on June 5th, 2008

With the demands of algorithmic trading, the drive to find and eliminate all possible points of latency has become something of an obsession. When you are hunting for milliseconds and microseconds, a stopwatch simply won’t do.

So when the Intel fasterLAB was setting up, a top priority – right after getting the hardware and networks installed – was to bring in the measurement experts to generate numbers that would be credible and useful.

Intel leveraged its relationship with Endace, which develops hardware that can drive latency measurement applications, and Trading Metrics, which has built software for the task.

“For measuring packet delay in digital networks, you need very accurate time stamping, and frankly, NIC (network interface cards) give awful timestamps,” says Steve Gleave, vice president at Endace.

Endace developed a solution to this problem with their data acquisition and generation (DAG) cards, that work on any interface (circuit, packet, IP and InfiniBand), at any speed, and for any packet size. Packets captured on a DAG card are immediately time-stamped, in hardware, using a clock that is synchronized to an external source with atomic accuracy. Time-stamps are 64-bit records, offering nanosecond granularity.

“The DAG card is fundamental to the entire Endace product portfolio. Once you guarantee to capture every packet, time-stamp it immediately, in hardware, and then write it directly into CPU RAM, applications are free to process the data as they please,” explains Gleave. “Data can be load-balanced, duplicated, filtered or statistically summarized. For latency measurement, Endace DAG cards can time¬stamp the same packet at different points within a trading architecture, thereby enabling partners like Trading Metrics to turn this data into trending information.”

With accurate views of latency, firms can assess where they will get the best ROI from technology investments in their infrastructure to improve the speed of operations. Nigel Woodward of Intel elaborates that “this is the fundamental raison d ’etre of the fasterLAB – to provide a sophisticated sandbox in which to simulate the trading environment and the increments to be gained from engineering at this level.”

Endace and Trading Metrics have worked together on measuring latency in the Reuters Market Data System (RMDS), and helped launch the Reuters Latency Monitor (RLM) service for measuring latency figures between a number of end points in the trading life cycle - from exchange to customer premise, in the on-site feed handler, the market data hubs, and the point-to-point servers. In addition to conducting diagnostics to uncover latency and help determine what, if any, action to take, the monitors provide alerts during the day if latencies grow beyond a defined range and allow important historical comparisons to be made.

“Our NinjaBox-LM was designed with Trading Metrics for the RLM service,” adds Gleave. “It’s a self-contained, passive monitoring appliance for accurate latency measurement. We aren’t being used to rip out every nanosecond but we are helping firms to know where the delays are in their trading cycle. In the case of RMDS sites we can measure all points from exchange to trading platform. Once delay issues are identified, diagnosis and remedy can be applied.”

Getting the facts moves discussions past a blame game between traders and data network operators. By pinpointing issues, the infrastructure teams can work significantly more effectively to meet the demands of the business.

“It is interesting to see how much money is thrown at latency before people measure it and look to see where it is and where it isn’t. We are al about helping people measure the delays very accurately. As official time-keepers of the low latency race, we ’re helping to pinpoint the difference between winning and losing.”

Trading Metrics

Banks will see real value in using the Intel fasterLAB because they can obtain reliable measures in an environment that simulates all the complexity, and potential latency of real production, says Marie Giangrande, director at Trading Metrics.

“Clients are using our software to meet needs in trading, operations and compliance. The most common call is one where the customer’s algo engine is missing the market. The second most common need is with customers who are obsessed with reducing latency end to end. Finally we also hear quite often that performance improvements in infrastructure upgrades did not meet the expectations the vendors have set.”

She recalls a client in London which tested new hardware and an upgraded network in a sterile environment and achieved a 50 to 70 percent performance boost. Back in the real world, however, the new investments led to a mere 10 percent improvement. Now the firm is using Trading Metrics to go through its production systems, benchmarking performance, and changing network or CPU configurations and then testing again to check for performance changes.

In this case the bank believes that it faces a combination of issues, Giangrande adds.

“Latency can be introduced at so many different points – by the application, by the throughput into a server and by external events like a huge spike in message traffic during a hot market. Looking at a single instance of performance is not sufficient, especialy in this marketplace with extremely variable traffic volumes combined with the capacity limitation of exchanges.”

The key to improvement is a holistic approach to the problem and precise measurements at each point in the process. Trading Metrics measures data flows across the enterprise to catch latency in networks, applications, and CPUs.

“Our correlations and comparisons help users diagnose the points causing latency. Over the years, we have found the biggest source of latency is applications that have been developed in-house. Code is old, has been grown organically and has not taken advantage of the tuning features available today from most vendors in the stack.”

Together, Endace and Trading Metrics enable Intel fasterLAB users to see definitive numbers in a large-scale environment that accurately simulates real-world production. Exacting comparism can be gained into the respective contributions of infastructure technology and software tuning to the latency arms race.