In the highly competitive Investment Banking market, low latency is absolutely critical as it is directly connected to profitability. Many organisations are making big investments to improve their latency performance, carrying out testing to ensure the latency achieved in a project environment is competitive in the market. However, low latency levels achieved in a project environment do not always continue into the production environment. This is a result of not having realistic latency test scenarios.
Latency Scenario Builder (LSB) is a methodology for efficiently building scenarios to test the latency of an application and address the three main challenges in latency testing:
- Building scenarios that cover various permutations and combinations of latency
- Analysing a latency issue, as it is generally particularly complex and difficult to identify the exact root cause
- Project environment differing from the realistic production environment
The LSB methodology
Firstly, a comprehensive list of latency scenarios for each use case should be determined. Each latency scenario is then paired up with a benchmark level that is agreed between business and IT stakeholders. These benchmark levels should represent the minimum expected results of the latency measurements after go-live, ensuring that latency testing is well managed and driven by clear business expectations.
For each latency scenario, the ‘source’ and ‘destination’ points are defined and within this, multiple measurement points, say x, y and z, are then identified. This identification is performed for each application. After this procedure, a matrix including the source, key sequence points and the destination points for all latency test scenarios is designed for measurement. This measurement then becomes part of the non-functional test design and will help to ensure detailed test coverage.
Test implementation for each latency scenario should be designed in such a way that it allows easy identification of bottlenecks when a latency issue is detected. For example, the overall latency is to be measured ultimately between points ‘A’ and ‘B’. As there are multiple points connecting ‘A’ and ‘B’ that are measured in a sequence, the tests expose the latency measurements between each of these.
By compartmentalising the approach, LSB provides an accurate account of the overall latency performance and, more importantly, clear identification of exactly where a latency ‘bottleneck’ occurs.
Another strength of LSB lies within the root cause analysis of the latency issues. As server and client machines’ internal configuration are included, such as CPU, Memory and Disk Input/Output usage and Garbage collection, the exact point where a bottleneck occurs is highlighted. This significantly contributes to improving efficiency in defect/issue resolution in latency testing.
A critical weakness of current latency testing models in the market is that latency testing is performed in a simplified version of the production environment. Therefore, the latency testing performed becomes meaningless and any improvements made during the project are not always realised in the production environment.
LSB replicates the production environment in a project environment so that when testing is performed, the capacity of the servers, client machines and other environmental variables match the production environment as closely as possible. If production-like configuration cannot be achieved due to budget issues, LSB analyses the difference and factors this impact into the latency test results. As a result, LSB will produce the most realistic latency measurement and the low latency achieved in the project environment will continue into production.
LSB is not limited to any particular project environment. It can be implemented into any application or asset class to measure any response time and test latency, and also to pin point the exact location of the latency issue.
- Thorough and detailed test coverage can be achieved.
- As latency directly translates to winning clients and business, managing business expectation is very important in this testing space. This methodology greatly contributes to managing business expectations.
- The matrix format in this methodology provides a very transparent and simple overview of test coverage to the business and IT.
- The matrix format helps to track the progress efficiently.
- The bottleneck identification and root cause analysis features of this methodology will efficiently identify the root-cause of the latency issues.
- This methodology helps to achieve the most realistic latency measurement and therefore avoids surprises when the application is released to production
- The methodology establishes a framework that is system agnostic and enhances the testing process regardless of business or location attributes.