Month: May 2011

Qualifying a Loadtest as PASS/FAIL/USEFUL

While trying to archive old data on my laptop I had come across a document I had written  long ago about load test statuses and thought I should blog it here. The document details about the PASS/FAIL/USEFUL criteria to qualify a test.  In order to apply it we need to understand the criteria under which the test result is qualified as pass/fail/useful. This will help the test team to characteristically collate the test data point(s) for further analysis. Qualifying and quantifying will also help the business understand the current state of the application.

Why should a load test be passed or failed?

Before an application is rolled out to production, an extensive capacity and performance validation testing is performed in order to test if the AUT satisfies certain non-functional requirements.  The test results from these tests are further analyzed to define the performance ( P ), availability (A) and reliability ( R ) of the application and these three variables (P, A, R) could be incorrectly quantified if the test does not run satisfactorily. Before analyzing for performance , availability and reliability it is important to identify if the load test was successful. The table below details a FEW criteria to identify the same.

 

    Note: Please read the NOTE below in order to understand the table

    BudgetThe allocated response time SLA (service level agreement) for each KPI (Key Performance Indicators) and peak hour transactions rates for each KPI.

    Optimal response time – The favorable response time calculated based on the system and network variability (ex. +20% of the budgeted response time) .

    Optimal transaction rateThe favorable transaction rate calculated based on the budgeted requirement for each Key Performance Indicator.

    1. The Optimal response time requirement for the Key Performance Indicators is considered to have response times <= (Budget + 20% of Budget)

      2. The Optimal transaction rate requirement for each KPI is considered to have a transaction rate: (Peak hour load – 10% of Peak hour load) <= Trans Rate <= (Peak hour load + 10% of Peak hour load)

      3. The test is marked as PASS if all of the criteria for “PASS” are satisfied.

    4. The test should be marked as FAIL if any of the criteria 7 – 9 exists during the test duration

5. The test should be marked as FAIL if extensive logging is turned on in the environment. The extent to which logging is enabled affects the performance of the application. Typically logging should contribute an increase in response time to about 1% – 2%.

6. The test should be marked as FAIL if all of the criteria for “FAIL” are satisfied.

 

Table 1:


S.No

CRITERIA

PASS

FAIL

1

Transaction Rate (X)

(Peak hour load – 10% of Peak hour load) <= (X) <= (Peak hour load + 10% of Peak hour load)

(X) < (Peak hour load – 10% of Peak hour load)

2

Average Transaction Response Time (Y)

(Y) <= Optimal resp time

(Y) > Optimal resp time

3

Resource Utilization

<= 80%

> 80%

4

Memory Utilization

<= 75%

> 75%

5

Memory Leaks

None

Identified

6

Load Balance

Balanced or is shared according to the implemented algorithm

Unbalanced or does not share according to the implemented algorithm

7

Script(s)

Passed

Failed

8

Server(s) Crash

None

Exists

9

Server(s) Restart

None

Exists

10

Logging

Turned off or enabled with allowable limit

Extensively turned on

11

HTTP 500 errors

None

Encountered

12

HTTP 404 errors

None

Encountered

 

Why should a Load test be marked USEFUL?

Typically an application is released when it satisfies all the requirements, in other words the PASS criteria. But many times the application is also released when it does not satisfy all of the criteria. Does this necessarily mean that the AUT failed to meet the requirements and hence be FAILED on the whole? Not really. In such situations the business takes optimum risk by adding acceptable thresholds to the current quantifiable data. This is where we need to identify the criteria that prompts the business to release the application and hence mark the Load tests “USEFUL”.

 

Technically a test is passed or failed when the criteria in Table 1 are all satisfied. There are a few cases where the test results can neither be passed nor be failed and falls in the grey area. In an effort to improve the performance of the application many changes are made in the application environment, a FEW of which are:

  • Memory problems
  • Incorrect configurations
  • Overloaded CPU’s
  • Cache sizes
  • Network bandwidth
  • Database indexing
  • Database deadlocks
  • Test scenarios

We learn the best practices/configurations/changes suitable for performance improvement by tweaking the above variables. When the Load tests are run with new changes, every change made in the environment is a learning experience to further improve the performance of the application. In the given test cycle time successful load testing all of these may or may not be accomplished. Among the failed tests we need to identify the usefulness of the load test results to further improve the application. Due to the range of load tests we run during the Testing Life Cycle it is important to differentiate & identify the usefulness of these tests continuously and characterize them accordingly.