fb
API Management 9 minutes

Stress-tests with WSO2 API Manager 2.0.0

ThomasZielinski.jpg
Thomas Zielinski
Java Developer
API Publisher login
Scroll

The WSO2 products’ installations should end with tests which proof that the system is working stable and fast. The tests can show that some settings are not correct or that installation can be tuned. It also can show that there are some problems with the backend system (like access, internal errors, etc.).

Goal

How to test a published REST APIs or web service? Usually we request the endpoints and check the results. The problem is how many time we have to request the endpoint to be sure that everything is checked? How to measure the time of responses? How many resources (people, computers) we should use for it?

In this blog will be presented how can we easy test endpoints. As an environment we will use REST API published in WSO2 API Manager 2.0 and Gatling as test stress tool.

  • WSO2 API Manager is a complete solution for designing and publishing APIs, creating and managing a developer community, and for securing and routing API traffic in a scalable way.
  • Gatling is a highly capable load testing tool. It’s designed for ease of use, maintainability and high performance. It comes with support of the HTTP protocol that makes it a tool for load testing any HTTP server.

Scenario

The Yenlo Car API is designed to get an information about the cars registered in Netherlands. We can get information about car for given vehicle license plate or about the cars for given mark or for given mark and type.

The Yenlo Car API we would like to be managed. It means that we want to have an API’s life cycle management, validate API payload or access control and apply security policies to it. For this purpose we will publish our API via WSO2 API Manager 2.0.0.

Requirements

  • WSO2 API Manager 2.0
  • Yenlo Cars API – can be build and run on your environment (local computer) or used by Heroku application servers.
    • Local
      • download project https://bitbucket.org/tszielinski/yenlo-cars
      • upload schema and data to MySQL
        mysql> source yenlo-cars-import/sql/schema.sql;

        mysql> source yenlo-cars-import/sql/cars-backup.sql;

      • build project (Java8 required) under yenlo-cars directory
        yenlo-cars$ mvn clean install
      • run server
        yenlo-cars$ java -jar yenlo-cars-api/target/dependency/webapp-runner.jar --path /cars --port 8080 yenlo-cars-api/target/*.war

        --port 8080 yenlo-cars-api/target/*.war

  • WSO2 API Manager Publisher
  • WSO2 API Manager Store
    • Sign-in to StoreSign in to your accounr - API Manager - Stress test.png
    • Subscribe to cars-api and generate the tokens.generate tokens cars-api.pngNo keys found - stress test apim.png

Tests

The tests will check the endpoint published in WSO2 API Manager Store. The test report gives us the measure of the times between requests and responses. The same tests can be run on backend system and we can compare the results.

Test scenario

The Gatling tool gives us a possibility to create test scenario which defines the list of requests to be called and virtual users involved in it.

Let’s define our scenario. We would like to test the requests on listed URL’s:

  • GET /count
  • GET /count/marks
  • GET /count/mark/{mark}
  • GET /vrp/{vehicleRegistrationPlate}
  • PUT /vrp/{vehicleRegistrationPlate}/notes
  • PUT /vrp/{vehicleRegistrationPlate}/notes/clear

We want to simulate 50 users which request the URLs in nearly same time. 25 users start the test at one time and next 25 users start test during 30 seconds.

Source code

Gatling tool is written in Scala and tests have to be written in this language. The classes can be run under Scala environment or the Maven project runs in Java environment using gatling-maven-plugin.

$ mvn gatling:execute

Test classes

The source code with the test is available on Bitbucket.

BaseSimulation.scala

package com.yenlo.cars;

import io.gatling.core.Predef._
import io.gatling.http.Predef._

abstract class BaseSimulation extends Simulation {
  // car’s mark
  val marks = Array("A.C.","ALFA ROMEO","ALKO","ALPINE-RENAULT","ALVIS","AMERICAN MOTORS","AUDI",
    "AUDI/PORSCHE","AUSTIN","AUSTIN-HEALEY","AUTO BIANCHI","AUTO UNION","BAVARIA","BAVARIA CAMP",
    "BECK","BENTLEY","BERTONE","BUERSTNER","BUICK OPEL","CHALLENGER","CHRYSLER","CLASSIC ROADSTERS",
    "CMS AUTO","COXX","D.K.W.","DACIA","DAIMLER BENZ","DATSUN","DERCKS","DETHLEFFS","DIGICROSS",
    "ESTEPE","EURA MOBIL","EUROPELIFT","EVOBUS","FIAT","FIAT CAM","FIAT-ABARTH","FIAT-FLEURETTE",
    "FIAT-NECKAR","FIAT/CHALLENGER","FORD","FRANKIA","FRANKIA-RMB","FSM","GALLOPER","HENDRICKS",
    "HONDA","HYUNDAI","IFAKRAFTFARZEUGWERK AUDI","ISUZU","IVECO","IVECO FIAT","KIA","KIA MOTORS",    "KIAMASTER","LATERAL DESIGN LIMITED","LECAPITAINE","MALLAGHAN ENGINEERING LTD","MARUTI",
    "MATTIG BAMBERG","MEIWA","MERCEDES","MERCEDES-AMG","MERCEDES-BENZ","MERCEDES-BENZ/SCHENK",
    "MERCEDES-HYMER","MERCEDES-RAPIDO","MITSUBISHI","MITSUBISHI FUSO","MORGAN","NEBIM","NECKAR",
    "NILSSON","NSU-FIAT","OPEL","OPEL MOD DRST","POESSL","POLSKI FIAT","PORSCHE","PORSCHE-DIESEL",
    "QINGQUI","QUATTRO","RAMBLER RENAULT","RELIANT","RENAULT","RENAULT ALPINE","RENAULT MOD DRST",
    "RENAULT MOD JPM","RENAULT RODEO","RENAULT SAMSUNG","RENAULT SPORT","RENAULT TRUCKS","ROVER",
    "SANTANA","SAVIEM-RENAULT","SEAT","SIMCA-FIAT","SINGER","SKODA","SOVAB","STEYR-FIAT",
    "STEYR-PUCH","STUDEBAKER","STX","SUZUKI","TEILHOL","TEMPO","TERBERG TECHNIEK","TESLA",
    "TESLA MOTORS","TOMOS","TOYOTA","TOYOTA-MINICRUISER","TSCHU-TSCHU","TVR","VAN HOOL-FIAT",
    "VOLKSWAGEN","VOLKSWAGEN-TABBERT","VOLKSWAGEN/DE VRIES","VOLVO","VOLVO-VERHEUL","VOLVO-ZABO",
    "VOLVO/SCHENK","VW-PORSCHE","WESTFIELD","WINNEBAGO","YAMAHA","YUGO","ZIE BIJZONDERHEDEN")
  // vehicle licence plate
val vrps = Array("00BBP2","00BBP9","00BBR1","ZB02VY","ZB02YX","ZB03BL","AE4781","AE4782","AE4794","1011EB","1016ZN","1017GB","1022HG","3023US","3027XB","3029SB","3030BE","3033BB","3035BB","3037TF","3037UB","3039GX","3043MR","60AJ88","60AZ51","60BA11","60BB42","60BBB1","60BBB3","60BBB7","60BBBB","60BBBD","60BBBJ","JB001K","JB001L","JB001P","JB001R","JB001V","JB001X","JB002B");

val rand = new java.util.Random(System.currentTimeMillis()); 

// URL endpoint
val count = http("count").get("/count");
val count_marks = http("count-marks").get("/count/marks");
val count_mark = http("count-mark").get("/count/mark/" + marks(rand.nextInt(marks.length)));
val vrp = http("vrp").get("/vrp/" + vrps(rand.nextInt(vrps.length)));
val vrp_notes_set = http("vrp-notes-set").
      put("/vrp/" + vrps(rand.nextInt(vrps.length)) + "/notes").
      body(StringBody("""{ "note": "Note about a car."}""")).asJSON;
  val vrp_notes_clear = http("vrp-notes-clear").
      put("/vrp/" + vrps(rand.nextInt(vrps.length)) + "/notes/clear");
}

APIManager.scala

package com.yenlo.cars; 


import io.gatling.core.Predef._
import io.gatling.http.Predef._

class TestAPIManager extends BaseSimulation {
  // headers assigned to request
  val headers = Map(
    "Content-Type"->"application/json",
    "Accept"->"application/json",
    // change to valid token
    "Authorization"->"Bearer 94d7de858eba4b8f3f564153f6a4f103");
  // scenario
  val scn = scenario("cars").
    exec(count.headers(headers)).pause(0,3).       // request endpoint /count and pause 0..3 seconds
    exec(count_marks.headers(headers)).pause(0,5). // request endpoint /count/marks and pause 0..5 seconds
    exec(count_mark.headers(headers)).pause(0,5).  // request endpoint /mark/{mark} and pause 0..5 seconds
    exec(vrp.headers(headers)).                    // request endpoint /vrp/{vehicleLicencePlate}
    exec(vrp_notes_set.headers(headers)).          // request endpoint /vrp/{vehicleLicencePlate}/notes
    exec(vrp_notes_clear.headers(headers));        // request endpoint /vrp/{vehicleLicencePlate}/notes/clear

  // URL
  val httpConf = http.baseURL("http://localhost:8280/cars/0.0.1");
  setUp(
    scn.inject(
      atOnceUsers(25),              // create 25 users at
once
      rampUsers(25) over (30))      // create 25 during
30 seconds
  ).protocols(httpConf);
}

API.scala

package com.yenlo.cars; 

import io.gatling.core.Predef._
import io.gatling.http.Predef._
abstract class API extends BaseSimulation {
  val headers = Map(
    "Content-Type"->"application/json",
    "Accept"->"application/json");
 

  val scn = scenario("cars").
    exec(count.basicAuth("yenlo-cars", "yenlo-cars-demo").headers(headers)).pause(0,3).
    exec(count_marks.basicAuth("yenlo-cars", "yenlo-cars-demo").headers(headers)).pause(0,5).    exec(count_mark.basicAuth("yenlo-cars", "yenlo-cars-demo").headers(headers)).pause(0,5).
    exec(vrp.basicAuth("yenlo-cars", "yenlo-cars-demo").headers(headers)).
    exec(vrp_notes_set.basicAuth("yenlo-cars", "yenlo-cars-demo").headers(headers)).
    exec(vrp_notes_clear.basicAuth("yenlo-cars", "yenlo-cars-demo").headers(headers));
}

LocalAPI.scala

package com.yenlo.cars;

import io.gatling.core.Predef._
import io.gatling.http.Predef._

class LocalAPI extends API {
val httpConf = http.baseURL(“http://localhost:8080/cars”);

setUp(
scn.inject(
atOnceUsers(25),
rampUsers(25) over (30))
).protocols(httpConf);
}

HerokuAPI.scala

package com.yenlo.cars;

import io.gatling.core.Predef._
import io.gatling.http.Predef._

class HerokuAPI extends API {
  val httpConf = http.baseURL("https://yenlo-cars.herokuapp.com"); 

  setUp(
    scn.inject(
      atOnceUsers(25),
      rampUsers(25) over (30))
  ).protocols(httpConf);
}

Test run

# run test on WSO2 API Manager calling Yenlo Cars API
yenlo-stress-test> mvn gatling:execute -Dgatling.simulationClass=com.yenlo.cars.APIManager
# run test on install Yenlo Cars API installed on local machine to compare with WSO2 API Manager
yenlo-stress-test> mvn gatling:execute -Dgatling.simulationClass=com.yenlo.cars.LocalAPI
# run test on install Yenlo Cars API installed on Heroku to compare with WSO2 API Manager
yenlo-stress-test> mvn gatling:execute -Dgatling.simulationClass=com.yenlo.cars.HerokuAPI

Test results

The tests show how response times are distributed among standard ranges. Also informs about the number of OK (success)/KO (failed) requests.

The tests’ results are presented as short information on the display (console)

Simulation com.yenlo.cars.APIManager completed in 56 seconds
Parsing log file(s)...
Parsing log file(s) done
Generating reports...

=============================================================================
---- Global Information --------------------------------------------------------
> request count                                   300   (OK=300    KO=0     )
> min response time                                 3   (OK=3      KO=-     )
> max response time                              31441  (OK=31441  KO=-     )
> mean response time                              5854  (OK=5854   KO=-     )
> std deviation                                   7778  (OK=7778   KO=-     )
> response time 50th percentile                   1100  (OK=1100   KO=-     )
> response time 75th percentile                  11329  (OK=11329  KO=-     )
> response time 95th percentile                  21986  (OK=21986  KO=-     )
> response time 99th percentile                  28585  (OK=28585  KO=-     )
> mean requests/sec                               5.263 (OK=5.263  KO=-     )
---- Response Time Distribution ---------------------------------------------
> t < 800 ms                                       147 ( 49%)
> 800 ms < t < 1200 ms                               4 (  1%)
> t > 1200 ms                                      149 ( 50%)
> failed                                             0 (  0%)
=============================================================================

Reports generated in 0s.
Please open the following file: /Users/thomas/private/projects/gatling/yenlo-stress-test/target/gatling/apimanager-1474029323378/index.html

Simulation com.yenlo.cars.LocalAPI completed in 48 seconds
Parsing log file(s)...
Parsing log file(s) done
Generating reports... 

=============================================================================
---- Global Information -----------------------------------------------------

> request count                                     300 (OK=300    KO=0     )
> min response time                                   3 (OK=3      KO=-     )
> max response time                               24511 (OK=24511  KO=-     )
> mean response time                               4567 (OK=4567   KO=-     )
> std deviation                                    5923 (OK=5923   KO=-     )
> response time 50th percentile                    1041 (OK=1041   KO=-     )
> response time 75th percentile                    7396 (OK=7396   KO=-     )
> response time 95th percentile                   16211 (OK=16211  KO=-     )
> response time 99th percentile                   23112 (OK=23112  KO=-     )
> mean requests/sec                               6.122 (OK=6.122  KO=-     )
---- Response Time Distribution ---------------------------------------------> t < 800 ms                                        149 ( 50%)
> 800 ms < t < 1200 ms                                4 (  1%)
> t > 1200 ms                                       147 ( 49%)
> failed                                              0 (  0%)
=============================================================================

Reports generated in 0s.
Please open the following file: /Users/thomas/private/projects/gatling/yenlo-stress-test/target/gatling/localapi-1474029770028/index.html
and as HTML file with details information and charts

Conclusion

The tests build using Gatling tool give us the possibility to check our WSO2 installation. We can simply test endpoints (http, https, jms) using many virtual users for it. We can proof that installation is stable and can by highly load without any problems.

The code is created with use of the data from: https://opendata.rdw.nl/Voertuigen/Open-Data-RDW-Gekentekende_voertuigen/m9d7-ebf2 

If you have any questions about this blogpost contact us via the comments section of this blog. View also our WSO2 Tutorialswebinars or white papers for more technical information. Need support? We do deliver WSO2 Product Support, WSO2 Development SupportWSO2 Operational Support and WSO2 Training Programs.