Message Transformation with WSO2 API Microgateway

Message transformation and mediation is a common requirement of any enterprise integration platform. The requirement intensifies when the integration platform grows and the number of different systems it needs to interact with increases.

In a typical integration platform API Management plays a key role in exposing transactional or streaming services/data as APIs whereas the enterprise integration layer handles interconnecting of different systems together via message transformation, protocol switching etc. Different organizations select different approaches when it comes to solving the problem of message transformation. Some might delegate the problem to integration or external layer, and some might prefer doing it on the API management layer itself. Both these approaches might have their merits and flaws depending on the use cases.

In this blog we will look at few different approaches on how to do message transformation with WSO2 API Microgateway and their pros and cons.

WSO2 API Microgateway is a lightweight, cloud native API gateway designed for microservices architecture. It has the inbuilt ability to do message transformation via interceptors. We will look at some of the ways interceptors can be used to achieve transformation within the API layer as well as how we can delegate transformation to external system or service.

Setting up echo backend service

First of all, lets deploy a sample backend service. In this blog I am using a simple echo Netty server, which responds back with the request payload. This service can be downloaded from here[1]. Once downloaded, you can run the jar using the following command. It will start the Netty echo service on port 8688

java -jar netty-http-echo-service-0.4.5.jar

Try out the sample service with following cURL command

curl -k "http://localhost:8688/echo" -X POST -d '{"foo":"val1"}' -i -v -H "Content-Type: application/json"

Setting up Microgateway Project

Next let’s create open API schema which defines the API that exposes the Netty service via microgateway. This API has a single POST resource with path /echo.

---
openapi: 3.0.0
servers:
- url: "http://localhost:8688"
info:
  description: 'This is a sampl echo service.'
  version: 1.0.0
  title: Echo_XML_Service
x-wso2-basePath: /transform
x-wso2-disable-security: true
paths:
  "/echo":
    post:
      summary: Echoes the reponse.
      operationId: echoMethod
      responses:
        '200':
          description: successful operation
          content:
            application/xml:{}
            application/json:{}
        '400':
          description: Invalid status value

Here the extension “x-wso2-disable-security” is used for the sake of simplicity as this property will allow us to invoke the API via microgateway without a secure token.

Next download the latest microgateway toolkit and runtime versions (3.2.X) from the location[2] and follow the instructions[3] to setup them.

Now we can initialize the Microgateway project using the following command using the toolkit. Here the project name used is “transform”

micro-gw init transform

Next copy the open API file to the “api_definitions” folder inside the project “transform”.

Then let’s build the project and create the runtime artifact.

micro-gw build transform

Finally, we can run the build runtime artifact using the microgateway runtime(binary) using the following command.

./gateway <PATH_TO THE_BUILD_JAR>

Now we have exposed the API without any transformation, and it can be invoked using below command via microgateway. This will return the same json payload we sent in the request by the Netty backend.

curl -k "https://localhost:9095/transform/echo" -X POST -d '{"foo":"val1"}' -i -v -H "Content-Type: application/json"

Now let’s assume that the backend Netty service expects the payload in xml format, and the client sends the request with json payloads. So, there should be transformation logic which convert json to xml before sending it to the backend. We will see how we can achieve this requirement in 3 different approaches using API Microgateway.

1. Transform within gateway using ballerina interceptors.

We can write the json to xml transformation within gateway using a ballerina interceptor. What happens here is we engage custom ballerina code during the request flow to do the transformation.

First let’s write the ballerina code that transforms the json to xml. The ballerina samples for the version 1.2.x which is the supported ballerina version in microgateway 3.2.x can be found here [4]. Let’s create a file with name json_to_xml.bal (any name can be used) and place the following content.

import ballerina/http;
import ballerina/xmlutils;
import ballerina/log;

public function jsonToXML (http:Caller outboundEp, http:Request req) {
    json payload = checkpanic req.getJsonPayload();
    xml|error x1 = xmlutils:fromJSON(payload);
    if (x1 is xml) {
        req.setXmlPayload(x1);
    } else {
        log:printError("Error while converting json to xml", x1);
    }
}

Here the function name can be any name but it should accept the parameters “http:Caller” and “http:Request” as the input parameters to the function in order for it to get picked up as an interceptor function by the gateway.

Now copy the json_to_xml.bal file inside the interceptors folder in the “transform” project which was created earlier.

Then let’s declare the interceptor in the particular API by adding it to the open API using the open API extension “x-wso2-request-interceptors”. The function name should be declared as the value for the open API extension. Modify the open API as in below. (Red colored text show the added open API extension)

paths:
  "/echo":
    post:
      summary: Echoes the reponse.
      operationId: echoMethod
      x-wso2-request-interceptor: "jsonToXML"

Finally let’s build the project and run the build artifact as done earlier.  Now when we invoke the API with the same curl command, we could see that we are receiving the response in xml format. What happens here is that microgateway transforms the json payload to xml before sending it to the backend and the Netty service as usual echoes the payload back.

That was pretty much easy, isn’t it?

But main issue with with this approach of running heavy transformation logic at the gateway, especially when said gateway is used in shared mode, to expose multiple APIs is, heavy resource consumption of one API can affect the performance of all the other APIs. And when doing minor/major version upgrades of micro gateway, the ballerina code for all the APIs have to be updated to match with the ballerina version supported by that particular micro gateway. This can be a maintenance nightmare. Vendor specific technology-lock-in is also a concern here.

2. Transform within gateway using java interceptors

Similar to ballerina interceptors, the transformation logic can be developed in java and can be made to run inside the gateway itself.  The java code will be engaged during the request flow within the gateway similar to ballerina interceptors.

First create java maven project and Implements the java interface “org.wso2.micro.gateway.interceptor.Interceptor” which implements the json to xml conversion (JsonToXML.java class).

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>json-to-xml-transform</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
    </properties>
    <dependencies>
        <dependency>
                <groupId>org.wso2.am.microgw</groupId>
                <artifactId>mgw-interceptor</artifactId>
                <version>3.2.0</version>
        </dependency>
    </dependencies>

</project>  

JsonToXML.java

package org.mgw.transform;

import org.json.JSONObject;
import org.json.XML;
import org.wso2.micro.gateway.interceptor.Caller;
import org.wso2.micro.gateway.interceptor.Interceptor;
import org.wso2.micro.gateway.interceptor.InterceptorException;
import org.wso2.micro.gateway.interceptor.Request;
import org.wso2.micro.gateway.interceptor.Response;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.nio.charset.StandardCharsets;

public class JsonToXML implements Interceptor {
    private static final Logger log = LoggerFactory.getLogger("ballerina");

    public boolean interceptRequest(Caller caller, Request request) {
        try {
            JSONObject payload = request.getJsonPayload();
            String xml = XML.toString(payload);
            request.setBinaryPayload(xml.getBytes(StandardCharsets.UTF_8));
            request.setContentType("application/xml");
        } catch (InterceptorException e) {
            log.error("Error while getting json payload from request", e);
        }

        return true;
    }

    public boolean interceptResponse(Caller caller, Response response) {
        return true;
    }
}  

Build the maven project with “mvn clean install” and copy the build jar to the lib directory of the “transform” project.

Then edit the open API file of the transform project to declare the java interceptor class instead of the ballerina function name. The extension value should be in the format “java:<Full_Qualified _Class_Name>” (Refer red colored text in the below sample)

Finally build the project and run the build artifact using the gateway runtime as done earlier. Now also when you send the json request body with cURL command it will reply with the xml payload.

Similar to the Ballerina interceptors, the concern with this approach is running a heavy transformation logic at the gateway when it used in shared mode (expose multiple APIs) an issue with one API can affect all the other APIs. The request processing thread might consume a lot of processing time for the transformation, which keeps the thread busy; depriving API requests from getting handled.

Also, consideration needs to be given for processing of transformation logic when scaling the gateways. This may cause abnormal scaling behaviors when deploying on platforms that offer auto-scaling like Kubernetes.

3. Externalize the transformation logic

With this approach, we are focusing on running the transformation logic via an external service. This can be any external service like Spring, Netty, CXF on tomcat or an enterprise integrator which is designed for this kind of transformations.

In this blog I am using WSO2 Micro Integrator (4.0.0) which is an enterprise integrator which is specifically designed for transformations, mediations and other integration requirements.

Download the micro integrator (MI 4.0.0 and integration studio (low code tool to develop integrations) [5]

Let’s develop a simple proxy service in developer studio which will transform the json to xml. Documentation on how to create a proxy service using integration studio and deploy it in micro integrator can be found here[6].

Please find the sample configuration of the proxy service and the sequence created using integration studio to perform json to xml conversion.

Proxy service

<?xml version="1.0" encoding="UTF-8"?>
<proxy name="jsontoxml" startOnLoad="true" transports="http https" xmlns="http://ws.apache.org/ns/synapse">
    <target inSequence="convert_json_to_xml">
        <outSequence/>
        <faultSequence/>
    </target>
</proxy>

In Sequence

<?xml version="1.0" encoding="UTF-8"?>
<sequence name="convert_json_to_xml" trace="disable" xmlns="http://ws.apache.org/ns/synapse">
    <log level="full"/>
    <property name="messageType" scope="axis2" type="STRING" value="application/xml"/>
    <enrich>
        <source clone="false" xpath="$body/jsonObject/*"/>
        <target type="body"/>
    </enrich>
    <respond/>
</sequence>

Here the json to xml conversion can be simply achieve by setting the messgeType property as “application/xml”.  WSO2 MI includes the converted xml inside <jsonObject> parent xml tag. Enrich mediator is used to remove that xml parent tag.

Once the artifacts are ready, create the CAR file using integration studio and deploy the CAR file in the micro integrator. Once deployed the proxy service can be invoked with any json payload and it will respond back with response having the converted xml format.

paths:
  "/echo":
    post:
      summary: Echoes the reponse.
      operationId: echoMethod
      x-wso2-request-interceptor: java:org.mgw.transform.JsonToXML

Now since our transformation logic is running on an external server, lets simply write a ballerina code to call this external endpoint and engage the logic as a ballerina interceptor similar to the approach 1.

Below is a ballerina interceptor code snippet which call the external proxy service running on the micro integrator.

import ballerina/http;
import ballerina/log;

http:Client clientEP = new ("http://localhost:8290/services/jsontoxml");

public function callMI (http:Caller outboundEp, http:Request req) {
    http:Response|error resp = clientEP->forward("/", req);
    if (resp is http:Response) {
        xml|error x1 = resp.getXmlPayload();
        if (x1 is xml) {
            req.setXmlPayload(validate(x1));
        } else {
        log:printError("Error while getting the xml payload from external transform service", x1);
        }
    }
}

function validate(xml input) returns @untainted xml {
    return input;
}

According to this logic microgateway forwards the request to the micro integrator proxy service which is running on “localhost:8290/services/jsontoxml”. This call happens in non-blocking manner where the request thread is released once the call to the external endpoint has been made.

Similar to approach 1, let’s define this interceptor in the open API definition, so it will engage as a ballerina interceptor.

  "/echo":
    post:
      summary: Echoes the reponse.
      operationId: echoMethod
      x-wso2-request-interceptor: "callMI"

 Finally let’s build the microgateway project and run it using the microgateway runtime.

When invoked with a json payload using cURL it will respond with a xml payload (the converted json payload)

This approach is more suitable for any architecture that requires extensive message transformations. Performance of the gateway, and all the other APIs exposed by the gateway will remain consistent since transformation logic is not taking gateway CPU cycles. The architecture is more future proof as you have the flexibility to maintain own implementation of transformation logic and you are not bound to a particular vendor or technology. Whenever the organization decided to do a major upgrade of API gateway layer, or switch from one vendor to another still the existing transformation logic can be used.

Conclusion

In this blog I have tried to use a simple transformation requirement like json to xml conversion as an example to discuss the pros and cons of three different architectural patterns commonly used with WSO2 Microgateway. I hope this article will help you to identify the most appropriate, future proof approach that best suits your organization’s requirements.