During the Covid pandemic we’ve seen numerous examples of cases where acting fast was a necessity. For instance, restaurants that suddenly were no longer allowed to open their doors for business, but instead were only allowed to offer take away food.
Given the abrupt circumstances in which we found ourselves at the time, we did not need a perfect solution, but we needed a solution that was ‘good enough’. Every day that this solution was not available yet, caused mass revenue loss. We could have improved this solution a lot if we had more time, but at that moment it worked.
Another example of such a solution is an ice cream parlor that allowed people to order online and pick up at the store. Something that was not in their standard way of working. But it allowed the restaurant to at least do some business.
Quick solution to register client data – Microsoft
Some time ago, one of my Microsoft contacts on LinkedIn shared an example of how existing Microsoft tools and applications could be used to quickly register client data, which is another important demand in Covid times. Government decided that restaurateurs now need to register data from visitors to track and manage possible virus spreading.
Pen and paper do not cut it
Of course, the simplest solution is something that has been around for ages: pen and paper. But if you store your information digitally, you can improve quality. In addition, if you do it in the right way, you can make sure that the data will get deleted after a specific amount of time, let’s say, a week or two. We do not want to store unnecessary data or retain data longer than needed.
How to set up a solution to quickly store data with WSO2 products?
I’m writing this blog to show how easy it is, and how quickly we can create something to try out. I’m not suggesting that we should have a perpetual beta, but it’s quite simple to develop something to try out and use in case of haste. Adding the PostNL API is easy, but it will allow you to validate and improve data and, more importantly, shorten the process.
A simple setup that stores data
What I’m going to show is that we can create a very simple setup that allows you to store data and, to some extent, also validate data with regards to the data that has been entered, like for instance, postcode, in just a matter of a couple of hours. For this we need a simple front-end, and a couple of APIs, together with a simple database management system, some stored procedures, and of course, the products from WSO2.
Enterprise Integrator
I’m going to use WSO2 Enterprise Integrator. In this case, I’m going to use the setup with a Micro Integrator. This way I can offer it in a Docker environment, allowing for more scalability but offering the functionality of the Enterprise Integrator.
MariaDB
So, let’s get started, what do we need? To begin we surely need a place to store the data. In this case, I’m using a database management system like MariaDB, which is a database management system that is easy to use and can actually perform a lot of the functionality that you would need.
Let’s get started with the database setup. What I need is: first name, last name, zip code or postcode, an NL mobile phone number and an email address. Of course, there is also going to be a date timestamp. When the information is put in the database, the date timestamp is going to be used for purposes of purging the data at a certain point in time. With purging the data I mean: complying to laws and regulations for storing this data.
This is quite a simple setup that shows you how quickly you can set something up like this. I am going to create a primary key that is, of course, going to be unique, and it is an auto incremented number. The database creation is shown below. We use the word Postcode which is the equivalent of zip code in the US, at least from a semantic perspective.
#coronapostcode
CREATE DATABASE IF NOT EXISTS coronapostcode;
USE coronapostcode;
CREATE TABLE IF NOT EXISTS guests (
GID INTEGER NOT NULL AUTO_INCREMENT,
NAME VARCHAR(40) NOT NULL,
CELLPHONE VARCHAR(10) NOT NULL,
ADDRESS VARCHAR (50) NOT NULL,
HOUSENR VARCHAR (5) NOT NULL,
POSTCODE VARCHAR(50) NOT NULL,
TOWN VARCHAR(50) NOT NULL,
DTSTAMP datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (GID)
);
SET insert_id=2020;
INSERT INTO guests (name,cellphone,address,housenr,postcode,town) VALUES ('Jill Doe', '0612345678','Rokin','12','1002AB','Amsterdam');
select * from guests;Ok, so the database setup primary key is created. I’ve inserted a record to see if this works. Now let us generate the Soap Services. You can see how that goes in this video.
It is important to realize that the generation of the data service that we just did, took less than two minutes, and already offers a SOAP interface that allows us to insert data into the database. So, with a simple front-end development that will call the SOAP service we are ready to use the service. I will not go into details on this because front-end development is not my expertise and beyond the scope of this blog (we focus on integration).
The Enterprise Integrator can also front the SOAP Service with an API so we can do a RESTful invocation that is more modern and easier to do for a front-end development team.
Create the API
And now of course, I need to create the API. Why an API? Well, it’s the most natural thing to do. If you’re storing information, APIs are the way to go. They’re simple. They have a large following as far as developer scope and of course, they are quite easy to create. In our case, we are just going to be doing an HTTP POST having information as a JSON payload to add to the database. I am going to create the API on the Enterprise Integrator because I want to do some mediation on the payload.
From API to Soap
Creating an API that talks to a SOAP backend is actually quite simple. You need to:
- Add a Header and set to the appropriate Action
- Add the PayloadFactory Mediator
- Add a Call Mediator and an Endpoint
- Respond back to the client
The video shows setting up the high-level structure.
The payload we can view when we use the WSDL in for instance SoapUI.
The source code shows the details.
<?xml version="1.0" encoding="UTF-8"?>
<api context="/add" name="Customer" xmlns="http://ws.apache.org/ns/synapse">
<resource methods="POST">
<inSequence>
<header name="Action" scope="default" value="insert"/>
<payloadFactory media-type="xml">
<format>
<soapenv:Envelope xmlns:dat="http://ws.wso2.org/dataservice" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Header/>
<soapenv:Body>
<dat:insert>
<dat:name>$1</dat:name>
<dat:cellphone>$2</dat:cellphone>
<dat:address>$3</dat:address>
<dat:housenr>$4</dat:housenr>
<dat:town>$5</dat:town>
<dat:postcode>$6</dat:postcode>
</dat:insert>
</soapenv:Body>
</soapenv:Envelope>
</format>
<args>
<arg evaluator="xml" expression="$body//name"/>
<arg evaluator="xml" expression="$body//cellphone"/>
<arg evaluator="xml" expression="$body//address"/>
<arg evaluator="xml" expression="$body//housenr"/>
<arg evaluator="xml" expression="$body//town"/>
<arg evaluator="xml" expression="$body//postcode"/>
</args>
</payloadFactory>
<call>
<endpoint>
<address format="soap11" uri="http://localhost:8280/services/Guest">
<suspendOnFailure>
<initialDuration>-1</initialDuration>
<progressionFactor>-1</progressionFactor>
<maximumDuration>0</maximumDuration>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>0</retriesBeforeSuspension>
</markForSuspension>
</address>
</endpoint>
</call>
<respond/>
</inSequence>
<outSequence/>
<faultSequence/>
</resource>
</api>But there is more. Because we can validate the data that’s coming into our API.
Validate the data with POSTNL API
To validate the data, we can use an API like the one from PostNL. They have several APIs to integrate their services, like shipping and tracking into all kinds of systems like webshops and so on. You can find the APIs online, where you can also request your own trial API Key to try it out yourself.
Strictly taken, the solution we created is ‘good enough’. We store data in a database, and we can recall the information. We can also make it easier, since in the Netherlands the combination of postcode and number (e.g., 1119 PT, 16) will give us the address if the combination exists. So, we can just ask for the name, phone number and postcode / house number and make it easier for our customers to automatically determine the street and town.
I’ve signed up to try out the API for Benelux Address Control. This allows me to try out the API. I have tried it with SoapUI to see what kind of information I get back for our own office. Although Postman is a more natural fit for APIs, I am using SoapUI because I also used it to copy the SOAP message in the payload factory. PostNL offers Postman collections online so if you want to explore Postman you can also do that.
This does however require some work.
We first need to store the original message (the individual fields) and call the PostNL API. In the documentation we found the URL and parameters. We will create the needed payload (the same as shown above, add the apiKey as a Header and call the API). The response we get back gives the information we need. Most of the fields in the API definition are optional so you can also validate on more information. You will find more information on the PostNL API in the documentation.
In this blog I am assuming that I input a correct Postcode and HouseNumber, of course this normally requires error handling as well. To keep it simple I am omitting this. I am also making two errors in the original data when invoking the API Later on (Sciphol and Beachavenue, both mispelled). By taking the data from PostNL as our basis (they are the source) I can always override the input from the customer to match the PostNL API’s response.
The code is simple. This is the graphical overview.
In code it looks like this.
<?xml version="1.0" encoding="UTF-8"?>
<api context="/add" name="Customer" xmlns="http://ws.apache.org/ns/synapse">
<resource methods="POST">
<inSequence>
<propertyGroup>
<property expression="$body//name" name="orgname" scope="default" type="STRING"/>
<property expression="$body//cellphone" name="orgcellphone" scope="default" type="STRING"/>
<property expression="$body//address" name="orgaddress" scope="default" type="STRING"/>
<property expression="$body//housenr" name="orghousenr" scope="default" type="STRING"/>
<property expression="$body//town" name="orgtown" scope="default" type="STRING"/>
<property expression="$body//postcode" name="orgpostcode" scope="default" type="STRING"/>
</propertyGroup>
<payloadFactory media-type="json">
<format>{
"CountryIso": "NL",
"HouseNumber": "$1",
"PostalCode": "$2"
}
</format>
<args>
<arg evaluator="xml" expression="$ctx:orghousenr"/>
<arg evaluator="xml" expression="$ctx:orgpostcode"/>
</args>
</payloadFactory>
<propertyGroup>
<property name="messageType" scope="axis2" type="STRING" value="application/json"/>
<property name="apikey" scope="transport" type="STRING" value="**Fill in the API key you received by requesting a trial API key**"/>
</propertyGroup>
<call>
<endpoint>
<http method="post" uri-template="https://api.postnl.nl/address/benelux/v1/validate">
<suspendOnFailure>
<initialDuration>-1</initialDuration>
<progressionFactor>-1</progressionFactor>
<maximumDuration>0</maximumDuration>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>0</retriesBeforeSuspension>
</markForSuspension>
</http>
</endpoint>
</call>
<propertyGroup>
<property expression="$body//City" name="City" scope="default" type="STRING"/>
</propertyGroup>
<filter xpath="$ctx:orgtown!=$ctx:City">
<then>
<property expression="$ctx:City" name="orgtown" scope="default" type="STRING"/>
</then>
</filter>
<header name="Action" scope="default" value="insert"/>
<payloadFactory media-type="xml">
<format>
<soapenv:Envelope xmlns:dat="http://ws.wso2.org/dataservice" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Header/>
<soapenv:Body>
<dat:insert>
<dat:name>$1</dat:name>
<dat:cellphone>$2</dat:cellphone>
<dat:address>$3</dat:address>
<dat:housenr>$4</dat:housenr>
<dat:town>$5</dat:town>
<dat:postcode>$6</dat:postcode>
</dat:insert>
</soapenv:Body>
</soapenv:Envelope>
</format>
<args>
<arg evaluator="xml" expression="$ctx:orgname"/>
<arg evaluator="xml" expression="$ctx:orgcellphone"/>
<arg evaluator="xml" expression="$ctx:orgaddress"/>
<arg evaluator="xml" expression="$ctx:orghousenr"/>
<arg evaluator="xml" expression="$ctx:orgtown"/>
<arg evaluator="xml" expression="$ctx:orgpostcode"/>
</args>
</payloadFactory>
<call>
<endpoint>
<address format="soap11" uri="http://localhost:8280/services/Guest">
<suspendOnFailure>
<initialDuration>-1</initialDuration>
<progressionFactor>-1</progressionFactor>
<maximumDuration>0</maximumDuration>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>0</retriesBeforeSuspension>
</markForSuspension>
</address>
</endpoint>
</call>
<payloadFactory media-type="json">
<format>{ "record status": "added" }</format>
<args/>
</payloadFactory>
<respond/>
</inSequence>
<outSequence/>
<faultSequence/>
</resource>
</api>Let us send a message to the API
Let us consider the payload below as something that a guest has filled in on their mobile device. I am using SoapUI to mimic the message.
Checking the data
Since we insert a record in a database and did not create any response other than the standard 202 Accepted, we need to see if the data is stored correctly.
A quick select on the database (select * from guests) shows that the data is there. A reader with a keen eye will see that Beachavenue is not changed, an additional filter statement should correct that. Or, and that is also something, you always override the data from the customer with the response from PostNL.
Conclusion
Creating ‘good enough’ applications is easy to do. Of course, this is a simplified example that does not concern a frontend and omits basic error handling. However, it is also easy to expand later on into a minimal viable integration. But for now, it is good enough.
You can find the source code at the Yenlo Bitbucket page.
