It might seem to you that museums are the ultimate example of brick and mortar. You have to go there in order to see the masterpieces on display. And of course, when you walk through a museum and see all the beautiful art it is a special kind of experience. However, some museums also have a digital section because their collections are often so big that they cannot display everything. Hundreds of thousands of images are not on display, but they’re kept in depots out of the sight of people. Digitalization, in other words taking a picture and putting it online allows them to show more of their collection and let people enjoy art. In some cases, even creating derivative works from masterpieces. The mural digital canvas is one of those devices that will allow you to upload your favorite piece of art onto the device and display it in your own home or office. It is a form of digital imagination to see what you can do with such a device and the digital resources that museums expose.
Let us continue
In the previous part of this blog, we developed an API that was able to retrieve a digitized image from the Rijksmuseum. And the second part we are going to extend that simply by allowing multiple objects in the adjacent array and also actually storing the images on your local drive.
The reason why I’m writing this blog is to show you that integration is something that is not only for experts or artists but that almost everyone can do it. Of course, you need training, and I would say to some extent aptitude in order to be able to integrate but I think that in general most developers are able to integrate systems.
The set up
Just to recap what you need if you want to try this out yourself:
- A version of the Micro Integrator (I am using version 4.2.0)
- Integration Studio in order to develop the artifacts that we’re going to deploy (I am using version 8.2)
- An API key from the Rijksmuseum in order to utilize their API
- A Meural canvas if you want to have the complete flow. If not, you can store the image locally
Iterate
When you look at the setup we’re trying to setup you’ll see that there is an array of values that is passed to the API. For each of these values in the array we would like to retrieve the information about the painting. Within the Micro Integrator there are two mediations that serve this purpose. The first one is the ForEach mediator and the second one is the Iterate mediator which will allow you to take a repeating element in a payload and execute logic for each of them.
In our case, as we would like to call another API based on the value that we’re getting from the array we should use the iterate mediator. The foreach mediator is not suitable because of the fact that we want to access an (external) system.
When you look at the syntax of the Iterate mediator there are a number of parameters that you can set depending on the use case that you have. The mediator is often accompanied by the aggregate mediator which allows us to first split something and then aggregate the results, you can however also discard the message. In our case we don’t need that because we’re not so much interested in combining the responses but more on executing our call based on the value that is concatenated in the URL. There are of course also other ways of doing this. We could also store the values that we’re getting from an array on the message queue and retrieve the messages from that queue and process them sequentially.
Store on a local drive
We already saw that we got a URL back that gives us the download location of a piece of art. What we would like to do now is to actually retrieve the image and then write it to a local drive. This is also a part of integration, in order to make this work we need to add the capability to write to a local device and we do that in the form of the WSO2 File connector.
Connectors are extensions to the Micro Integrator that will allow it to access cloud-based services or in our case with the file connector, our local file system. Rather than figuring everything out yourself these prebuilt zip files can be added to the Micro Integrator and will greatly expand the capabilities of the Micro Integrator.
Almost 200 connectors in the store
The connectors can be found online at this URL (https://store.wso2.com/store/pages/top-assets) and I’ve counted 193 connectors at the moment of writing this. Not all of them are current I must add but it will save you a lot of time because you don’t have to figure out how to make the connection to a specific system or service. The way that you go about using these connectors is that you take a look if the version that you’re using is supported and even when it isn’t mentioned that the version is supported you can always try it out. The connectors themselves are zip files and you can easily open them up and look at the configuration inside. In most cases it will be regular synapse configuration (the same language that is used in the message mediation we’re doing here), but it can also be that there is a Java jar file inside the connector. In the case of the file connector where we need to access the low-level file system this is indeed the case.
If you want to know more about connectors and what you can do with them, you can take a look at one of the blogs that I’ve written over the years on this topic.
The connector typically is downloaded both to your integration development environment like Integration Studio and to the product where you’ll be running the integration. In Integration Studio it’s added to the system in order to enable the use of the connector in the same way you build your mediations through high level drag and drop. Details can be configured in the Properties View or Source View.
In case of the deployment, it is also important that we add the connector to the artifacts that we would like to deploy since there is a dependency that needs to be met in order to be able to execute the artifacts.
Connectors can be added in a very simple way using an integration project where we simply check the box for connectors. We did this in the previous blog post already.
But there is something else we also need to do. And it has to do with the fact that when a message is received by the Micro Integrator by default it will try and parse the message to be able to mediate on the content. The mechanism uses transports like HTTP and HTTPS as well as builders and formatters that will take the message that is coming in and transform it into the internal axiom model. When the message is sent out it will be built according to the message type that you define. When you’re downloading a digital image there is no need to parse the message it simply needs to be wrapped in a soap envelope without Micro Integrator trying to open up the message. To enable this the Micro Integrator has a relay mechanism that will not try to open up the message simply will accept the message wrap it and send it through or do something else.
In order to configure this, we are going to make a change to the deployment.toml file where all of the product’s configurations are managed. This file can be found in the [MI-HOME]/conf directory of the product.
[[custom_message_builders]]
class = "org.wso2.carbon.relay.BinaryRelayBuilder"
content_type = "image/jpeg"
[[custom_message_formatters]]
class = "org.wso2.carbon.relay.ExpandingMessageFormatter"
content_type = "image/jpeg"Now when a message with a content type image/jpeg comes in it will be handled according to those two Java classes for builder and the formatter. When you’ve made this change, you need to restart the Micro Integrator because the deployment.toml file is only read at start up.
When we look at the API from part one let’s put the new pieces in there. The big change is of course the iterate mediator that will loop through the JSON array that we are going to submit to the API and the code that will write the actual file to our local drive.
I’m not going to explain every part of this configuration because it would take up too much space and the blog would be too long. I have created a Bitbucket repo for you to download the resources / artifacts.
Tinkering
You will find settings that you need to set when you start building these kinds of mediations.
For instance, when you’re working with the Json array, and you want to retrieve the value I found that it was necessary to set the “attachPath” so that the individual values within the array are prefixed into a value keypair. If you were to do this using an XML message it is not necessary to do so. Another thing is that we are currently using a small number of objects that we would like to process. Whenever you are going to larger numbers you might need to change memory settings because when it starts processing this it will pick up and start a number of threads and we’ll load the messages in memory. For this purpose, I had to set the memory to 4096 megabytes, four times the normal setting for the maximum memory that Micro Integrator is going to use. This setting is actually done in the startup script rather than in the deployment.toml. But at that time, I was able to process over 60 objects in one payload.
Below I’ve created JSON Array payload, {“object”: [“SK-C-5”, “SK-C-4”] } which I am passing to the API.
Two files are created in the assigned directory. They are complete jpg files as you can see.
Transfer of art completed! That’s enough for this blog. In the next episode, called Part 3, we are doing the final step and that is uploading it to our Meural. Stay tuned.
