In the previous blog we looked at of the lesser documented transports in the WSO2 ESB is the VFS (Virtual Files System) transport. This Axis2 transport allows you to move files from one location to another. But how do we actually use it? In this WSO2 Tutorial we will dive a little deeper in the WSO2 ESB VFS transport and see how we can actually do something with the content of a file.
We continue with the proxy and file service we used in the previous blog. If you haven’t read it, VFS transport is enabled on axis2. The file can be found at [ESB-HOME]/repository/conf/axis2/axis2.xml
The [ESB-HOME] refers to the fully qualified path to the installed version of the ESB.
Remove the <!—and –> to enable the Receiver and Listener. Contrary to other transports there are no other parameters to be defined. The receiver looks like this:
And the sender like this.
Restart the ESB since transports are only loaded at start, not when they are changed on the fly.
Create the proxy from the first blog on VFS or download the car from bitbucket.
Restrict by file size
What we are going to do is that we are first going to restrict by filesize in the VFS transport. When we want to do that we need to set the transport.vfs.FileSizeLimit parameter. This parameter dictates the maximum file size (in bytes) , only smaller files will be picked up.
Let’s say that we want only files that are less or equal to a 1000 bytes.
Creating the files
Need files?
We will create four sets of files:
- 20 files of exactly 1000 bytes;
- 20 files of exactly 2000 bytes;
- 20 files of exactly 750 bytes;
- 20 files of exactly 1001 bytes.
We will use the FSUTIL command in windows to create them but will name them differently.
For the purpose of this blog we will create the files in the directory C:WSO2ESBVFSINPUT directory.
If you haven’t created this directory structure, the commands are for Windows:
cd
md WSO2ESBVFSINPUT
cd WSO2ESBVFSINPUT
Follow these commands with the commands to create the files.
For /L %i in (1,1,20) do fsutil file createnew VFS_750_%i.dat 750
For /L %i in (1,1,20) do fsutil file createnew VFS_1000_%i.dat 1000
For /L %i in (1,1,20) do fsutil file createnew VFS_1001_%i.dat 1001
For /L %i in (1,1,20) do fsutil file createnew VFS_2000_%i.dat 2000
Now you have the files in the right location. These files are completely empty of course but for the purpose of this blog quite suitable!
We need to create three more directories in order to make it work. These are the windows commands:
cd
md WSO2ESBVFSOUTPUT
MD WSO2ESBVFSFAILURE
MD WSO2ESBVFSORIGINAL
This is of course a command line command in Windows.
We add a parameter to the FileProxy called transport.vfs.FileSizeLimit with a value of 1000. As you know parameters are not mediators and therefore not shown in Developer Studio Graphical Design but are shown in source:
<parameter name="transport.PollInterval">15</parameter>
<parameter name="transport.vfs.FileSizeLimit">1000</parameter>
<parameter name="transport.vfs.FileURI">file:///C:/WSO2/ESB/VFS/INPUT/</parameter>
<parameter name="transport.vfs.ContentType">text/xml</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.vfs.MoveAfterFailure">file:///C:/WSO2/ESB/VFS/FAILURE/</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
<parameter name="transport.vfs.FileNamePattern">.*.dat</parameter>
<parameter name="transport.vfs.MoveAfterProcess">file:///C:/WSO2/ESB/VFS/ORIGINAL/</parameter>
We redeploy the proxy to the ESB and watch what is going to happen. Since we created 20 files of each category, the ones that match our criteria will pass. Those criteria are the extension (*.dat) and a maximum file size of 1000.
| Name | Size | Result |
| VFS_750_xxx | 750 | Will pass |
| VFS_1000_xxx | 1000 | Will pass |
| VFS_1001_xxx | 1001 | Will not pass |
| VFS_2000_xxx | 2000 | Will not pass |
Now we have more control. But if you need files of a certain size:
- Below x bytes;
- Above x bytes and below y bytes;
- Between x and y bytes?
The first situation is quite simple and possible with the current setup by defining the maximum file size.
In example two we need to do a bit more. We create a first proxy that:
Filters with a maximum of the orange bar, those files are moved. A second proxy will filter the remaining files away with a maximum up to the blue bar. However, these files that are now moved should be discarded. The remaining files (the ones over 100) now need to be gathered and moved. This can again be done using a proxy without any file limitation, simply gathering the files to a directory.
The only thing you need to do is to actually make sure that the proxies are done sequentially for instance by setting a polling interval that is big enough to make sure that they act sequentially. Possibly even with a proxy that moves from a receiving URL to a Processing URL so during second processing no new files are introduced.
Example number 3 works in a similar fashion. First move all files below 100, a second proxy to move all files below 60 and retain the files in between. But it seems like a lot of work, isn’t there a better way?
Alternatives
Alternatively, you could create a sequence that allows you to define the parameters. You have access to a number of variables that are filled when the VFS is used. An XPATH statement can be used to determine the files that need to be moved. These are some of these values:
- $trp:FILE_PATH
- $trp:FILE_NAME
- $trp:FILE_URI
- $trp:FILE_LENGTH
The $trp is a shortcut for the get-property statement on the transport level and work just as fast as the get-property statement.
Setting up the test environment
In order to setup the files we create a number of files of different sizes. It is not important to have a massive number of files as long as we have different sizes.
For /L %i in (1,1,5) do fsutil file createnew VFS_750_%i.dat 750
For /L %i in (1,1,5) do fsutil file createnew VFS_1000_%i.dat 1000
For /L %i in (1,1,5) do fsutil file createnew VFS_1500_%i.dat 1500
For /L %i in (1,1,5) do fsutil file createnew VFS_2000_%i.dat 2000
For /L %i in (1,1,5) do fsutil file createnew VFS_2001_%i.dat 2001
We create 5 instances of these different sized files.
What we need to do next is to turn off the automatic movement of files in favor of a manual movement based on a Filter mediator. It could also be a SWITCH mediator if we have multiple criteria.
In this case we keep it simple. We create a proxy FileProxyMaxSize with a filter mediator that in the Then branch writes the file to another directory and on the Else branch does nothing.
Of course, the code shows what is going on.
<?xml version="1.0" encoding="UTF-8"?>
<proxy name="FileProxyMaxSize" startOnLoad="true" transports="vfs" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<property expression="$trp:FILE_LENGTH" name="FL"
scope="default" type="STRING"/>
<log level="custom">
<property expression="get-property('FL')" name="FL2"/>
<property name="Status" value="FileProxyMaxSize"/>
</log>
<filter xpath="($ctx:FL >= 1000) and ($ctx:FL <= 2000)">
<then>
<log level="custom">
<property name="sequence" value="Will move this one"/>
<property expression="$ctx:FL" name="Size"/>
</log>
<clone>
<target sequence="fileWriteSequence"/>
</clone>
</then>
<else>
<log level="custom">
<property name="sequence" value="Wont move this one"/>
<property expression="$ctx:FL" name="Size"/>
</log>
</else>
</filter>
</inSequence>
<outSequence/>
<faultSequence/>
</target>
<parameter name="transport.PollInterval">15</parameter>
<parameter name="transport.vfs.FileSizeLimit">-1</parameter>
<parameter name="transport.vfs.FileURI">file:///C:/WSO2/ESB/VFS/INPUT/</parameter>
<parameter name="transport.vfs.ContentType">text/plain</parameter>
<parameter name="transport.vfs.ActionAfterProcess">DELETE</parameter>
<parameter name="transport.vfs.MoveAfterFailure">file:///C:/WSO2/ESB/VFS/FAILURE/</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
<parameter name="transport.vfs.FileNamePattern">.*.dat</parameter>
</proxy>
The FileWriteSequence copies the file to the output directory. Note that we are less using the VFS principle but are using the functionality.
In the source you can see the simple steps.
<?xml version="1.0" encoding="UTF-8"?>
<sequence name="fileWriteSequence" trace="disable" xmlns="http://ws.apache.org/ns/synapse">
<property expression="$trp:FILE_NAME" name="filename" scope="default" type="STRING"/>
<property expression="get-property('filename')" name="transport.vfs.ReplyFileName" scope="transport" type="STRING" xmlns_ns2="http://org.apache.synapse/xsd"/>
<property name="OUT_ONLY" scope="default" type="STRING" value="true"/>
<send>
<endpoint name="FileEpr">
<address uri="vfs:file:///c:/WSO2/ESB/VFS/OUTPUT/"/>
</endpoint>
</send>
</sequence>
Testing the setup
Let’s give it a spin. We create a C-App and CAR file from the two artifacts we just created and deploy them to the server. We will not show this process here to make the article not too long.
However, we want to look if the CAR is deployed and the proxy and sequence are on the ESB:
And the sequence is there.
Before
Before we start the proxy we have 25 files created using the FSUTIL command described earlier.
After processing we will have 15 files (1000 bytes, 1500 bytes and 2000 bytes) in our OUTPUT directory. The files that did not match were deleted since after process we DELETE the files, rather than move them.
I am now starting the ESB and immediately starts picking up and moving files.
In the Input directory, all files are gone.
And in output the files we wrote there.
[2017-07-27 13:30:25,292] INFO - LogMediator sequence = Will move this one, Size = 2000
[2017-07-27 13:30:25,297] INFO - LogMediator FL2 = 2000, Status = FileProxyMaxSize
[2017-07-27 13:30:25,298] INFO - LogMediator sequence = Will move this one, Size = 2000
[2017-07-27 13:30:25,303] INFO - LogMediator FL2 = 2001, Status = FileProxyMaxSize
[2017-07-27 13:30:25,303] INFO - LogMediator sequence = Wont move this one, Size = 2001
[2017-07-27 13:30:25,325] INFO - LogMediator sequence = Wont move this one, Size = 750
On the console we find the log mediators, showing files that were and that weren’t moved.
This is of course a very simple setup of a VFS transport, just moving it depending on xpath based rules on FILE_LENGTH. In the next blog we will extend the functionality and see how we can do something with the content of such a file.
If you have any questions about this blogpost contact us via the comments section of this blog. View also our WSO2 Tutorials, webinars or white papers for more technical information. Need support? We do deliver WSO2 Product Support, WSO2 Development Support, WSO2 Operational Support and WSO2 Training Programs.
