info@yenlo.com
eng
Menu
WSO2 Enterprise Integrator 6 min

Transforming EDI Messages to XML and JSON

Rob Blaauboer
Rob Blaauboer
Integration Consultant & WSO2 Trainer
Transforming messages

Transforming messages Electronic Data Interchange (EDI) has been around for many decades and has been replaced to a large extend by other ways of data interchange like XML (computer readable) and newer formats like JSON. Still there are EDI messages in use, perhaps in your organization. How do you turn these messages into more modern formats? With WSO2 EI of course. In this blog, we will take a dummy EDI message file and turn it into an XML file. As a second step, we store the data in a database as well. First things first. How will we process files?


VFS transport

We will use the VFS transport to process an EDI file (a simple plaintext file). The VFS transport comes in two flavors, the sender and receiver. Uncomment both in the [EI-HOME]/conf/axis2/axi2.xml. Keep in mind that these two are not next to each other in the axis2 file:

<transportSender name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportSender"/>

  <transportReceiver name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportListener"/>

Furthermore, we will use the PlainTextBuilder, a message builder that is available in axis2 file.

<messageBuilder contentType="text/plain" class="org.apache.axis2.format.PlainTextBuilder"/>

All changes to axis2 require a restart of the WSO2 EI since these parameters are only loaded at startup.

Configuration of the VFS

As you can read in other blogs on VFS we need to configure the input directory on which we listen to a new file:

<?xml version="1.0" encoding="UTF-8"?>
<proxy
       name="SmooksSample"
       startOnLoad="true"
       statistics="disable"
       trace="disable"
       transports="vfs">
   <target>
      <inSequence>
         <smooks config-key="smooks-key">
            <input type="text"/>
            <output type="xml"/>
         </smooks>
      </inSequence>
   </target>

   <parameter name="transport.PollInterval">5</parameter>
   <parameter name="transport.vfs.FileURI">file:///C:EIwso2ei-6.2.0smooksin</parameter>

<parameter name="transport.vfs.ContentType">text/plain</parameter>
   <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
   <parameter name="transport.vfs.MoveAfterFailure">file:///C:EIwso2ei-6.2.0smooksfail</parameter>
   <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
   <parameter name="transport.vfs.FileNamePattern">.*.txt</parameter>
   <parameter name="transport.vfs.MoveAfterProcess">file:///C:EIwso2ei-6.2.0smooksoriginal</parameter>
   <description/>
</proxy>

Smooks

The graphical overview is extremely simple, especially since parameters are not visible in the graphical mode but are in the source mode. Of course, you need to change the parameters in the proxy and local entry in order to match your setup.

Smooks

Smooks is an extensible framework for building applications for processing XML and non XML data (CSV, EDI etc). The Smooks mediator allows you to do that within the Enterprise Integrator. It can be used to perform a wide range of Data Transforms – XML to XML, CSV to XML, EDI to XML, XML to EDI, XML to CSV etc.

The highlevel syntax is as follows:

<smooks [config-key="string"]>
<input [type="|text|xml"]/>
<output [type="|text|xml|java"] [property="string"] [action="string"]/>
</smooks>

The config-key is used to access the Smooks configuration. The Smooks configuration should be saved in the Registry as a local entry before it can be used here. You can also refer to a file for the config-key. The Smooks configuration file uses a number of other technologies to make it work.

First of all, we see a number of xsd’s included like javabean, file-routing and freemarker. The data will be read one line at a time by the SAX parser.

In the EDI mapper the line is transformed according to the layout of the EDI message and subsequently put into a hashtable and using the freemarker template written to an xml file.

<smooks-resource-list
       
        xmlns_jb="http://www.milyn.org/xsd/smooks/javabean-1.2.xsd"
        xmlns_file="http://www.milyn.org/xsd/smooks/file-routing-1.1.xsd"
        xmlns_ftl="http://www.milyn.org/xsd/smooks/freemarker-1.1.xsd">
 
      <params>
              <param name="stream.filter.type">SAX</param>
              <param name="default.serialization.on">false</param>
       </params> 

       <!-- Configure the EDI Parser to parse the message stream into a stream of SAX events. -->
       <resource-config selector="org.xml.sax.driver">
              <resource>org.milyn.smooks.edi.EDIReader</resource>
              <param name="mapping-model">samples/service-bus/resources/smooks/edi-mapping.xml</param>
       </resource-config>

          <!-- Populate java objects to message data-->

       <jb:bean beanId="order" class="java.util.Hashtable" createOnElement="order">
              <jb:value property="id" decoder="Integer" data="order/id"/>
              <jb:value property="price" decoder="Double" data="order/price"/>
              <jb:value property="quantity" decoder="Integer" data="order/quantity"/>
              <jb:value property="symbol" data="order/symbol"/>
       </jb:bean>

 
      <!-- Apply freemarker templating -->
       <ftl:freemarker applyOnElement="order">
              <ftl:template><!--<order id="${order.id}">
       <quantity>${order.quantity}</quantity>

        <price>${order.price}</price>
       <product-symbol>${order.symbol}</product-symbol>
       </order>--></ftl:template>
              <ftl:use>
                      <!-- Output the templating result to the
"orderItemSplitStream" file output stream... -->
                      <ftl:outputTo outputStreamResource="orderItemSplitStream"/>
              </ftl:use>
       </ftl:freemarker> 
       <!-- Create/open a file output stream.  This is writen to by the freemarker template (above).. -->
        <file:outputStream resourceName="orderItemSplitStream" openOnElement="order">

<file:fileNamePattern>order-${order.id}.xml</file:fileNamePattern>
                <file:destinationDirectoryPattern>C:EIwso2ei-6.1.1smooksorders</file:destinationDirectoryPattern>
                <file:highWaterMark mark="100000"/>
        </file:outputStream>

 </smooks-resource-list>

Setup

We have adapted this blog from the sample 659 included with the Enterprise Integrator. Setup an EI-6.2.0 on your machine. Do it close to the root when working on Windows due to the PATH length restriction.

I made a setup where the proxy will listen on a directory called C:EIwos2ei-6.2.0smooks with in, original, fail and orders directories. As you can see, this is a Windows setup. So you will need to change the settings to match your setup. At the Yenlo Bitbucket you can find all resources needed. I’ve also changed the location of the smooks-config.659 in the Bitbucket Sample to be in the smooks subdirectoru

Open the proxy and change to the directory locations applicable to you.

  • transports.vfs.FileURI– The input file location.
  • transports.vfs.MoveAfterProcess –  The location to move the input file after it is processed.
  • transports.vfs.MoveAfterFailure–  The location to move the input file if there happens to be a failure at the time of processing.

Open the <[EI_HOME]/repository/sample/smooks-config-659.xml and the proxy (see below) and change to the directory locations applicable to you. It is the <file:destinationDirectoryPattern> you need to change.

The proxy uses the VFS system to listen to a directory and will take the file, parse it and write the file to the directory orders. The Name is assembled from the EDI file and the order number.

<order id="1">
       <quantity>500</quantity>
       <price>50</price>
       <product-symbol>IBM</product-symbol>
</order>

So what can we do with this?

The setup we have created can be used for other EDI messages by changing the EDI mapping to support other layouts. You now know that an EDI message can be read by the Enterprise Integrator with minimal changes. Only the VFS needs to be enabled and a plain text message builder added to the axis2.xml configuration.

We can extend this example by taking the orders and write them to a database, JMS queue or something similar. Please let us know if you interested to hear or read more about that.

Yenlo is the leading, global, multi-technology integration specialist in the field of API-management, Integration technology and Identity Management. Known for our strong focus on best-of-breed hybrid and cloud-based iPaaS technologies.

eng
Close