Discover our knowledge. Read our blogs!

Learn more

We build all our solutions with WSO2 and we are proud that we are WSO2 Premier Certified Integration Partner and Value-Added Reseller.

Learn more

Transforming EDI Messages to XML and JSON

14 min read

Transforming messagesElectronic Data Interchange (EDI) has been around for many decades and has been replaced to a large extend by other ways of data interchange like XML (computer readable) and newer formats like JSON. Still there are EDI messages in use, perhaps in your organization. How do you turn these messages into more modern formats? With WSO2 EI of course. In this blog, we will take a dummy EDI message file and turn it into an XML file. As a second step, we store the data in a database as well. First things first. How will we process files?


VFS transport

We will use the VFS transport to process an EDI file (a simple plaintext file). The VFS transport comes in two flavors, the sender and receiver. Uncomment both in the [EI-HOME]/conf/axis2/axi2.xml. Keep in mind that these two are not next to each other in the axis2 file:

<transportSender name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportSender"/>

  <transportReceiver name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportListener"/>

Furthermore, we will use the PlainTextBuilder, a message builder that is available in axis2 file.

<messageBuilder contentType="text/plain" class="org.apache.axis2.format.PlainTextBuilder"/>

All changes to axis2 require a restart of the WSO2 EI since these parameters are only loaded at startup.

Configuration of the VFS

As you can read in other blogs on VFS we need to configure the input directory on which we listen to a new file:

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="SmooksSample"
       startOnLoad="true"
       statistics="disable"
       trace="disable"
       transports="vfs">
   <target>
      <inSequence>
         <smooks config-key="smooks-key">
            <input type="text"/>
            <output type="xml"/>
         </smooks>
      </inSequence>
   </target>

   <parameter name="transport.PollInterval">5</parameter>
   <parameter name="transport.vfs.FileURI">file:///C:\EI\wso2ei-6.2.0\smooks\in</parameter>

<parameter name="transport.vfs.ContentType">text/plain</parameter>
   <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
   <parameter name="transport.vfs.MoveAfterFailure">file:///C:\EI\wso2ei-6.2.0\smooks\fail</parameter>
   <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
   <parameter name="transport.vfs.FileNamePattern">.*.txt</parameter>
   <parameter name="transport.vfs.MoveAfterProcess">file:///C:\EI\wso2ei-6.2.0\smooks\original</parameter>
   <description/>
</proxy>

Smooks

The graphical overview is extremely simple, especially since parameters are not visible in the graphical mode but are in the source mode. Of course, you need to change the parameters in the proxy and local entry in order to match your setup.

Smooks

Smooks is an extensible framework for building applications for processing XML and non XML data (CSV, EDI etc). The Smooks mediator allows you to do that within the Enterprise Integrator. It can be used to perform a wide range of Data Transforms - XML to XML, CSV to XML, EDI to XML, XML to EDI, XML to CSV etc.

The highlevel syntax is as follows:

<smooks [config-key="string"]>
<input [type="|text|xml"]/>
<output [type="|text|xml|java"] [property="string"] [action="string"]/>
</smooks>

The config-key is used to access the Smooks configuration. The Smooks configuration should be saved in the Registry as a local entry before it can be used here. You can also refer to a file for the config-key. The Smooks configuration file uses a number of other technologies to make it work.

First of all, we see a number of xsd’s included like javabean, file-routing and freemarker. The data will be read one line at a time by the SAX parser.

In the EDI mapper the line is transformed according to the layout of the EDI message and subsequently put into a hashtable and using the freemarker template written to an xml file.

<smooks-resource-list
        xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
        xmlns:jb="http://www.milyn.org/xsd/smooks/javabean-1.2.xsd"
        xmlns:file="http://www.milyn.org/xsd/smooks/file-routing-1.1.xsd"
        xmlns:ftl="http://www.milyn.org/xsd/smooks/freemarker-1.1.xsd">
 
      <params>
              <param name="stream.filter.type">SAX</param>
              <param name="default.serialization.on">false</param>
       </params> 

       <!-- Configure the EDI Parser to parse the message stream into a stream of SAX events. -->
       <resource-config selector="org.xml.sax.driver">
              <resource>org.milyn.smooks.edi.EDIReader</resource>
              <param name="mapping-model">samples/service-bus/resources/smooks/edi-mapping.xml</param>
       </resource-config>

          <!-- Populate java objects to message data-->

       <jb:bean beanId="order" class="java.util.Hashtable" createOnElement="order">
              <jb:value property="id" decoder="Integer" data="order/id"/>
              <jb:value property="price" decoder="Double" data="order/price"/>
              <jb:value property="quantity" decoder="Integer" data="order/quantity"/>
              <jb:value property="symbol" data="order/symbol"/>
       </jb:bean>

 
      <!-- Apply freemarker templating -->
       <ftl:freemarker applyOnElement="order">
              <ftl:template><!--<order id="${order.id}">
       <quantity>${order.quantity}</quantity>

        <price>${order.price}</price>
       <product-symbol>${order.symbol}</product-symbol>
       </order>--></ftl:template>
              <ftl:use>
                      <!-- Output the templating result to the
"orderItemSplitStream" file output stream... -->
                      <ftl:outputTo outputStreamResource="orderItemSplitStream"/>
              </ftl:use>
       </ftl:freemarker> 
       <!-- Create/open a file output stream.  This is writen to by the freemarker template (above).. -->
        <file:outputStream resourceName="orderItemSplitStream" openOnElement="order">

<file:fileNamePattern>order-${order.id}.xml</file:fileNamePattern>
                <file:destinationDirectoryPattern>C:\EI\wso2ei-6.1.1\smooks\orders</file:destinationDirectoryPattern>
                <file:highWaterMark mark="100000"/>
        </file:outputStream>

 </smooks-resource-list>

Setup

We have adapted this blog from the sample 659 included with the Enterprise Integrator. Setup an EI-6.2.0 on your machine. Do it close to the root when working on Windows due to the PATH length restriction.

I made a setup where the proxy will listen on a directory called C:\EI\wos2ei-6.2.0\smooks\ with in, original, fail and orders directories. As you can see, this is a Windows setup. So you will need to change the settings to match your setup. At the Yenlo Bitbucket you can find all resources needed. I’ve also changed the location of the smooks-config.659 in the Bitbucket Sample to be in the smooks subdirectoru

Open the proxy and change to the directory locations applicable to you.

  • transports.vfs.FileURI- The input file location.
  • transports.vfs.MoveAfterProcess -  The location to move the input file after it is processed.
  • transports.vfs.MoveAfterFailure-  The location to move the input file if there happens to be a failure at the time of processing.

Open the <[EI_HOME]/repository/sample/smooks-config-659.xml and the proxy (see below) and change to the directory locations applicable to you. It is the <file:destinationDirectoryPattern> you need to change.

The proxy uses the VFS system to listen to a directory and will take the file, parse it and write the file to the directory orders. The Name is assembled from the EDI file and the order number.

<order id="1">
       <quantity>500</quantity>
       <price>50</price>
       <product-symbol>IBM</product-symbol>
</order>

So what can we do with this?

The setup we have created can be used for other EDI messages by changing the EDI mapping to support other layouts. You now know that an EDI message can be read by the Enterprise Integrator with minimal changes. Only the VFS needs to be enabled and a plain text message builder added to the axis2.xml configuration.

We can extend this example by taking the orders and write them to a database, JMS queue or something similar. Please let us know if you interested to hear or read more about that.New Call-to-action

 

Care to share?
   
Picture of Rob Blaauboer
Published June 21, 2018

Rob Blaauboer

Rob is a Senior Business Consultant and Solution Architect with more than twenty years experience. In addition to his work he is an active blogger working on a number of articles on the ‘Internet of Things’ and a WSO2 ‘Getting Started with …’ series (WSO2 tutorial) in which he talks about WSO2 components and their purpose especially aimed at non technical readers. Rob is a WSO2 expert and official WSO2 trainer.

Responses

Stay up to date with the latest articles