WSO2Torial: Building WSO2 products from source

Introduction

WSO2 (Web Services O²) is open source. This means that it provides the source code for all its products. This allows the open source community to inspect the product in detail, identifying and fixing potential issues. Another added benefit of open source software is that developers (and users) can customize and build their own version of the applications.

Lastly and probably most importantly, having access to the original source code (not a decompiled versioni) allows advanced developers to connect a debugger to a running product and easily troubleshoot problems (bugs, configuration errors, etc…)

In this blog we explain in detail how a user/developer can build their own version of WSO2 products (in this case the Identity Server 5.0.0) from the original source code available online.

Required tools & Setting up your environment

The online documentation is quite specific about the requirements for building WSO2 products from source and how to setup your development/build environment.

If you wish you can follow the online instructions. It is not however not necessary to setup a full IDE if you only want to build the source code. It is possible to complete the build process using only command line tools.

For OS it is advisable to use a unix based OS (such as Linux or MacOS). This is due the deep file paths that are generated in JAVA projects. Windows has a file path length limitation that might prevent some files from being created. This restriction does not exist on unix filesystems.

The source code is stored in subversion repositories (https://subversion.apache.org/), to access it a subversion client should be present on the system. For this article we used subversion (packages for different OS’es available at https://subversion.apache.org/packages.html). This tool is installable on most Linux systems with the command:

sudo apt-get install subversion

Secondly a build tool is necessary. In WSO2’s case this will be Maven since the project already contains a build.xml description file:

sudo apt-get install maven

Some system environment properties for Maven need to be set in order for the build to correctly run. These variables increase the amount of memory allocated to the build process:

MAVEN_OPTS=”-Xms1024m -Xmx4096m -XX:MaxPermSize=1024m”

Lastly it is required to have the Java Development Kit installed on the machine. For this version of the Identity Server (5.0.0), both version 6 and version 7 are required. Two versions are required because the source code is written for JDK 6. in some steps of the build process maven plugins are used that depend on JDK 7. It is therefore also advisable to have a easy and simple way of swapping between JDKs:

sudo update-alternatives –config java

This command displays a list of the available alternatives for Java. Specifically it shows to the user all the installed Java JDKs/JREs and allows the user to specify which one will be used by default. In effect this changes the JAVA version used by the build tools in the build process.

Available resources

WSO2 provides very extensive and detailed instructions on how to build the products on its website (https://docs.wso2.com/display/IS500/Building+from+Source). These instructions are applicable for products based on the Carbon Platform version 4.2.0

One important reference that must not be ignored/neglected is the release matrix (https://wso2.com/products/carbon/release-matrix/). WSO2 products are not all released simultaneously. WSO2 releases its products in chunks, with each chunk containing specific versions of specific products. To build any of these products/versions it is necessary to identity the chunk which contains the appropriate source code. This product/version to chunk association is mapped in the release matrix.

To use the release matrix, the user must search for the desired product & version in the table. From there the platform name (Turing for carbon 4.2.0) and the chunk number can be identified along with the required patch (for the carbon platform).

For our case (building the WSO2 Identity Server 5.0.0) this means we need the Turing platform, chunk 11 and Patch0008. Patches are incremental, in practice this means that we need all patches from Patch0001 through Patch0008.

Project structure

To build the WSO2 products it is necessary to understand the project structure. All WSO2 products are based on a common layer/platform called Carbon. It is therefore essential to obtain and build this layer before attempting to build the desired product itself.

The build process can be divided into three distinct steps:

1. Build the carbon platform
2. Build the patches for the carbon platform
3. Build the desired product

The goal of steps #1 and #2 is to generate the required JAVA artifacts (JAR files) used as dependencies by the product (#3). Upon completion of the first two steps, the user will have on his local artifact repository (configured through Maven) updated and patched versions of the carbon platform.

Without performing the first two steps the product cannot be built as the build tool will attempt to retrieve the dependencies from the local repository and when it does not find them it will attempt to retrieve them from the WSO2 central repository. These will most likely not be the correct version required.

This Maven mechanism is also used during step #2, and for this reason the patches need to be built incrementally (starting with the lowest number patch and building all the patches up until the one specified by the release matrix: patch0001 through patch0008 in this example).

Checking out the source code

1. Checking out the carbon source & patches

The first step in the build process is to check out the source for the carbon platform and the associated patches. This is done with the subversion client. The specific command is indicated on the wso2 building from source documentation:

svn checkout https://svn.wso2.org/repos/wso2/carbon/kernel/branches/4.2.0/ <platform-folder>

! Note that we checkout the platform source code into a separate folder from the product source later.

! The carbon platform repository is available via regular https at Carbon 4.2.0 SVN Repository should you want to manually explore its contents.

2. Checking out the product source

Once the carbon platform and patches are successfully built the next step is to proceed with building the product itself. We start by checking out the appropriate source code referring to the Release Matrix we know that we require chunk 11 of the Turing platform. The checkout command is then:

svn checkout https://svn.wso2.org/repos/wso2/carbon/platform/tags/turing-chunk11/ <product-folder>

Just as with the carbon platform, the source code for the product/chunk is also available via regular HTTP at https://svn.wso2.org/repos/wso2/carbon/platform/tags/turing-chunk11/.

! There is an issue with the SVN repository for the product. It is possible that the checkout ends abruptly with an error:

svn: E120104: ra_serf: An error occurred during decompression

If this happens just restart the process by running the svn checkout command again.

Building the product

3. Building the carbon platform

The following command should be executed on the <platform-folder> where the source code was checkout into:

mvn clean install

This command will build the base version of the carbon platform. Additionally the flag -Dmaven.test.skip=true may be specified to skip running the tests (allowing for a faster build).

! Keep in mind that the carbon platform must be used with JDK 6.

After building the platform, the required patches should be built. Each one must be built individually in an incremental order starting from patch0001 up to patch0008 (the one mentioned in the Release Matrix). To do this just navigate into the appropriate folder <platform-folder>/patches/patch000x and run the build command (mentioned above).

! Patch0003 makes use of a special maven plugin (maven-felix-bundle) which requires JDK 7. To build this specific patch the JAVA alternative should be (temporarily!) set to JDK 7. Once patch0003 is built the JAVA alternative should be set back to JDK 6.

4. Building the product

Running the maven build command on the <product-folder> will now build the product using the updated artifacts for the carbon platform that are stored on the local maven repository. Once the build completes we should have the following message:

Result

The end result of building the product from source consists of three zip files and can be found on the /products/is/5.0.0/modules/distribution/target folder:

• Distributable product (wso2is-5.0.0.zip)
• Documentation for the product (wso2is-5.0.0-docs.zip)
• Source code for samples and test cases

Current state / New products

Products based on the Turing platform (Carbon version 4.2.0) were the last ones to use the Subversion based repository. From Wilkes (4.4.0) onwards all products (including the carbon platform) are available through GitHub (https://github.com/wso2). This allows for easier collaboration via Git (pull requests, branches, version tags).

Once again, with the help of the Release matrix it is possible to identify the version of the carbon kernel (https://github.com/wso2/carbon-kernel) required by a specific product and obtain the source. In Github this is made easier because every release is tagged with the complete version number (Major.minor.revision). To obtain for example the source code for the carbon kernel version 4.4.11 a developer should clone the repository and checkout the commit tagged 4.4.11.

The same applies for the products themselves. It must be noted that every product has its own repository. These repositories always follow the pattern “product-<product name>”. So for example the Identity Server is in the https://github.com/wso2/product-is repository. Here the same rule for versions and tags applies.

In all, compiling WSO2 is quite possible but will require considerable time and depending on your experience and the issues can be frustrating.

It does however create a compiled version that upon comparison is equal to the version WSO2 offers for download.

If you have any questions about this blogpost contact us via the comments section of this blog. View also our WSO2 Tutorials, webinars or white papers for more technical information. Need support? We do deliver WSO2 Product Support, WSO2 Development Support, WSO2 Operational Support and WSO2 Training Programs.

Thanks to Rob Blaauboer for his contribution to this blogpost.