Continuous Integration for WSO2 Artifacts

This blog will walk you through a set of best practices, guidelines, tips and tricks to setup an efficient software delivery lifecycle for WSO2 middleware platform. It explains how to apply generic software development best practices such as managing configurations, continuous integration, continuous deployment and build-once-deploy-everywhere concepts to WSO2 platform. The result is an efficient software delivery process for WSO2 platform that integrates into the organization software development toolset and lifecycle. 

WSO2 middleware platform has a story for comprehensive development, deployment and delivery. WSO2 platform ships with Developer Studio - an Eclipse based graphical editor to create artifacts such as services and integrations and manage links dependencies between the artifacts. When Developer Studio integrates with continuous integration and deployment it will provide a very comprehensive story.

Development Time Best Practices

All development efforts universally go through minimally two stages - dev and production environment. Ideally in a large organization, there are well established environments on which artifacts needs to be tested before going live. When WSO2 platform is introduced into such an environment it needs to plugin to the existing software development lifecycle tool set seamlessly. An existing software delivery process can be as follows.

WSO2 artifacts can plugin to the above CI seamlessly, provided that the your project is structure properly.

Organizing your Project

Applications/artifacts on WSO2 platform connect to systems and databases, which will change depending on the environment. Mostly, we have observed that the following vary between environments. We are going to call these “external references” in the rest of the article.

• Endpoints and credentials to external systems - In ESB artifacts
• Database connection details - ESB artifacts, Services, Web apps and Data Services

As these external references keep changing per-environment. How do we change values between environments? What if the code en-composes external references in the integration logic itself? This is a big NO. You’ll have to modify the deploy-able artifact at each stage, which is error prone and very primitive way of development. It is not at all deploy-friendly or CI/CD friendly. The key is to separate the main logic from external references, external references and configurations will keep changing based on the environment but the main logic will stay the same throughout.

Use Developer Studio for the development. Separate out main logic from environment specific configurations. You can follow the steps here [1]. A sample workspace that configured for the Dev/Test environments is as follows.

A separate java project has been added to write tests for the CRM integration project. Any technology that can send HTTP calls would serve the purpose, including Jmeter tests. The MainESBConfig project has the main logic of the project.

Setting up CI/CD for Environments

When a project workspace is organized to as above it can be configured in a CI system to do continuous build and deployment. The typical steps of a CI includes,

• Build the CRM integration project each time a commit happens
• Deploy the artifacts each time the build is successful
• Run tests against the Development environment
• Then periodically at a predefined time deploy the CRM integration project to the PreProd environment, if all the tests are successful

When all of the above actions are configured in a CI, it would create a build pipeline, that delivers software to Preprod stage. Let’s see how to configure that using Jenkins.

Step 1 - Configure CI System to build the CRM integration project each time a commit happens

Deploying artifacts can be handled by the maven-car-deployer plugin. It can handle the following artifacts successfully.

• ESB artifacts
• AS artifacts
• DSS artifacts
• DAS artifacts
• BPS artifacts

Create a CAR file per environment and add the maven-car deployer plugin to the pom.xml of each CAR fie. By default Developer Studio adds the maven-car-deployer plugin to every CAR project.

For example dev and test environment CAR files can be shown as below along with their CAR deployer plugin.

Now Configure the Dev CAR file to be deployed to the above dev environment using the CI system. First build the maven-multi-module project each time a developer commits.

Step 2 - Configure CI to deploy the artifacts each time the build is successful.

When deploying BPS artifacts such as BPEL processes it will delete all of the old instances. If the preferred behavior is, old instances using the old BPEL process and the newly initiated instances using the new BPEL process, then BPS artifacts needs to be deployed as a *.zip archive separately. It can be done using ant scp task inside a maven-antrun-plugin.

Step 3 - Configure CI to run the tests against the development environment

Configure the automated tests as a downstream build job of the development build job defined above. Each time the deployment happens, the tests are run against it.

Step 4 - Configure CI to periodically at a predefined time deploy the CRM integration project to the PreProd environment, if all the tests are successful

If the above build is stable then move the artifact to the PreProd environment by adding a periodic build that runs only if the CRMTest build job is stable. A build is stable only if all the tests are passing.

Conclusion

It is possible to setup a CI/CD for WSO2 platform artifacts very easily and create an efficient software delivery process with development best practices. In this way you can automate deployment to production as well, but the “revert back” procedure which must kick-in if something goes wrong must be planned along with it.