Enterprise SOA Governance with WSO2 Greg - Evaludate, Direct, Control

SOA governance is an extension to IT governance. IT governance ensures that enterprise IT is inline with business strategy, architectural best practices, government regulations, laws and contribute to the value addition process of the organization. SOA governance ensures this factors against the SOA efforts in the organization. Governance is best implemented in an organization in a 3 step procedure - evaluate, direct and control. This blog post will show how WSO2 Greg will assist governing bodies in each of the steps.

Evaluate - Distributed nature of SOA means dis-aggregation of service data by hosted location, ownership and domain, making it a challenge to evaluate SOA efforts. Primary task of a governance product is to build a centralized database of information of SOA. This will be the basis of SOA governance in industry. WSO2 Greg provides capabilities to build a asset store.WSO2 Greg ships with a default service asset, that can be modified or used as it is. Building a service catalog using the registry is easy. However if the service catalog is going to be manually populated it is hard to keep it in-sync with the alive services. Populating services can be automated in WSO2 Greg as it has a REST API. Any service hosting platform can publish service information by calling the WSo2 Greg APIs.

Direct - Guiding people and technology in SOA efforts require a lot of information. High level business decisions comes from business analytic/intelligence layer that provides direct insight into what is going on in the business. Once the business goals are foamed then comes the SOA based implementation strategy. Here Governance can play a significant role as it can provide information as,

• A service catalog with usage, interface and owner for each service
• Decide how much a service being reused
• Service usages and dependency graph
• Data to deduce service maturity - versions and creation times 
• Different versions and their life-cycle movements
• Social features such as service rating, comments

WSO2 governance registry is designed to provide all of the information,

Control - What needs to be controlled in a SOA effort? Some of the things that needs to be controlled are service replication, services with bad APIs, bad code, existence of unused services and etc ..

How much control is too much control is a good question to ask each control step the governance body is going to enforce. Whatever the control step that put in place to hinder the innovation of developer and devops is too much control. If a developer is not allowed to create a service without approval that would hinder innovation. However Lifecycle management is an ideal place for enforcing control. It would not hinder innovation and still ensures the quality of the SOA architecture. For this purpose WSO2 Greg provides and extensible, customizable lifecycle management feature in Greg.

Continuous Integration for WSO2 Artifacts

This blog will walk you through a set of best practices, guidelines, tips and tricks to setup an efficient software delivery lifecycle for WSO2 middleware platform. It explains how to apply generic software development best practices such as managing configurations, continuous integration, continuous deployment and build-once-deploy-everywhere concepts to WSO2 platform. The result is an efficient software delivery process for WSO2 platform that integrates into the organization software development toolset and lifecycle. 

WSO2 middleware platform has a story for comprehensive development, deployment and delivery. WSO2 platform ships with Developer Studio - an Eclipse based graphical editor to create artifacts such as services and integrations and manage links dependencies between the artifacts. When Developer Studio integrates with continuous integration and deployment it will provide a very comprehensive story.

Development Time Best Practices

All development efforts universally go through minimally two stages - dev and production environment. Ideally in a large organization, there are well established environments on which artifacts needs to be tested before going live. When WSO2 platform is introduced into such an environment it needs to plugin to the existing software development lifecycle tool set seamlessly. An existing software delivery process can be as follows.

WSO2 artifacts can plugin to the above CI seamlessly, provided that the your project is structure properly.

Organizing your Project

Applications/artifacts on WSO2 platform connect to systems and databases, which will change depending on the environment. Mostly, we have observed that the following vary between environments. We are going to call these “external references” in the rest of the article.

• Endpoints and credentials to external systems - In ESB artifacts
• Database connection details - ESB artifacts, Services, Web apps and Data Services

As these external references keep changing per-environment. How do we change values between environments? What if the code en-composes external references in the integration logic itself? This is a big NO. You’ll have to modify the deploy-able artifact at each stage, which is error prone and very primitive way of development. It is not at all deploy-friendly or CI/CD friendly. The key is to separate the main logic from external references, external references and configurations will keep changing based on the environment but the main logic will stay the same throughout.

Use Developer Studio for the development. Separate out main logic from environment specific configurations. You can follow the steps here [1]. A sample workspace that configured for the Dev/Test environments is as follows.

A separate java project has been added to write tests for the CRM integration project. Any technology that can send HTTP calls would serve the purpose, including Jmeter tests. The MainESBConfig project has the main logic of the project.

Setting up CI/CD for Environments

When a project workspace is organized to as above it can be configured in a CI system to do continuous build and deployment. The typical steps of a CI includes,

• Build the CRM integration project each time a commit happens
• Deploy the artifacts each time the build is successful
• Run tests against the Development environment
• Then periodically at a predefined time deploy the CRM integration project to the PreProd environment, if all the tests are successful

When all of the above actions are configured in a CI, it would create a build pipeline, that delivers software to Preprod stage. Let’s see how to configure that using Jenkins.

Step 1 - Configure CI System to build the CRM integration project each time a commit happens

Deploying artifacts can be handled by the maven-car-deployer plugin. It can handle the following artifacts successfully.

• ESB artifacts
• AS artifacts
• DSS artifacts
• DAS artifacts
• BPS artifacts

Create a CAR file per environment and add the maven-car deployer plugin to the pom.xml of each CAR fie. By default Developer Studio adds the maven-car-deployer plugin to every CAR project.

For example dev and test environment CAR files can be shown as below along with their CAR deployer plugin.

Now Configure the Dev CAR file to be deployed to the above dev environment using the CI system. First build the maven-multi-module project each time a developer commits.

Step 2 - Configure CI to deploy the artifacts each time the build is successful.

When deploying BPS artifacts such as BPEL processes it will delete all of the old instances. If the preferred behavior is, old instances using the old BPEL process and the newly initiated instances using the new BPEL process, then BPS artifacts needs to be deployed as a *.zip archive separately. It can be done using ant scp task inside a maven-antrun-plugin.

Step 3 - Configure CI to run the tests against the development environment

Configure the automated tests as a downstream build job of the development build job defined above. Each time the deployment happens, the tests are run against it.

Step 4 - Configure CI to periodically at a predefined time deploy the CRM integration project to the PreProd environment, if all the tests are successful

If the above build is stable then move the artifact to the PreProd environment by adding a periodic build that runs only if the CRMTest build job is stable. A build is stable only if all the tests are passing.

Conclusion

It is possible to setup a CI/CD for WSO2 platform artifacts very easily and create an efficient software delivery process with development best practices. In this way you can automate deployment to production as well, but the “revert back” procedure which must kick-in if something goes wrong must be planned along with it.