Using UrbanCode Deploy for Continuous Delivery of Dockerized Applications，urbancodedeploy

Using UrbanCode Deploy for Continuous Delivery of Dockerized Applications，urbancodedeploy

Note: This is original work of authorship. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author is strictly prohibited.

注：谢绝任何形式的转载。本文的中文版将UrbanCode Deploy用于Docker应用的持续交付在developerWorks中国发布。

As one of the most popular cloud computing technologies, Docker uses “container” thinking in the software “ship”. Many people believe that Docker will radically change the way people build, deploy and deliver software. This article will discuss how to use UrbanCode Deploy (UCD) to establish a basic DevOps delivery pipeline for the dockerized application, to promote the application through a series of environments, including different testing environments, and production environments. UCD allows you to centrally manage the multiple sets of development and test environments for the dockerized application. It also provides an out of the box plug-in for Docker, to support the Docker image import from different registries automatically, and to run the Docker images and then execute the required test suites.

Docker and the Docker Application

Docker comes from an open source project established in 2013. In a short period of two years, there have been over 100,000,000 downloads of the Docker images. As one of the most popular cloud computing technologies, Docker uses “container” thinking in the software “ship”. Considered as the international container written in code, Docker can package any application and related dependencies into a lightweight, portable, self-contained type of container. Technical people are very excited by the prospective application of Docker and the similar container technologies. Many people believe that container technology will radically change the way people build, deploy and deliver software.

Introduction of Docker

Docker is an open platform for developers and systems administrators to build, ship, and run distributed applications. Docker enables apps to be quickly assembled from components and eliminates the friction between the development, QA, and production environments. As a result, IT can ship faster and run the same app, unchanged, on laptops, data center VMs, and any cloud.
We can take a look at what characteristics make Docker so popular:

• Development efficiency
  With Docker, developers can build any application in any language, using any tool chain. It fully supports continuous integration and continuous delivery, from the developer laptops, through the automated test, to the production environment. The application does not require any modification to enable the scaling.
• Flexible and portable
  Docker supports moving the same application across multiple clouds. Docker eliminates the “lock-in” of one specific cloud, and becomes a cloud broker for hybrid cloud architectures. Docker helps IT engineers deploy and run any app, on any infrastructure, quickly and reliably.
• Deployment efficiency
  Container technology brings higher density of compute resources, including CPU, memory and storage. The Docker Engine’s lightweight runtime enables rapid scale-up and scale-down in response to changes in demand.

Dockerized J2EE Application

In this article, we will use a typical J2EE application named JKE Banking (JKE) as an example, to illustrate how to do the continuous delivery of a dockerized application. This application consists of two basic deployable components, the web/business component (jke-web) and the data component (jke-db), running on the jetty application server and the MySQL database.

Note: The JKE Docker application example code can be obtained from the git repository at https://github.com/wanyuxiang/jke-docker-sample

First, we’ll package the source code of the two JKE components into two Docker images, and then show how to use the Docker images with the binaries to deploy a JKE application environment. In the whole process, we will use two public Docker images from Docker Hub, java and mysql.

For the web component jke-web, we first package the application binary with a jetty server into a zip package, and write the jke-web component Dockerfile as follows:

The jke-web Dockerfile:

FROM java:7
# Install jke.zip
COPY jke.zip /opt/jke/
RUN unzip /opt/jke/jke*.zip
RUN Chmod u+x *.sh
CMD./jke.server.linux.sh restart -F and tail /var/log/faillog

Run the build command to generate a new jke-web image

sudo docker build -t jke-web

For the jke-db data component, we package the database initialize script into the MySQL based image.

The jke-db Dockerfile:

FROM MySQL
# Run SQL
COPY / init.sql
CMD MySQL - hmysql-server -uroot - ppassword init.sql

Run the build command to generate a new jke-db image

sudo docker build -t jke-db

To run the JKE application, we need to execute commands to start the three containers in order. The first container runs a MySQL server. The second container connects to the MySql server and runs the database initialize script. The third one runs JKE application as a daemon, and connects to the MySql server (In the database configuration file of the jke-web component, JKEDB.properties, the hostname has been set to mysql-server).

sudo docker run --name MySQL -e MYSQL_ROOT_PASSWORD=password -d MySQL
sudo docker run --link mysql:mysql-server - D jke-db
sudo docker run --link mysql:mysql-server - P 8080:8080 -d jke-web

Now the JKE applications can be accessed at http://<docker-hostserver>: 8080

Setup the Private Docker Registry

We need to setup a private Docker registry, in order to store the different versions of application components that are Docker images. This will allow developers and testers to upload and download the application component images, quickly deploy the applications into the testing environments, and mark those images with tags.
In the new version of Docker (1.6 and above), it is quite simple to setup a private registry with a one line command.

sudo docker run -d -p 5000:5000 registry

Now we will tag the jke component images with the version number and upload them to the private registry.

sudo docker tag jke-web:1.1 192.168.27.100:5000/jke-web:1.1
sudo docker push 192.168.27.100:5000/jke-web:1.1
sudo docker tag jke-db:1.1 192.168.27.100:5000/jke-db:1.1
sudo docker push 192.168.27.100:5000/jke-db:1.1

Then we can access and download the images from the internet. For example, we can direct access to the following URL to view the images and version information.

http://192.168.27.100:5000/v1/search
http://192.168.27.100:5000/v1/repositories/jke-web/tags

Note: 192.168.27.100 is our private Docker registry server IP address.

Use UCD to Setup the DevOps Pipeline of a Dockerized Application

Now we will use IBM UrbanCode Deploy (UCD) to setup the application DevOps delivery pipeline, which consists of multiple sets of environments. The application deployment automation tool, UCD, provides an automation framework that allows developers and IT engineers use the same automation through the whole application delivery lifecycle. It provides a centralized deployment infrastructure to manage the development, test and production environments. It also provides a Docker plug-in, to pull the latest version of the Docker images periodically for different environments, run the Docker images, and then execute the required test suites.

The DevOps Pipeline of the Dockerized App

In the continuous delivery of a traditional application, we focus on the lifecycle of the application deployment package, including the package generation and the package promotion across the different environments, and then eventually to the production environment. In the continuous delivery of the dockerized application, we manage the lifecycle of the Docker images with application binary code, from image generation to the image ship/run between various environments. The difference between the traditional application delivery and the dockerized application delivery, is in the carrier and the new way we ship application changes. The general workflow is the same, with continuous management of the application changes and multiple environments, and with control of the continuous delivery of the application.
For the dockerized application, we can roughly divide the continuous delivery process into two stages: Build and Ship/Run. When we setup the DevOps pipeline for the dockerized application, we use build tools and UCD to complete the two stages of the work, as shown below:

• Build
  Building the Docker images is primarily done by the developers with the build tools, we can use UCB or Jenkins to write the automated build process of Docker images. The build process will be automatically triggered after the code is committed, which will add a new image version in the private registry.
• Ship and Run
  With the generated Docker images, testers and IT engineers can start to deploy the application. UCD manages pulling different versions of the Docker images, running the associated Docker container to deploy the application, testing the application in multiple environments as well as publishing the images into the public registry. UCD can not only manage the complex application deployment with multiple environments, but also set the environment gates for key environments (such as the production environment) for quality control purposes. It can also support manual approval processes, to ensure the proper supervision and governance of key operations or key environments.

Building the Docker Images

Regardless of if this is done with Jenkins or UCB, the build process of the two docker images always consists of three steps. The first step is to get the source, including the Dockerfile, from GitHub. The second step is to generate jke.zip and init.sql with scripts or commands, then copy these to the working directory (The github repository already contains the default zip file for jke-web and a SQL script for jke-db). The third step runs the docker build command to generate the images and upload them to the private registry.

• Step1: get source from GitHub

git clone https://github.com/wanyuxiang/jke-docker-sample

• Step2: simply use the default binary files in github

• Step3: image generated and uploaded to the private registry

sudo docker build -t jke-db.
sudo docker tag jke-db:1.1 192.168.27.100:5000/jke-db:1.1
sudo docker push 192.168.27.100:5000/jke-db:1.1
sudo docker build -t jke-web.
sudo docker tag jke-web:1.1 192.168.27.100:5000/jke-web:1.1
sudo docker push 192.168.27.100:5000/jke-web:1.1

After a successful build, the new jke-db and jke-web image versions can be found in the private Docker registry (in our case, it is IP address 192.168.27.100).

Introduction of UCD

UrbanCode Deploy (UCD) is the enterprise tool for application deployment automation. It takes applications as the center, provides version control for deployed artifacts and deployed configurations, management of deployment through multiple environments, graphic design of complex application deployment processes, and supports the use of role based cooperation operations - like approvals. In addition, it also provides more than one hundred of out of the box plug-ins, allowing it to integrate with different types of third party tools from different vendors.
In particular, UCD provides a Docker plugin, used to map the Docker images into application components. In the following chapters, we will discuss in detail how to use UCD to do the continuous delivery of the JKE application.

The Multiple Environment Management and DevOps Pipeline in UCD

First we create an application named “Dockerized JKE” for JKE Banking application in UCD. According to the pipeline chart, the JKE release in our DevOps pipeline will involve four environments: development environment, FVT (Function Verification Test) environment, SVT (System Verification Test) and internal production environment. We use the development environment for images built in the build phase. During the ship/run stage, UCD needs to cover three deployment environments. So in UCD, we have four environments, FVT, SVT, internal Prod, as well as an environment for the Docker image publishing in the public registry.
In practice, we can also use UCD to manage the development environment, using the same process to deploy and push the images. The only thing to note is the Docker image source configuration for the development environment in UCD. It can be another Docker private registry, or you can leave it blank to obtain images directly from the development environment. It this case, you only need to modify the configuration of the environment in UCD accordingly, but use the same processes.
From the picture below, we can have a quick overview of all the environments for the application “Dockerized JKE”. We can see all the environments included in the JKE DevOps pipeline, and the current versions and compliance status for each environment.

We set the environment gates for the different environments, to define the requirement that must be met before component versions can be deployed to an environment. For example, a component version needs to pass BVT (with “BVT Passed”) to enter the Functional Test environment. To deploy to the internal production environment, the component version must has statues “BVT Passed”, “FVT Passed” and also “SVT Passed”.

Docker Image Version Management in UCD

As we know, the JKE application requires an external DB Server, and consists of two components for deployment. In UCD, to map the 2 Docker images generated before, we create 3 application components: jke-web, jke-db and mysql.
It is very simple to create and configure the 3 dockerized application components. First of all we need to go to the UCD official website to download and install the Docker plugin. After installation, UCD will automatically generate a Docker component template, which contains the reusable deployment process.

When creating a component for Docker, we only need to select the Docker template, fill in the registry information and run options. If we leave the “Docker Registry” field as blank, UCD will search and pull the images from the public Docker Hub registry.

We set the mysql component to pull the images from the public Docker Hub. For component jke-web and jke-db, we need to fill the URL of the private docker registry in the “Docker Regitry” field to pull the application component from the private registry.

If the “Import Versions Automatically” option is checked, UCD will periodically monitor the Docker registry to import the new versions. In the component “Versions” tab, we can see all the imported versions of the component images in the registry. For one particular version, we can assign different statuses to track the version states. The version statuses can be added manually or automatically by the UCD processes. For example, after the test process on the SVT environment is completed successfully, UCD will add the status “SVT Passed” to the current version.

When the image versions are imported into UCD, the default statue “Existed” is added automatically.

Design the Deployment Processes in UCD

In UCD, we mainly have 2 levels of deployment process: the application process and the component process. The component process is composed of plug-in steps, which describe a specific sequence of deployment or configuration operations. The application process consists of several component processes, and defines the dependence between the components. In our case, we need to create two application processes for the JKE application:

• The Deploy and Test
  Deploy the JKE application and run automation suites
• The Tag and Push
  Tag the Docker image version and push to the Docker registry

First, we design the Deploy and Test application process, and the related component processes.
The Docker plugin already provides a component template with a “pull and run” process. This process allows you pull the specific image versions into a new environment, and run it with the options you filled in.
After the component images are deployed into the test environments, we need to run the automated tests to verify the deployment. UCD provides a variety of out of box test plug-ins, allowing different testing tools to be integrated with UCD. Here, we will use the simple web test plug-in to design a “test” process for the jke-web component. The plug-in provides a way via HTTP to quickly check the usability of web applications. We only need to drag this plugin step to the design interface, and fill in the web page URL and also the string to validate on the page.

After the test, we need to update the component version with status. So we create a new reusable process of “Set status to version” for the Docker component template, to mark status on the selected versions.

Then, we will design the “Deploy and Test” application process, to specify how the components are deployed, and set the component version status after deployment and test.

Next we’ll design the application process “Tag and Push”. We create another reusable process, “tag and push”, for the Docker component template by dragging and dropping the plugin steps as shown below, and filling in the corresponding parameters.

Finally, create the application process “Tag and Push” to publish the application component images. The 2 component push processes can be executed in parallel.

Before pushing the application Docker image into the public Docker registry, we will need to obtain management approval.
To do this kind of approval in UCD, firstly we need check “Require Approvals” option on the corresponding environment, and then design the approval process to assign the appropriate roles for environment or application approval.

After the option “Reqiure Approvals” is checked, there will be a new tab “Approve Process” to allow you design the approval process using drag and drop in a graphical interface.

Run the DevOps Deliver Processes for the Dockerized App

After developers upload the new application image versions to the private Docker registry with any Build tool, testers and IT engineers can rapidly deploy and test the application using UCD.
Here, we take the “System Integration Test” environment as an example, to show how to do quick application deployment and testing in UCD. The operation is very simple, just click the > button on the environment, select “Deploy and Test” process and the component versions (or snapshot) to be deployed. In the following picture, the snapshot of “SN1” actually contains the latest versions of the three JKE components.

The deploy and test process will take a few minutes to complete. During the runtime, or after the execution, we can always check the logs in UCD console to monitor the process or identify any errors. Once the process completes successfully, the latest version of the three components will get the “SVT Passed” label.

At some point, we will push the application images to the public registry. To start the tag and push application process, we choose the right process and the same snapshot “SN1” with required statuses.

Any approvers will receive an email notification for the approval. When they log into UCD, they can see that the “Tag and Push” process is waiting for a response to be made on the approval, before the real execution of the push application process begins.

After the process is done, we use the command to check if the Docker images have pushed successfully into the Docker Hub registry.

$ sudo docker search wanyux
[sudo] password for rational:
NAME             DESCRIPTION   STARS     OFFICIAL   AUTOMATED
wanyux/jke-web                 1
wanyux/jke-db                  1

Summary of Docker and UCD

Development efficiency, flexibility, portability and deployment efficiency lead more and more people begin to follow and adopt Docker technology. Docker provides a new way to deploy applications and middleware into the cloud. Use of a private registry to provide a more simple way of sharing multiple versions of the application, and the use of tags, allow the downstream consumer of the DevOps pipeline to easily find the latest “good” versions.
UCD provides an automation framework for the dockerized application to use same automation within the deployment lifecycle management setup. It also provides deployment artifact versioning and approvals in production environments. It makes the deployment of application changes fast and convenient, which can support continuous application delivery.
In combining Docker and UCD, the most popular technologies in DevOps, we expect to provide a new and more efficient way for applications to realize the benefits of continuous delivery. We also look forward to it bringing a better solution for application deployment in hybrid cloud environments.

Thank my manager Daniel Toczala for his great help on the proofread and accurate translation.