• Wouter Van Hecke

  • Software Engineer

How to create an app that fully automates the setup of a test environment, so that testers, too, can quickly and easily set up a test environment? At Foreach, we have been looking for a solution for this for years, and during our last Ship It edition four of our developers finally developed a solution.

The solution: Neos

Our solution is comprised of quite a few existing, open source components that we have bundled together in our own custom-made Java application. This application is the heart of our setup: It fully manages the deployment flow. We call it “Neos”, which is Greek for “new” (at Foreach it is tradition to give all internal applications a Greek name).

At its core, our solution is Docker-based. Docker is an open source project that allows you to package your application inside a container. This means that everything that can be put in a Docker image - which should be possible for pretty much all the applications we have here at Foreach - can be automatically deployed using Neos.

Neos

Besides Docker, the following components are being used:

    Bamboo: Our continuous delivery platform of choice;
    Docker Registry: The place where we store and distribute our Docker images. See it as a library for Docker images, the functionality of which is comparable to Maven repositories;
    Rancher: A Docker orchestrator packed with features that makes it easy to deploy and manage containerized applications on any infrastructure;
    Docker hosts: Various Linux servers that are used as Docker hosts, where we can deploy these containers;
    Load balancers: These run on some of the Docker hosts and their function is to balance the incoming requests to the correct Docker host;
    Last but certainly not least, Neos. This is our own custom-made application that bundles all of the above components and offers an easy-to-use interface that automates and manages our complete deployment flow.

How Neos works

As you can see in the above screenshot, Neos features an easy-to-use screen where the user can select the right customer, the project involved, and the Bitbucket branch he wants to deploy. He can also choose on which URL the application will be deployed. After he clicks “Deploy!”, Neos starts deploying the application and tells the user that the application is ready - and this within a minute. But how does this work behind the scenes?

Building the Docker container

The first step happens automatically after one of our developers pushes new code to our Bitbucket repository. Bamboo detects this, and “builds” the project. That includes the running of all our unit tests and the building of the Docker image. When finished, Bamboo pushes the new image to our Docker Registry. As soon as the image is in our Registry, it is possible to deploy it.

Step 1: The interface

If someone wants to test the new code, he goes to the Neos web interface. There he can select the customer, project and branch he wants to deploy. For this, we call the web service of the Docker Registry to see which projects are available:

GET /v2/_catalog
   "repositories":[ 
      "foreach/neos",
      "foreach/theta"
   ]
}
 
GET /v2/foreach/theta/tags/list
   "name":"foreach/theta",
   "tags":[ 
      "latest",
      "shipitday-demo"
   ]
}

The web service calls mentioned above tell Neos which projects are available for deployment. We do assume that the first part of the repository name is the customer, and the second part is the project (example: “repository foreach/theta” is seen as “customer foreach, project theta”). The list of available tags corresponds with the branches in our Bitbucket repository.

Step 2: Creating a new service

Next, we do a service call to Rancher in order to create a new service (a service consists of one or more docker containers based on the same image):

POST /v2-beta/projects/<ENVIRONMENT_ID>/services/
{
    "name""foreach-theta-t6crzu9FpLTactfvHTQXUTYwgMBRGVzF",
    "stackId""1st6",
    "startOnCreate"true,
    "launchConfig": {
        "count"1,
        "imageUuid""docker:registry.cloud.be/foreach/theta:shipitday-demo",
        "networdMode""managed",
        "healtCheck": {
            "port""8080",
            "requestLine""GET \"/healthcheck\" \"HTTP/1.0\"",
            "type""instanceHealthCheck",
            "healthyThreshold"2,
            "initializingTimeout"60000,
            "interval"2000,
            "reinitializingTimeout"60000,
            "responseTimeout"2000,
            "strategy""recreate",
            "unhealthyThreshold"3
        }
    }
}

Basically this is where we are telling Rancher to take a certain Docker image, namely docker:registry.cloud.be/foreach/theta:shipitday-demo, and deploy it.

Some of the parameters we supply are:

    stackId: This is the identifier of the stack in Rancher we are deploying this service to. A stack is a "group" of various Docker containers that belong together in some way (in our case, we bundle all containers of the same customer under the same stack);
    launchConfig.count: Launch 1 instance of this container. For test environments this should suffice easily, but if you decide to use this in production, specifying a higher value will give you high-availability;
    launchConfig.imageUuid: The Docker image you want to deploy.
    launchConfig.networkMode: "managed" tells Rancher to manage the networking. The container will receive an IP in its private network. Other available network modes include host, bridge and none (more information can be found here);
    launchConfig.healthCheck: Here we specify which path Rancher can use to healthcheck the application and to test whether the application has been successfully deployed, or whether it is still running. This healthcheck url as well as the port the application runs on can be defined for every application in a configuration interface provided by Neos.

This service call will return a serviceId property. This one is very important since we will use it in the next steps.

Step 3: Configuring the load balancers

Next up, we need to tell the load balancers that a new application has been deployed and that it should be available under a certain URL. For this, we do the following service call to Rancher:

PUT /v2-beta/projects/<ENVIRONMENT_ID>/loadbalancerservices/<LOAD_BALANCER_ID>
{
    "lbConfig": {
        "defaultCertificateId""1c1",
        "portRules": [
            {
                "type""portRule",
                "hostname""foreach-theta-t6crzu9FpLTactfvHTQXUTYwgMBRGVzF.cloud.be",
                "protocol""http",
                "serviceId""1f16",
                "sourcePort"80,
                "targetPort"8080,
                "priority"100,
                "path"""
            }
        ]
    }
}

Here we tell the load balancers to route the request sent to foreach-theta-t6crzu9FpLTactfvHTQXUTYwgMBRGVzF.cloud.be on port 80 to the service with id 1f16 (obtained with the previous service call) on port 8080.

Step 4: Application health check

While Rancher deploys this container instantly, it may take a little bit of time for our actual application to start up. Since many of our projects are Spring Boot applications, a lot of them are really working within about 15 seconds. During this timeframe, Rancher is sending a request every 2 seconds to the health check URL that we defined in a configuration interface in Neos. As soon as Rancher receives a status code of 200 or 30X on this URL, Rancher assumes that the application is up and running.

While Rancher is doing this, Neos queries Rancher every 2 seconds whether the application is considered “healthy”. For this we do the following service call: GET /v2-beta/projects//services/1fs16 The response JSON will contain a field “healthState”. As soon as Rancher sees it as "healthy", we display the URL the application is hosted on.

Future features of Neos

While the solution we implemented here does solve all of the problems we initially had, we do see a lot of opportunities to improve Neos. For example, we haven’t provided a way yet to deal with the dependencies of our applications (databases, memcached, …). This might be solved by extending our configuration interface with dependency containers, and this for every application. Also, It would be nice if Neos provided an API, so that other applications could integrate with it. All in all, there is a lot of potential for new features in Neos. If you have any more ideas, do let us know.