How to Implement Blue-Green Deployment with Kubernetes
Introduction: Why blue and green deployment
For the past 8 to 10 months, I have been working with an exceptional individual named Farzam Mohammadi. Together, we have implemented Blue and Green deployment at our company using Azure Container Apps.
I have been in software development for over a decade, and one of the most challenging processes, in my experience, is releases.
When releasing, if you find a bug during deployment, you need to decide if rolling back is an option. If there is an outage, your users will lose access to your system during that time.
That's why we need a better way to release—one that allows your team to test the deployment with production data and infrastructure without affecting your users or infrastructure, and reduces downtime to near zero.
Blue and Green deployment provides these features and allows for better CI/CD interaction.
If you want to dive into the code first, by all means, go for it: blue-green-with-k8s.
Benefits:
Zero User Impact: Updates and enhancements are made in the Beta environment, ensuring that Live services remain uninterrupted and available to users throughout the deployment process.
Zero Downtime: User traffic is redirected to the new version immediately after promoting from Beta to Live. (This may need some adjustments based on your existing infrastructure and its operations.)
Multi-Release Flexibility: If a Beta deployment is found unfit for any reason, it can be re-deployed with a new version or completely removed without affecting any users in the Live environment.
Stable and Secure Releases: Separating the Beta environment from Live services allows for thorough and secure testing. This prevents any untested changes from affecting the Live environment, enhancing both stability and security.
Efficient Management: Our approach provides precise control over how and when changes are promoted from Beta to Live. This careful management ensures that only fully tested and approved updates impact end users.
Flexible Deployment Scheduling: The Beta environment allows deployments at any time, independent of the Live service schedule. This flexibility enables extensive testing and verification well before the live deployment date, ensuring thorough readiness and peace of mind.
Requirements
To follow this example, you need to be familiar with the following technologies:
Bash scripting
Kubernetes
Docker and Kubernetes over Docker
YAML
React (basic configuration)
.NET (basic configuration)
Implementation platforms:
Azure Container Apps
You can create a similar strategy with Azure Container Apps as you do with Kubernetes. If you are focusing on this infrastructure, I will be creating a tutorial for that soon. Keep in mind that both platforms work similarly, but with Kubernetes, you have more control over the infrastructure. With Azure Container Apps, Azure has more control, but Microsoft is adding more options every day to make Azure Container Apps more robust for your implementations. You can get a taste of that here: Blue-Green Deployment in Azure Container Apps.
K8s
K8s provides an excellent set of tools that let you create robust releases using containers as a foundation.
I know some people have similar blogs about blue-green deployments with K8s, but based on our experience, we want to share the challenges we faced. This way, you can easily switch your releases to a blue-green deployment if you already use K8s and benefit from our trial and error.
The next section describes the step-by-step implementation.
Step-By-Step Implementation:
We'll be deploying a suite of projects using Blue-Green (B/G) deployment. In our example, we will first deploy a stable version as our initial live deployment. Then, we'll make some small changes in a separate branch and deploy that as our Beta version. Once we see both versions in action, we'll promote our Beta to Live, simulating a real production deployment. In our example, the database will be shared between both live and beta services, while the frontend and backend services will be created new for each version.
We've prepared all the steps for you and consolidated the deployment into one script with a single command. First, I'll walk you through our services. Then, I'll show you our Kubernetes configuration files to explain what we're deploying. Finally, I'll guide you through the actual deployment process. This is where we'll examine our implementation of B/G and see firsthand on our local machine how easy it is to get started!
The database has two tables:
Greetings
Farewells
First, we'll deploy our stable version of the services, which retrieves and displays the first record it finds in the Greetings table.
Next, in a separate branch, we'll update our backend and frontend services to retrieve and display the first record from the Farewells table.
After deploying our feature branch, we'll see the results of both deployments. Then, we'll promote the feature branch, or Beta environment, to our Live environment, replacing the initial version that only displayed the first Greetings record.
Step 1: Creation of Dockerfiles
The starting point for this infrastructure is to containerize your apps. To do this, we create two Dockerfiles for our frontend and backend, and we use MariaDB, which is an open-source, containerized version:
Backend: Dockerfile
Frontend: Dockerfile
Step 1.1: Building images with build-images script
For MariaDB, since the image is already available on Docker Hub, we just need to include the image path in our Kubernetes configuration file, which you'll see in the next step. However, for our own Backend and Frontend services, we must prepare and build them ourselves.
To build our images, we've created an automated bash script: build-images. This script will be used in our deployment-manager bash script later on to create our images using random tags..
Step 2: Creation of Kubernetes config files
Our standard method is to first create default deployment files. These files allow us to test our non-Blue/Green deployments before creating a template for Blue/Green deployments.
Here are the default non-template deployment files::
Backend: backend.yaml
Frontend: frontend.yaml
Database: db.yaml
Step 3: Creation of Kubernetes config templates from config files
Since only one version of the database will be deployed, there's no need to create a database Kubernetes configuration template file. However, because we are deploying the backend and frontend dynamically, we need to use templates for their Kubernetes configuration files.
Backend: backend.yaml.template
Frontend: frontend.yaml.template
Step 4: Deployment of Live services (starting point)
Starting from this step, we'll use the deploy-manager bash script to automate our entire deployment process.
At this point, Docker should be running with the Kubernetes cluster enabled. You also need to be able to run a bash script. (You can use Git Bash or any other bash.)
Use the following command to deploy our Live services:
./deploy-manager.sh deploy stable
This is going to build our images and deploy three services:
DB
Frontend
Backend
now you get this message:
==============================
Current Deployment Status
==============================
NAME READY UP-TO-DATE AVAILABLE AGE
b-g-backend-stable 1/1 1 1 5s
b-g-frontend-stable 1/1 1 1 2s
b-g-mariadb 1/1 1 1 7s
--- Current Services ---
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
b-g-backend-service-stable NodePort 10.96.120.158 <none> 8080:32573/TCP 5s
b-g-frontend-service-stable NodePort 10.107.120.141 <none> 3000:31253/TCP 2s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5h54m
mariadb-service NodePort 10.107.41.133 <none> 3306:30306/TCP 7s
--- Access URLs ---
ℹ Stable Backend: http://localhost:32573
ℹ Stable Frontend: http://localhost:31253
ℹ MariaDB: http://localhost:30306
==============================
Deployment Script Completed
==============================
This indicates that the process was successful. The ports will be different each time because we select a random ports. If you access the frontend, you will see this:
Step 5: Deployment of beta service
Now, if you switch your branch to feat/farewell and execute:
./deploy-manager.sh deploy beta
you are going to get this:
==============================
Current Deployment Status
==============================
NAME READY UP-TO-DATE AVAILABLE AGE
b-g-backend-beta 1/1 1 1 4s
b-g-backend-stable 1/1 1 1 7m5s
b-g-frontend-beta 1/1 1 1 2s
b-g-frontend-stable 1/1 1 1 7m2s
b-g-mariadb 1/1 1 1 7m7s
--- Current Services ---
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
b-g-backend-service-beta NodePort 10.99.108.162 <none> 8080:30625/TCP 4s
b-g-backend-service-stable NodePort 10.96.120.158 <none> 8080:32573/TCP 7m5s
b-g-frontend-service-beta NodePort 10.102.122.89 <none> 3000:32712/TCP 2s
b-g-frontend-service-stable NodePort 10.107.120.141 <none> 3000:31253/TCP 7m2s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6h1m
mariadb-service NodePort 10.107.41.133 <none> 3306:30306/TCP 7m7s
--- Access URLs ---
ℹ Stable Backend: http://localhost:32573
ℹ Beta Backend: http://localhost:30625
ℹ Stable Frontend: http://localhost:31253
ℹ Beta Frontend: http://localhost:32712
ℹ MariaDB: http://localhost:30306
==============================
Deployment Script Completed
==============================
This indicates that the process was successful. If you call the frontend, you will get this:
Let me pause to explain what we have now. In our environment, we are running two versions of the same app. Each service connects to its own backend using the same database but different configuration files based on the deployment. This ensures that current users can continue using our app while our team starts testing the new deployment. Nice, right? It looks something like this:
Step 6: Promote beta into stable
After we are satisfied with the new version, we will want to promote it. At this point, we are not releasing a new version; we are just replacing the old version. We can do this by running the following command:
./deploy-manager.sh promote
Then you will see the following message:
==============================
Current Deployment Status
==============================
NAME READY UP-TO-DATE AVAILABLE AGE
b-g-backend-beta 1/1 1 1 10m
b-g-backend-stable 1/1 1 1 17m
b-g-frontend-beta 1/1 1 1 10m
b-g-frontend-stable 1/1 1 1 17m
b-g-mariadb 1/1 1 1 17m
--- Current Services ---
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
b-g-backend-service-beta NodePort 10.99.108.162 <none> 8080:30625/TCP 10m
b-g-backend-service-stable NodePort 10.96.120.158 <none> 8080:32573/TCP 17m
b-g-frontend-service-beta NodePort 10.102.122.89 <none> 3000:32712/TCP 10m
b-g-frontend-service-stable NodePort 10.107.120.141 <none> 3000:31253/TCP 17m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6h12m
mariadb-service NodePort 10.107.41.133 <none> 3306:30306/TCP 17m
--- Access URLs ---
ℹ Stable Backend: http://localhost:32573
ℹ Beta Backend: http://localhost:30625
ℹ Stable Frontend: http://localhost:31253
ℹ Beta Frontend: http://localhost:32712
ℹ MariaDB: http://localhost:30306
==============================
Deployment Script Completed
==============================
Remember, there are other ways to handle this, such as redirecting users from old nodes to new nodes or using different strategies. For now, we are simply replacing the images those nodes are running.
Your new nodes are now running the new images. You can keep your beta version for more testing or remove it using this command:
./deploy-manager.sh teardown beta
And if you want to clean up everything, you can use this command:
./deploy-manager.sh teardown all
Recap:
Let's recap what we did:
Created two images: frontend and backend.
Used those images to create our first environment with a basic database.
Deployed a new version with a new tag into the same environment.
Overrode the initial deployment with the tested new image version.
These steps allow us to have a Blue-Green deployment, keeping users with near-zero downtime and giving us the opportunity to test our environment before releasing it.
I enjoyed creating this for you, and I hope you can start using these strategies in your current or future projects.