In this course, we will create a CI/CD Pipeline using the following tools, AWS with Terraform, NodeJS Simple API, Jenkins, Ansible, Docker, and DeletifyBot for deployment. You must have basic knowledge of these tools as well.
If you don’t know anything about DevOps then don’t worry. I am recording Jenkins & Terraform Full Course nowadays, and will be recording full courses of these tools listed above as well.
If you are new here, please subscribe to my YouTube Channel! https://bit.ly/3AGYyaZ.
All Lectures in one video with timestamps: https://bit.ly/2ZMwOVG
What do we cover?
What is DevOps?
DevOps is a set of cultural, practices, and tools, which improves the collaboration between developers and operations. It aims to shorten the SDLC process and provide continuous delivery with high software quality.
To learn more about DevOps visit.
What is CI/CD?
In CI or Continuous Integration, development work is divided into small portions for better and faster production. Engineers commit code in small chunks multiple times a day for it to be easily tested. If bugs and vulnerabilities are revealed, they are sent back to the engineering team.
CD or Continuous delivery is an extension of continuous integration since it deploys your changes on staging or production environment after the build stage is completed. This means that you can deploy your application at any time by clicking a button.
Most of the deployments happen manually because we require approval to push new stuff live and even on staging we deploy different branches multiple times a day.
Auto deployment right after the build stage is completed, is dead (R.I.P)… 2 Minutes silents, please!!!!
K.O! You ready???
AWS with Terraform
Create an AWS Account if you don’t have one.
Then, install awscli
package on your local system. I am using Ubuntu (Bionic 64) vagrant machine….
1
2
3
4
# to learn more, visit this website.
# https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html
sudo apt update
sudo apt install -y awscli
Now, let’s generate security credentials using AWS Management Console. Go to Services
-> IAM
-> Users
-> Your-User-Name
-> Security Credentials
-> Create Access Key
.
Once access keys are generated, copy and paste them into the following commands.
1
2
3
4
5
6
7
# to setup aws configuration for terraform
aws configure
# AWS Access Key ID [None]: <PASTE>
# AWS Secret Access Key [None]: <PASTE>
# to see, where are your credentials saved.
cat $HOME/.aws/credentials
Install terraform
on your local system.
1
2
3
4
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt update
sudo apt install terraform
Then, copy the following code and paste it into the main.tf
file.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
# set aws provider specific version
# https://www.terraform.io/docs/language/settings/index.html
terraform {
required_version = ">= 0.14"
required_providers {
aws = {
# https://registry.terraform.io/providers/hashicorp/aws/latest/docs
source = "hashicorp/aws"
version = "3.63.0"
}
}
}
# set aws as a provider
provider "aws" {
profile = "default"
region = "us-east-1"
}
# default vpc
# Virtual Private Cloud (VPC) is a virtual network that you define to secure and easy access to resources.
# https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/default_vpc
resource "aws_default_vpc" "this" {}
# security groups to enable ports
# Security Group acts as a virtual firewall for controlling traffic of instances.
# https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group
resource "aws_security_group" "this" {
vpc_id = aws_default_vpc.this.id
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
from_port = 8080
to_port = 8080
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
# Key Pair
# It is a combination of a public and private key that is used to encrypt or decrypt data.
# https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/key_pair
resource "aws_key_pair" "this" {
key_name = "my_demo_key"
public_key = file("my_demo_key.pub")
}
# EC2 (Elastic Compute Cloud)
# It is a virtual machine or a server that runs your applications in the cloud.
# https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance
resource "aws_instance" "this" {
count = "2" # instances: 1st for staging/live env and 2nd for jenkins pipelines.
ami = "ami-0747bdcabd34c712a" # Ubuntu (us-east-1)
instance_type = "t2.micro" # free tier
key_name = aws_key_pair.this.key_name # authorized the machine, so we can login later on.
vpc_security_group_ids = [aws_security_group.this.id]
}
# Output
output "public_ip" {
value = aws_instance.this.*.public_ip
}
Finally, To deploy your infrastructure on AWS, run the following commands.
1
2
3
4
ssh-keygen -f my_demo_key # create ssh key for terraforming.
terraform init # download the required provider or packages.
terraform apply -auto-approve # deploy your infrastructure and approve automatically.
terraform destroy -auto-approve # remove infrastructure from aws once you are done.
In case, you get a credentials error, then use AWS_PROFILE=default
as a prefix of Terraform. For example AWS_PROFILE=default terraform ...
.
Once your EC2 instances are ready try to login to a machine via SSH. Go to Services
-> EC2
> Instances
> INSTANCE_ID
> Connect to instance
> SSH client
> and get the username & IP.
1
2
3
# Keep one machine for cicd.
# and another for staging or stg.
ssh -i my_demo_key ubuntu@<YOUR_INSTANCE_IP_ADDRESS> # type yes, finally you are in.
Jenkins
Before installing Jenkins
, you need to install nodejs
, docker
, java
packages on cicd machine to complete CI/CD Pipeline Project.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# required for nodejs
curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
# install all pkgs in one command
sudo apt install -y nodejs docker.io openjdk-8-jdk
#### for Docker - required
sudo usermod -aG docker $USER # current user must have a group call docker.
sudo chown $USER:docker /var/run/docker.sock # docker.sock file must have current user's onwership.
sudo chmod 666 /var/run/docker.sock # change the permission too.
#### for JAVA - optional
# this is optional, but in case if you get a Java path error in Jenkins,
# then you can use this method to resolve the issue.
# find out java path using
# readlink -f /usr/bin/javac | sed "s:/bin/javac::"
# set JAVA_HOME path into .bashrc
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
PATH="$JAVA_HOME/bin:$PATH"
#### finally
node --version
docker --version
java -version
Install Jenkins
on cicd machine.
1
2
3
4
5
sudo apt install ca-certificates
wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
sudo sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt update
sudo apt install -y jenkins
- Open
PUBLIC_IP_ADDRESS:8080
into your browser. - You may need to get the password from
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
. - Setup Jenkins, click on the close button and finally start using Jenkins.
Login with docker first and then run the following commands to connect Jenkins with docker.
1
2
3
4
5
6
# login with docker first.
docker login -u "USERNAME" -p "PASSWORD" docker.io
sudo cp -r $HOME/.docker /var/lib/jenkins/.docker
sudo chown -R jenkins:jenkins /var/lib/jenkins/.docker
sudo usermod -aG docker jenkins # current user must have a group call docker.
Go to Manage Jenkins
> Plugin Manager
> Available
and install the following required plugins without restart.
- GitHub
- Slack Notification
Now, Run your first and simple job for the test.
- Click on
New Item
. - Enter the name
node-api
, choosefreestyle project
, and clickok
. - Choose
git
from Source Code Management (SCM) section and paste this GitHub public repository link (https://github.com/deletify/node-api.git), If your repository is private then you need to add GitHub credentials as well. - Choose
Poll SCM
and add cronjob* * * * *
into the input box, which means the Jenkins polls the SCM every minute. - Choose
Add build step
>Execute shell
and add the commands which you want to execute during the build process, use the following commands.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
npm install
npm test
username="deletify" # replace your username
v_tag=$JOB_NAME:$GIT_COMMIT # tag for version
l_tag=$JOB_NAME:latest # tag for latest
# build a tag locally
docker build -t $v_tag .
# make tag ready to push to docker registry.
docker tag $v_tag $username/$v_tag
docker tag $v_tag $username/$l_tag
# push both tags to docker registry.
docker push $username/$v_tag
docker push $username/$l_tag
# remove both tags cicd machine,
# because we don't need them anymore
docker rmi $v_tag $username/$v_tag $username/$l_tag
Finally, save the project, click on the Build Now
button and check the status under Build History
.
Ansible
Install Ansible
on cicd machine. To learn about Ansible visit.
Copy the following configuration and paste it to /etc/ansible/hosts
.
1
2
3
4
5
6
7
8
[servers]
stg ansible_host=<HOST>
# .....
[all:vars]
ansible_connection=ssh
ansible_ssh_extra_args='-o StrictHostKeyChecking=no -o IdentitiesOnly=yes'
ansible_python_interpreter=/usr/bin/python3
Copy the following code and paste it to deploy.yml
. The following command will pull, run, and remove docker images.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
- hosts: all
become: true
tasks:
- name: pull new docker image
shell: docker pull "{{org}}/{{app_name}}:{{tag}}"
- name: remove old docker container
shell: docker rm -f "{{app_name}}"
- name: remove old docker image
shell: docker rmi -f "{{org}}/{{app_name}}:{{tag}}"
- name: run new docker container
shell: docker run --name "{{app_name}}" -it -p "{{_port}}" -d "{{org}}/{{app_name}}:{{tag}}"
Install docker on stg machine first, otherwise, the following command will give you an error about docker.
Run ansible-playbook to pull docker image from docker registry and deploy it onto the stg env.
1
2
3
4
5
6
ansible-playbook --private-key my_demo_key -l stg -u ubuntu \
-e org=deletify \
-e _port=80:9008 \
-e app_name=node-api \
-e tag=latest \
deploy.yml
Now go back to cicd machine and notify engineers about Jenkins builds on Slack.
Slack
- Open this url https://my.slack.com/apps/A0F7VRFKN-jenkins-ci.
- Click on the
Add to Slack
button - Create a new channel where Jenkins notifications will be posted and select it from the dropdown once it’s created.
- Click on the
Add Jenkins CI Integration
button. - Copy the
Token
. - Goto
Manage Jenkins
>Configuration System
>Slack
> and configure it as mentioned in https://plugins.jenkins.io/slack/. - Goto
Dashboard
>choose your job
>configure
>Add post-build action
>Slack Notifications
> choosenotify success
andnotify every failure
. - Save it, build it, and you will see success or failure notification on your slack.
Finally, let’s work on deletify bot which you can use to deploy your changes on stg or live env.
DeletifyBot
- Clone this
https://github.com/deletify/slack-deletify-bot/
. - Watch my YouTube video to setup deletifybot with Slack. https://bit.ly/2ZMwOVG
1 2 3 4 5 6 7 8
# to install sudo apt install -y ruby ruby-bundler ruby-dev build-essential ruby-build # install bundle for deletifybot bundle install --path vendor/bundle # run deletifybot SLACK_API_TOKEN=YOUR_BOT_TOKEN bundle exec ruby deploy.rb
- Use this command to deploy your stuff on staging
deploy node-api master to stg
.