Cancel

DevOps Project CI/CD Pipeline Full Course

Yafiz Abraham on Nov 11, 20212021-11-11T00:00:00+00:00
10 min read
tech-world
linux, devops, jenkins

In this course, we will create a CI/CD Pipeline using the following tools, AWS with Terraform, NodeJS Simple API, Jenkins, Ansible, Docker, and DeletifyBot for deployment. You must have basic knowledge of these tools as well.

If you don’t know anything about DevOps then don’t worry. I am recording Jenkins & Terraform Full Course nowadays, and will be recording full courses of these tools listed above as well.

If you are new here, please subscribe to my YouTube Channel! https://bit.ly/3AGYyaZ.

All Lectures in one video with timestamps: https://bit.ly/2ZMwOVG

What do we cover?

visit

What is DevOps?

DevOps is a set of cultural, practices, and tools, which improves the collaboration between developers and operations. It aims to shorten the SDLC process and provide continuous delivery with high software quality.

To learn more about DevOps visit.

What is CI/CD?

In CI or Continuous Integration, development work is divided into small portions for better and faster production. Engineers commit code in small chunks multiple times a day for it to be easily tested. If bugs and vulnerabilities are revealed, they are sent back to the engineering team.

CD or Continuous delivery is an extension of continuous integration since it deploys your changes on staging or production environment after the build stage is completed. This means that you can deploy your application at any time by clicking a button.

Most of the deployments happen manually because we require approval to push new stuff live and even on staging we deploy different branches multiple times a day.

Auto deployment right after the build stage is completed, is dead (R.I.P)… 2 Minutes silents, please!!!!

K.O! You ready???

AWS with Terraform

Create an AWS Account if you don’t have one.

Then, install awscli package on your local system. I am using Ubuntu (Bionic 64) vagrant machine….

1
2
3
4

# to learn more, visit this website.
# https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html
sudo apt update
sudo apt install -y awscli

Now, let’s generate security credentials using AWS Management Console. Go to Services -> IAM -> Users -> Your-User-Name -> Security Credentials -> Create Access Key.

Once access keys are generated, copy and paste them into the following commands.

1
2
3
4
5
6
7

# to setup aws configuration for terraform
aws configure
# AWS Access Key ID [None]: <PASTE>
# AWS Secret Access Key [None]: <PASTE>

# to see, where are your credentials saved.
cat $HOME/.aws/credentials

Install terraform on your local system.

1
2
3
4

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt update
sudo apt install terraform

Then, copy the following code and paste it into the main.tf file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

# set aws provider specific version
# https://www.terraform.io/docs/language/settings/index.html
terraform {
  required_version = ">= 0.14"
  required_providers {
    aws = {
      # https://registry.terraform.io/providers/hashicorp/aws/latest/docs
      source  = "hashicorp/aws"
      version = "3.63.0"
    }
  }
}

# set aws as a provider
provider "aws" {
  profile = "default"
  region = "us-east-1"
}

# default vpc
# Virtual Private Cloud (VPC) is a virtual network that you define to secure and easy access to resources.
# https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/default_vpc
resource "aws_default_vpc" "this" {}

# security groups to enable ports
# Security Group acts as a virtual firewall for controlling traffic of instances.
# https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group
resource "aws_security_group" "this" {
  vpc_id  = aws_default_vpc.this.id

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 8080
    to_port     = 8080
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# Key Pair
# It is a combination of a public and private key that is used to encrypt or decrypt data.
# https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/key_pair
resource "aws_key_pair" "this" {
  key_name   = "my_demo_key"
  public_key = file("my_demo_key.pub")
}

# EC2 (Elastic Compute Cloud)
# It is a virtual machine or a server that runs your applications in the cloud.
# https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance
resource "aws_instance" "this" {
  count                  = "2" # instances: 1st for staging/live env and 2nd for jenkins pipelines.
  ami                    = "ami-0747bdcabd34c712a" # Ubuntu (us-east-1)
  instance_type          = "t2.micro" # free tier
  key_name               = aws_key_pair.this.key_name # authorized the machine, so we can login later on.
  vpc_security_group_ids = [aws_security_group.this.id]
}

# Output
output "public_ip" {
  value = aws_instance.this.*.public_ip
}

Finally, To deploy your infrastructure on AWS, run the following commands.

1
2
3
4

ssh-keygen -f my_demo_key # create ssh key for terraforming.
terraform init # download the required provider or packages.
terraform apply -auto-approve # deploy your infrastructure and approve automatically.
terraform destroy -auto-approve # remove infrastructure from aws once you are done.

In case, you get a credentials error, then use AWS_PROFILE=default as a prefix of Terraform. For example AWS_PROFILE=default terraform ....

Once your EC2 instances are ready try to login to a machine via SSH. Go to Services -> EC2 > Instances > INSTANCE_ID > Connect to instance > SSH client > and get the username & IP.

1
2
3

# Keep one machine for cicd.
# and another for staging or stg.
ssh -i my_demo_key ubuntu@<YOUR_INSTANCE_IP_ADDRESS> # type yes, finally you are in.

Jenkins

Before installing Jenkins, you need to install nodejs, docker, java packages on cicd machine to complete CI/CD Pipeline Project.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# required for nodejs
curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -

# install all pkgs in one command
sudo apt install -y nodejs docker.io openjdk-8-jdk

#### for Docker - required
sudo usermod -aG docker $USER # current user must have a group call docker.
sudo chown $USER:docker /var/run/docker.sock # docker.sock file must have current user's onwership.
sudo chmod 666 /var/run/docker.sock # change the permission too.

#### for JAVA - optional
# this is optional, but in case if you get a Java path error in Jenkins,
# then you can use this method to resolve the issue.
# find out java path using
# readlink -f /usr/bin/javac | sed "s:/bin/javac::"
# set JAVA_HOME path into .bashrc
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
PATH="$JAVA_HOME/bin:$PATH"

#### finally
node --version
docker --version
java -version

Install Jenkins on cicd machine.

1
2
3
4
5

sudo apt install ca-certificates
wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
sudo sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt update
sudo apt install -y jenkins
  • Open PUBLIC_IP_ADDRESS:8080 into your browser.
  • You may need to get the password from sudo cat /var/lib/jenkins/secrets/initialAdminPassword.
  • Setup Jenkins, click on the close button and finally start using Jenkins.

Login with docker first and then run the following commands to connect Jenkins with docker.

1
2
3
4
5
6

# login with docker first.
docker login -u "USERNAME" -p "PASSWORD" docker.io

sudo cp -r $HOME/.docker /var/lib/jenkins/.docker
sudo chown -R jenkins:jenkins /var/lib/jenkins/.docker
sudo usermod -aG docker jenkins # current user must have a group call docker.

Go to Manage Jenkins > Plugin Manager > Available and install the following required plugins without restart.

  • GitHub
  • Slack Notification

Now, Run your first and simple job for the test.

  1. Click on New Item.
  2. Enter the name node-api, choose freestyle project, and click ok.
  3. Choose git from Source Code Management (SCM) section and paste this GitHub public repository link (https://github.com/deletify/node-api.git), If your repository is private then you need to add GitHub credentials as well.
  4. Choose Poll SCM and add cronjob * * * * * into the input box, which means the Jenkins polls the SCM every minute.
  5. Choose Add build step > Execute shell and add the commands which you want to execute during the build process, use the following commands.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

npm install
npm test

username="deletify" # replace your username
v_tag=$JOB_NAME:$GIT_COMMIT # tag for version
l_tag=$JOB_NAME:latest # tag for latest

# build a tag locally
docker build -t $v_tag .

# make tag ready to push to docker registry.
docker tag $v_tag $username/$v_tag
docker tag $v_tag $username/$l_tag

# push both tags to docker registry.
docker push $username/$v_tag
docker push $username/$l_tag

# remove both tags cicd machine,
# because we don't need them anymore
docker rmi $v_tag $username/$v_tag $username/$l_tag

Finally, save the project, click on the Build Now button and check the status under Build History.

Ansible

Install Ansible on cicd machine. To learn about Ansible visit.

Copy the following configuration and paste it to /etc/ansible/hosts.

1
2
3
4
5
6
7
8

[servers]
stg ansible_host=<HOST>
# .....

[all:vars]
ansible_connection=ssh
ansible_ssh_extra_args='-o StrictHostKeyChecking=no -o IdentitiesOnly=yes'
ansible_python_interpreter=/usr/bin/python3

Copy the following code and paste it to deploy.yml. The following command will pull, run, and remove docker images.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

- hosts: all
  become: true
  tasks:
    - name: pull new docker image
      shell: docker pull "{{org}}/{{app_name}}:{{tag}}"

    - name: remove old docker container
      shell: docker rm -f "{{app_name}}"

    - name: remove old docker image
      shell: docker rmi -f "{{org}}/{{app_name}}:{{tag}}"

    - name: run new docker container
      shell: docker run --name "{{app_name}}" -it -p "{{_port}}" -d "{{org}}/{{app_name}}:{{tag}}"

Install docker on stg machine first, otherwise, the following command will give you an error about docker.

Run ansible-playbook to pull docker image from docker registry and deploy it onto the stg env.

1
2
3
4
5
6

ansible-playbook --private-key my_demo_key -l stg -u ubuntu \
  -e org=deletify \
  -e _port=80:9008 \
  -e app_name=node-api \
  -e tag=latest \
  deploy.yml

Now go back to cicd machine and notify engineers about Jenkins builds on Slack.

Slack

  1. Open this url https://my.slack.com/apps/A0F7VRFKN-jenkins-ci.
  2. Click on the Add to Slack button
  3. Create a new channel where Jenkins notifications will be posted and select it from the dropdown once it’s created.
  4. Click on the Add Jenkins CI Integration button.
  5. Copy the Token.
  6. Goto Manage Jenkins > Configuration System > Slack > and configure it as mentioned in https://plugins.jenkins.io/slack/.
  7. Goto Dashboard > choose your job > configure > Add post-build action > Slack Notifications > choose notify success and notify every failure.
  8. Save it, build it, and you will see success or failure notification on your slack.

Finally, let’s work on deletify bot which you can use to deploy your changes on stg or live env.

DeletifyBot

  1. Clone this https://github.com/deletify/slack-deletify-bot/.
  2. Watch my YouTube video to setup deletifybot with Slack. https://bit.ly/2ZMwOVG
    1
2
3
4
5
6
7
8

     # to install
 sudo apt install -y ruby ruby-bundler ruby-dev build-essential ruby-build

 # install bundle for deletifybot
 bundle install --path vendor/bundle

 # run deletifybot
 SLACK_API_TOKEN=YOUR_BOT_TOKEN bundle exec ruby deploy.rb
  3. Use this command to deploy your stuff on staging deploy node-api master to stg.
Recent Update
Contents

Cheatsheets for Bash Scripting

Setup and Manage Log Rotation in Linux