Setup jenkins server on AWS EC2 with slave agents

Posted on 23-Jan-2019

Create a security group for master server

In top menu, click on Services and then click on EC2 under Compute section

In the left menu sidebar, under NETWORK & SECURITY click on Security Groups

Here, click on the blue button Create Security Group

In Security group name type jenkins-master

In Security group rules click on Add Rule button.

settings:

Label	Value
Type	HTTP
Protocol	TCP
Port Range	80
Source	My IP

Add a new rule:

Label	Value
Type	Custom TCP
Protocol	TCP
Port Range	8080
Source	My IP

Add a new rule:

Label	Value
Type	SSH
Protocol	TCP
Port Range	22
Source	My IP

and click on Create

Create security group for agent server

First, copy to clipboard the group id for the security group that you have just created

Then Click on Create Security Group

In Security group name type jenkins-agent

Click on Add Rule

Label	Value
Type	Custom TCP
Protocol	TCP
Port Range	50000
Source	jenkins-master-group-id

Click on Add Rule

Label	Value
Type	SSH
Protocol	TCP
Port Range	22
Source	jenkins-master-group-id

Create Special user for jenkins

Open IAM dashboard, click on Users, click on Add user, add user name and then for Access type select Programmatic access and then click on Next: Permissions
Click on Create group and set a name for your group and then click on Create group in bottom right.
Make sure that the newly created group is selected and it has None for Attached policies
Click Next: Tags
Click Next: Review
Click Create user
Click on Download .csv to download your Access key ID and Secret access key
Click Close

Now, your user should not have any permissions now. Let's add them:

In the users list page, click on the username you just created
Click Add inline policy (is in right side)
Click JSON

Add the following JSON code

{
    "Version": "<here you can add your current date>",
    "Statement": [
        {
            "Sid": "Stmt1312295543082",
            "Action": [
                "ec2:DescribeSpotInstanceRequests",
                "ec2:CancelSpotInstanceRequests",
                "ec2:GetConsoleOutput",
                "ec2:RequestSpotInstances",
                "ec2:RunInstances",
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:TerminateInstances",
                "ec2:CreateTags",
                "ec2:DeleteTags",
                "ec2:DescribeInstances",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeRegions",
                "ec2:DescribeImages",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "iam:ListInstanceProfilesForRole",
                "iam:PassRole"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Click on Review policy
Add a name for your new policy
Click on Create policy

Create master instance

In AWS Management Console click on EC2 under Compute service

Click on Launch Instance. Here select Amazon Linux 2 AMI (HVM), SSD Volume Type

In the next window select at least t2.small and then click on Next: Configure Instance Details

You don't need to change anything in this window so click on Next: Add Storage

On Step 4, change the SSD size from 8GB to 10GB or more (Master jenkins will not run any code and it doesn't need so much space).

Next, click on Next: Add Tags. Here click on Add Tag and for Key type Name and for Value type jenkins master.

Then click on Next: Configure Security Group

Here, on Assign a security group click on the second option: Select an existing security group and then select the group named jenkins-master that you created in the first step.

Then click on the blue button Review and Launch and then Launch button.

In this step you will be prompted to select or to create a key pair. If you don't have it, then create a new one and download it.

If you create a new key pair make sure you change the permissions to the file by running the following command: chmod 0400 my-key.pem. Replace my-key.pem with the real file name.

Connect to new created instance for the next steps

ssh -i /path/to/my-key.pem ec2-user@<master_server_public_ip>

Genearte SSH key

ssh-keygen

Do not add any password for the key. You'll need it to connect to the agent server later

Install java 1.8

sudo yum install java-1.8.0-openjdk.x86_64

After the installation, you can confirm it by running the following command:

java -version

This command will tell you about the Java runtime environment that you have installed:

openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

In order to help Java-based applications locate the Java virtual machine properly, you need to set two environment variables: JAVAHOME and JREHOME.

sudo cp /etc/profile /etc/profile_backup
echo 'export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk' | sudo tee -a /etc/profile
echo 'export JRE_HOME=/usr/lib/jvm/jre' | sudo tee -a /etc/profile
source /etc/profile

Finally, you can print them for review:

echo $JAVA_HOME
echo $JRE_HOME

Install Jenkins

Use the official YUM repo to install the latest stable version of Jenkins, which is 2.150.2 at the time of writing:

cd ~ 
sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
yum install jenkins

Start the Jenkins service and set it to run at boot time:

sudo systemctl start jenkins.service
sudo systemctl enable jenkins.service

In order to allow visitors access to Jenkins, you need to allow inbound traffic on port 8080:

sudo firewall-cmd --zone=public --permanent --add-port=8080/tcp
sudo firewall-cmd --reload

Start jenkins

sudo /etc/init.d/jenkins start

Now, test Jenkins by visiting the following address from your web browser:

http://<your-server-IP>:8080

After opening this url in the browser you should see a page with the title Unlock Jenkins

Grab the acministrator password and add it into the field:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

After you click Continue, click on Install suggested plugins button and wait until jenkins install it's needed plugins.

Whenn plugins are installed you'll be prompted with Create First Admin User form page. Fill in the form to create the Admin user as you wish and then click on Save and Continue

On Instance Configuration page click Save and Finish then Start using Jenkins

Now you should be able to have Jenkins up and running.

Create agent jenkins server

Create a new server instance as you did with the master, but select select Ubuntu Server 18.04 LTS (HVM), SSD Volume Type as server type, jenkins-agent as security group and set the name jenkins-agent

Connect to the agent server

ssh -i /path/to/my-key.pem ubuntu@<agent_server_public_ip>

Allow master server to connect to agent server

On jenkins master server run the following command and copy the output
cat ~/.ssh/id_rsa.pub

On agent server add the output of the previous command to ~/.ssh/authorized_keys

echo '<paste here the id_rsa.pub content>' >> ~/.ssh/authorized_keys

To check if it works, connect to jenkins master server and run the following command:
ssh ubuntu@<jenkins-agent-private-ip>
it should connect to jenkins agent server without problems

Install java on agent server

SSH into agent server and install java 1.8

sudo apt-get update
sudo apt-get install -y openjdk-8-jdk

If your project requires additional dependencies, install them as well

Enable SSH

sudo apt-get install openssh-server

Create a jenkins directory on the agent server

mkdir ~/jenkins

Install EC2 plugin in jenkins master

Open the jenkins url in the browser and navigate to Manage Jenkins > Manage Plugins

Click on Available and in the top right side on the Filter search type ec2. Then select Amazon EC2, then search for SSH and add SSH and SSH Agent plugins and then click Download now and install after restart

It will redirect you to a page when you see a list of plugins and the current status and below you have an option to restart jenkins when installation is complete. Click that checkbox and wait until is ready. You may refresh the page.

When the installation is complete, click again on Manage Jenkins, then click on Manage Nodes, click on Master then in the left sidebar click on Configure. In this form, change from 2 executors to 0 (zero) executors. Master jenkins doesn't need to run any job. Only agents will do this.

Configure agent cloud connection in jenkins

Navigate to Manage Jenkins > Configure System and in the bottom you'll find Cloud. Here click on Add a new cloud and select Amazon EC 2

Add a name for the clould connection
add ec2 credentials (you can find them in users/IAM)
select desired region to use
add you private key that you'll use to connect to the servers
click on Advanced
set Instance Cap to 10 (it will allow maximum 10 agents to be used in the same time. you can change the value as you wish)
select no delay provisioning if you want to create a new agent imediately
session name add what you want
click on Test Connection to make sure that jenkins can communicate to AWS
click on Apply

Add jenkins agent

click on Add button from AMIs section
type jenkins agent for Description
add AMI id (you can configure and EC2 instance and save the AMI to be used here)
select desired instance type (for beginning T2Micro is enough)
add security groups that you created earlier: jenkins-agent, ssh-everywhere
set /home/ubuntu/ for Remote FS root
set ubuntu for Remote user
under AMI Type set sudo for Root command prefix
set 22 for Remote ssh port
set jenkins-agent for Labels (you can set different values here to target specific pipelines)
click on Advanced below Init script
set 1 for Number of Executors
add a Tag with Name for Name and jenkins-agent for Value
everythoing else can be left as it is

Now click on Save to save your changes

Preview final configuration in jenkins

config 1

config 2

From here, you can configure the agent server as you wish and save it as an AMI that you'll use in your projects

Tutorials

Helpers