Setup Jenkins X on AWS with EKS

From Bonus Bits
Jump to: navigation, search

Purpose

This article gives the steps to install and configure a Jenkins X environment using AWS Elastic Kubernetes Service (EKS). It is a more summarised or command-by-command article. Read the sources to get more in-depth information and diagrams.


Prerequisites

  • Homebrew
  • Docker Desktop
  • AWS Account
    • Creation Permissions for many services such as, Route53, VPC, EKS, CloudFormation,
    • Access Keys/STS Token


Example Environment

  • macOS 10.14.2
  • Docker Desktop for macOS Version 2.0.0.0-mac82 (29268) [Edge]
    • Kubernetes-cli v1.10.3


Summary

  1. Setup Local CLI Tools
  2. Create a DNS Domain
  3. Cloudformation Template created by jx and used to create environment
    1. 3 VPC created
    2. 3 InternetGateway
    3. 3 NATIP
    4. 3 NATGateway
    5. 1 EKS Cluster
    6. 9 RouteTableAssociationPublic
    7. 9 RouteTableAssociationPrivate
    8. 6 ControlPlane
    9. 3 PublicSubnetRoute
    10. 3 PolicyNLB
    11. 3 PolicyCloudWatchMetrics
    12. 9 SubnetPublic
    13. 9 SubnetPrivate
    14. 3 PrivateRouteTable
    15. 3 PublicRouteTable
    16. 3 ControlPlaneSecurityGroup


Issues

  • There doesn't seem to be a way to nuke the entire stack using the jx tool. For when you're done with say a lab or if the CloudFormation fail there isn't a way to destroy/delete.
  • I'm not seeing a blue/green or rehydration solution for the jx stack. New baked AMI, plugins, Jenkins version etc.


My Questions

  • Can you blue/green or is it in-place upgrades only?
  • How can you restore lost data?
  • How to restore history to a new stack?
  • Dev, QA and Prod in separate AWS accounts?
  • How do you upgrade Jenkins?
  • How to upgrade plugins with automation?
  • Can you set AMI used by cluster (Corp secure/approved AMI blah)
  • How do you swap out AMIs?
  • What Linux OS is used for the master/agents?

Install Jenkins X CLI Client Locally

brew tap jenkins-x/jx
brew install jx


Install Helm Locally

brew install kubernetes-helm


Install eksctl Locally

If you don't have the tools installed jx will install them for you.


Install hipto authenticator for aws

If you don't have the tools installed jx will install them for you.


Set Path to jx Installed Binaries

export PATH="$PATH:$HOME/.jx/bin"


Create DNS or Use Existing

Create

blah

Existing

blah

Example

  1. Domain Name: bonusbits


Launch Stack

jx create cluster eks --cluster-name=<cluster name> --skip-installation=true

Example

jx create cluster eks --cluster-name=bonusbits-jx-lab --profile bonusbits-dev --region us-west-2 --skip-installation=true


Delete All

Currently I don't see an easy way using the jx command tool to nuke everything. So, my current solution is to just delete the CloudFormation stacks that created everything in the AWS Console or using aws-cli.


Set Specific Jenkins Version (Optional)

  1. blah


Include Specific Plugins (Optional)

To add plugins to the master we need to create our own custom Docker image for the master with the plugins based on the jenkinsx image and using a shell script they offer.

  1. Create a text file plugins.txt with a list of the plugins and versions
    myplugin:1.2.3
    anotherplugin:4.5.6
  2. Create a Dockerfile like the example below
    # Dockerfile for adding plugins to Jenkins X 
    FROM jenkinsxio/jenkinsx:latest
    
    COPY plugins.txt /usr/share/jenkins/ref/openshift-plugins.txt
    RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/openshift-plugins.txt
    
  3. Build Image
    docker build -f Dockerfile -t <name to give container> .
    


Troubleshooting

  • if the jx command erroring on creating the cluster then try enabling debug output. --eksctl-log-level=4 --log-level='debug'
  • Check for latest jx and dependency versions
  • Check the Events on the CloudFormation stacks
  • Check AWS permissions. One way is go in the AWS web console and see if you have read/write for the summary of services listed at the top of this article.


Notable Configurations

  • $EKS_AVAILABILITY_ZONES


Useful Args for EKS Cluster Create

  • jx create cluster eks --help
    
  • --skip-installation=false: Provision cluster only, don't install Jenkins X into it
  • --eksctl-log-level=-1: set log level, use 0 to silence, 4 for debugging and 5 for debugging with AWS debug logging (default 3)
  • --log-level='info': Logging level. Possible values - panic, fatal, error, warning, info, debug.
  • -p, --profile='': AWS profile to use. If provided, this overrides the AWS_PROFILE environment variable
  • -r, --region='': The region to use. Default: us-west-2
  • --version='': The specific platform version to install
--node-type='m5.large': node instance type
  -o, --nodes=-1: number of nodes
      --nodes-max=-1: maximum number of nodes
      --nodes-min=-1: minimum number of nodes


Sources