Using Spot Instances as Jenkins Slaves for Cost Optimization

Customer Profile

Our customer is a major player in providing SaaS applications for Oil and Gas industry. They are based in Nordics and work extensively with large data sets to solve complex problems for their customers. Their applications use Machine Learning, Data Science and AWS Services to solve some of the most difficult tasks, accurately for their customers.

Goal

During our discussion with customer CTO we identified following goals:

  1. Increase effective build capacity of Jenkins server
  2. Categorize multiple build slaves based on build requirements such as UI, AWS Sam, Docker, Python etc.
  3. Use different build slaves for different use cases
  4. Lower overall cost of running Jenkins CI

Solution

By utilizing right sizing and spot instances feature of AWS, we implemented following solution:

  1. Decresed instanace size of Jenkins master to a Small instance
  2. Updated Jenkins configurations to run all build jobs on Build Slaves
  3. Used AWS Spot fleet to request and maintain required build capacity. By utilizing Spot instances, we were able to save about 70% on instance cost while still maintaining required build capacity with Spot Fleet.
  4. Created 2 Spot Fleets with different CPU & memory requirements based on build types:
    • First for Node.js, UI builds
    • Second for AWS Sam and Docker builds
  5. This allowed us to maintain different build slave pools for different job types based on project requirements.

Tech Stack

This solution used following tool, platforms, services and programming languages:

  1. Public cloud platform – AWS. Following services are prominently used:
    • AWS EC2
    • AWS Spot Fleet
    • AWS SAM Templates
    • Cloud Formation
    • Lambda
    • AWS Elastic Container Registery
    • AWS Elastic Container Service
    • Simple Storage Service
  2. Jenkins – Continuous integration server
  3. Docker – Containers for Node.js and Python
  4. AWS SAM CLI – for bulding and packageing Lambda applications for AWS platform
  5. Programming languages – Node.js, Python

Results

Based on changes implemented with right sizing and spot instances, customer gained following goals:

  1. Almost 2X increase in Jenkins CI build capacity
  2. 40% reduction in cost of running JenkinsCI
  3. No Queued builds due to increased Build Capacity
  4. Faster build and deployment

Join our list

Subscribe to our mailing list and get the latest news and updates regarding Cloud Management.

Thank you for subscribing.

Something went wrong.

error: Content is protected !!

Join our list

Subscribe to our mailing list and get the latest news and updates regarding Cloud Management.

Thank you for subscribing.

Something went wrong.

Send this to a friend