How to Autoscale your GitLab runners on OpenStack
GitLab Scaling CI/CD Docker

How to Autoscale your GitLab runners on OpenStack

Autoscaling your GitLab runners on Fuga has many advantages. For example, you can only have runners active during office hours or just when you need them. The unused runners will be removed automatically. Since Fuga Cloud is pay for what you use, your total costs for your GitLab CI/CD will go down. In this tutorial is explained on how to add autoscaling GitLab runners to your GitLab using GitLab CI and Docker Machine.

When a commit is made, a CI/CD pipeline will be started. GitLab will notify the runner-manager that a new job is available. The runner-manager will pass down this job to an available runner, if there is no available, the runner-manager will create a new instance with a runner. This pipeline can be customized to your liking, for example how many idle runners are available or the amount of jobs each runner can handle concurrently. The flow looks like this:

GitLab Autoscale flow

Requirements

  • GitLab installed on Fuga. See our other Gitlab tutorial
  • SSH keys added to your GitLab admin account

Setting up the security groups

In order to autoscale your runners, Docker Machine requires port 2376 to be opened for every new runner that gets automatically deployed. In this chapter is explained how you can open this port with a security group.

  1. To open this port, go to Access and security and click on the Security Groups tab.

  2. Click on + Create Security Group to create a new security group.

  3. Enter a fitting name for this Security Group, for example gitlabrunner. You can enter a description for clarification if you’d like to.

  4. Click on Create Security Group to create the new security group.

  5. Once the new security group has been created, click on the Manage Rules button that belongs to the new security group you just made.

  6. In the top right corner, click on + Add Rule to add a new security rule.

  7. In the Port field enter 2376 and keep the rest of the form at the default values. It should look like this:

    Docker machine security rule
  8. Click on Add to add the security rule.

Creating new SSH key pair for GitLab

Between the runner-manager and the runner instances, an SSH connection is used by Docker-Machine. Instead of using your personal private SSH keys, a new key is generated and used specifically for the runners.

  1. On your Fuga dashboard go to Access & Security.

  2. Click on the tab Key Pairs.

  3. Click on + Create Key Pair.

  4. Choose a fitting name for the Key Pair, this tutorial uses name gitlab.

  5. Save this file to your machine, it will be needed later for the SSH connections between the runners.

Installing and registering the GitLab runner

In this chapter the installation and the registration of the GitLab runner will be done. This runner, called the runner-manager, will be used to connect to your GitLab and manages the runners with Docker Machine. These runners will receive the jobs from the runner-manager which received the jobs from GitLab.

  1. Launch a new instance on Fuga, in this tutorial the instance is called runner-manager. The flavor c1.micro is chosen with the image Ubuntu 18.04.

  2. Log on into your runner-manager instance using the following example command in your Terminal:

    ssh ubuntu@FLOATING_IP
    

    For other examples of SSH follow the Getting Started guide.

  3. To install the GitLab runner packages, the repositories need to be added first. Run the following command:

    curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
    
  4. Install the GitLab runner with the following command:

    sudo apt-get install gitlab-runner
    
  5. Install Docker with the following command:

    sudo apt install docker.io
    
  6. Install Docker-Machine with the following command:

    base=https://github.com/docker/machine/releases/download/v0.14.0 &&
      curl -L $base/docker-machine-$(uname -s)-$(uname -m) >/tmp/docker-machine &&
      sudo install /tmp/docker-machine /usr/local/bin/docker-machine
    
  7. Log in to your GitLab with an admin account (the default admin account is root).

  8. Click on the wrench icon Admin Area located in the top bar of GitLab.

  9. Go to Overview -> Runners in the left menu bar. Keep this page open because the URL and registration token is needed in the following steps.

  10. Enter the following command in your GitLab runner instance to start the registration:

    sudo gitlab-runner register
    
  11. When asked for your GitLab URL, enter the URL of your GitLab. For example http://gitlab.example.com or http://185.xx.xx.xx.

  12. When asked for a GitLab-ci token, enter the token on the page of step 9.

  13. When asked for a description of your GitLab runner, enter a fitting description.

  14. When asked for tags for your GitLab runner, enter tags that you deem fitting.

  15. When asked to lock your GitLab runner to the project, enter false.

  16. When asked for an Executor for your GitLab runner enter docker+machine.

  17. When asked for a default Docker image, enter nginx. You can choose anything you like but for this tutorial the nginx image is used.

Setting up SSH between the runner-manager and the runners

Once the GitLab runner is installed, some additional configuration is necessary. By default our configuration will look for a gitlab.pem file in the home/ubuntu/.ssh folder. You need to transfer your new .pem file to the runner-manager instance, so it can be used for the SSH connections between the runners.

  1. Show the contents from the pem file you downloaded from the chapter Creating new SSH key pair for GitLab with the following command:

    cat gitlab.pem
    
  2. Copy the entire contents to your clipboard.

  3. In your .ssh folder on your runner-manager instance, create a new gitlab.pem file:

    touch .ssh/gitlab.pem
    
  4. Open the newly created gitlab.pem file with your favorite text editor and paste in the contents from your clipboard.

  5. Save the file.

Configuring the runner-manager

Once everything is installed, the last changes to the runner-manager instance have to be fulfilled.

  1. Open the configuration file /etc/gitlab-runner/config.toml in your runner-manager instance with your favorite text editor, for example:

    sudo vim /etc/gitlab-runner/config.toml
    
  2. Copy the name, url and token under the [[runners]] and save them temporarily on your machine.

  3. Copy and paste the following code:

    concurrent = 8
    check_interval = 0
    
    [[runners]]
      name = "NAME_OF_YOUR_MANAGER_INSTANCE"
      limit = 4
      url = "YOUR_GITLAB_URL_OR_IP_WITH_HTTP(S)"
      token = "YOUR_GITLAB_RUNNER_MANAGER_TOKEN"
      executor = "docker+machine"
      [runners.docker]
        tls_verify = false
        image = "nginx"
        privileged = true
        disable_cache = true
        volumes = ["/var/run/docker.sock:/var/run/docker.sock"]
        shm_size = 0
      [runners.machine]
        IdleCount = 1
        IdleTime = 1800
        MaxBuilds = 100
        MachineDriver = "openstack"
        MachineName = "gitlab-ci-as-%s"
        MachineOptions = [
        "openstack-username=YOUR_FUGA_USERNAME",
        "openstack-password=YOUR_FUGA_PASSWORD",
        "openstack-tenant-name=YOUR_FUGA_USERNAME",
        "openstack-auth-url=https://identity.api.fuga.io:5000/v2.0",
        "openstack-flavor-name=c1.small",
        "openstack-image-name=Ubuntu 18.04 LTS - Bionic Beaver - 64-bit - Fuga Cloud Based Image",
        "openstack-net-name=YOUR_FUGA_NETWORK_NAME",
        "openstack-floatingip-pool=external",
        "openstack-ssh-user=ubuntu",
        "openstack-keypair-name=gitlab",
        "openstack-private-key-file=/home/ubuntu/.ssh/gitlab.pem",
        "openstack-sec-groups=default,gitlabrunner"
        ]
        OffPeakPeriods = ["* * 0-7,19-23 * * mon-fri *", "* * * * * sat,sun *"]
        OffPeakTimezone = "Europe/Amsterdam"
        OffPeakIdleCount = 0
        OffPeakIdleTime = 1200
    
  4. Change the NAME_OF_YOUR_MANAGER_INSTANCE to the name you saved in step 2.

  5. Change the YOUR_GITLAB_URL_OR_IP_WITH_HTTP(S) to the url you saved in step 2.

  6. Change the YOUR_GITLAB_RUNNER_MANAGER_TOKENto the token you saved in step 2.

  7. Under MachineOptions, change openstack-username to your Fuga username.

  8. Under MachineOptions, change openstack-password to your Fuga password.

  9. Under MachineOptions, change openstack-tenant-name to your Fuga username.

  10. Under MachineOptions, change openstack-net-name to your Fuga Network name.

  11. Save the file.

This config will by default always have a runner active during office hours. You can change the settings to anything you want under the OffPeakPeriods. For more information about the settings in this file, see the official GitLab documentation under Advanced configuration.

Testing the configuration

Your GitLab configuration should be ready to be used. To test all the changes you can create a test project and see if the jobs will successfully run.

  1. To create a new project on your GitLab, click the plus icon located on the top of the screen next to the search bar and click New project.

  2. Give it a name and click on Create project.

  3. Clone your new project to your system using the git link located on the project page:

    git clone git@gitlab.example.com:root/your-project.git
    
  4. Open your new git folder inside your terminal:

    cd your-project
    
  5. Create a new file that will contain our CI/CD settings:

    touch .gitlab-ci.yml
    
  6. Open the file in your favorite text editor and copy paste the following code inside:

      image: nginx
      stages:
        - test
        - deploy
    
      test:
        stage: test
        script:
          - echo "This is the test stage."
    
      deploy:
        stage: deploy
        script:
          - echo "This is the deployment stage."
    
  7. Save the file and we’re now ready to test our CI/CD pipeline.

  8. Enter the following command to create a test file:

    echo Testfile >> testfile
    
  9. Execute the following command to add all the files to your git staged changes:

    git add .
    
  10. Execute the following command to create a new commit:

    git commit -m "Initial commit"
    
  11. Push all the changes to your GitLab:

    git push
    
  12. Go to your GitLab project and in the left menu bar click on CI/CD.

  13. There should be two green check marks indicating both of your CI/CD stages as successful.

Was this article helpful?


Next article:

How to migrate your ObjectStore to the Fuga Cloud ObjectStore

It’s possible to move your Object Store from a different platform to the Fuga Object Store. Since almost all of the Object Stores are S3 compatible, you are free to migrate to any compatible platform. This tutorial will explain how to create a Fuga Object Store container and how to migrate from different platforms. Requirements Linux/macOS/BSD system Creating a Fuga Object Store container Before you can move your S3 compatible Object Store from a different platform to the Fuga Object Store, you first need a Fuga Object Store container.