by iterative.ai
Get started
Edit on GitHub

Self-hosted (On-premise or Cloud) Runners

GitHub Actions and GitLab CI workflows are run on GitHub- and GitLab- hosted runners by default. However, there are many great reasons to use your own runners: to take advantage of GPUs, orchestrate your team's shared computing resources, or train in the cloud.

☝️ Tip! Check out the official documentation from GitHub and GitLab for more information on self-hosted runners.

⚠️ Using self-hosted runners for Bitbucket Pipelines is not yet supported.

Allocating Cloud Compute Resources with CML

When a workflow requires computational resources (such as GPUs), CML can automatically allocate cloud instances using cml runner. You can spin up instances on AWS, Azure, GCP, or Kubernetes (see below). Alternatively, you can connect any other compute provider or on-premise (local) machine.

For example, the following workflow deploys a p2.xlarge instance on AWS EC2 and trains a model on the instance. After the job runs, the instance automatically shuts down.

You might notice that this workflow is quite similar to the basic use case. The only addition is cml runner and a few environment variables for passing your cloud compute credentials to the workflow.

Note that cml runner will also automatically restart your jobs (whether from a GitHub Actions 72-hour timeout or a AWS EC2 spot instance interruption).

name: CML
on: [push]
jobs:
  deploy-runner:
    runs-on: ubuntu-latest
    steps:
      - uses: iterative/setup-cml@v1
      - uses: actions/checkout@v2
      - name: Deploy runner on EC2
        env:
          REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          cml runner \
              --cloud=aws \
              --cloud-region=us-west \
              --cloud-type=p2.xlarge \
              --labels=cml-gpu
  train-model:
    needs: deploy-runner
    runs-on: [self-hosted, cml-gpu]
    timeout-minutes: 4320 # 72h
    container:
      image: docker://iterativeai/cml:0-dvc2-base1-gpu
      options: --gpus all
    steps:
      - uses: actions/checkout@v2
      - name: Train model
        env:
          REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
        run: |
          pip install -r requirements.txt
          python train.py

          # Create CML report
          cat metrics.txt >> report.md
          cml publish plot.png --md >> report.md
          cml send-comment report.md
deploy-runner:
  image: iterativeai/cml:0-dvc2-base1
  script:
    - |
      cml runner \
          --cloud=aws \
          --cloud-region=us-west \
          --cloud-type=p2.xlarge \
          --cloud-spot \
          --labels=cml-gpu
train-model:
  needs: [deploy-runner]
  tags:
    - cml-gpu
  image: iterativeai/cml:0-dvc2-base1-gpu
  script:
    - pip install -r requirements.txt
    - python train.py
  image: iterativeai/cml:0-dvc2-base1
  script:
    - pip install -r requirements.txt
    - python train.py

    # Create CML report
    - cat metrics.txt >> report.md
    - cml publish plot.png --md >> report.md
    - cml send-comment report.md

In the workflow above, the deploy-runner step launches an EC2 p2.xlarge instance in the us-west region. The train-model job then runs on the newly-launched instance. See Environment Variables below for details on the secrets required.

🎉 Note that jobs can use any Docker container! To use commands such as cml send-comment from a job, the only requirement is to have CML installed.

Docker Images

The CML Docker images (docker://iterativeai/cml or docker://ghcr.io/iterative/cml) come loaded with Python, CUDA, git, node and other essentials for full-stack data science. Different versions of these essentials are available from different iterativeai/cml image tags. The tag convention is {CML_VER}-dvc{DVC_VER}-base{BASE_VER}{-gpu}:

{BASE_VER}Software included (-gpu)
0Ubuntu 18.04, Python 2.7 (CUDA 10.1, CuDNN 7)
1Ubuntu 20.04, Python 3.8 (CUDA 11.0.3, CuDNN 8)

For example, docker://iterativeai/cml:0-dvc2-base1-gpu, or docker://ghcr.io/iterative/cml:0-dvc2-base1.

Options

The cml runner command supports many options (see the command reference). Notable options are:

  • --labels: One or more comma-delimited labels (e.g. "cml,gpu").
  • --idle-timeout: Seconds to wait for jobs before shutting down.
  • --single: Terminate after running a single job.
  • --reuse: Don't launch a new runner if an existing one has the same name or overlapping labels.
  • --cloud: Cloud to deploy the runner to ("aws", "azure", "gcp", or "kubernetes").
  • --cloud-spot: Whether to use spot instances.
  • --cloud-spot-price: Maximum spot instance bidding price in USD.
  • --cloud-region: For example, "us-east" or "eu-west".
  • --cloud-type: "m", "l", "xl", or native types such as "t2.micro".
  • --cloud-gpu: GPU type ("nogpu", "k80", "v100", "tesla").
  • --cloud-hdd-size: Disk size in GB.

Environment Variables

Sensitive values like cloud and repository credentials can be provided through environment variables with the aid of GitHub secrets or GitLab masked variables; the latter also supports external secrets for added security.

⚠️ You will need to create a personal access token (PAT) with repository read/write access. In the example workflow above, this token is stored as PERSONAL_ACCESS_TOKEN.

🛈 If using the --cloud option, you will also need to provide access credentials for your cloud compute resources as secrets. In the above example, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (with privileges to create & destroy EC2 instances) are required.

Personal Access Token

This token serves as a repository access credential, and is especially required for cml runner to function.

Use either:

Ideally, you should not use personal access tokens from your own account, as they grant access to all your repositories. Instead, it's highly recommended to create a separate bot account that only has access to the repositories where you plan to deploy runners to. Bot accounts are the same as normal user accounts, with the only difference being the intended use case.

PAT

For instance, to use a personal access token:

  1. Generate a new personal access token under GitHub developer settings

    • in the "Note" field, type PERSONAL_ACCESS_TOKEN
    • select repo scope
    • click "Generate token" and copy it
  2. In you GitHub repository and/or organization, navigate to SettingsSecretsNew repository/organization secret

    • in the "Name" field, type PERSONAL_ACCESS_TOKEN
    • in the "Value" field, paste the token
    • click "Add secret"

Step 2 can also be used for adding other secrets such as cloud access credentials.

App

Alternatively, a GitHub App ID (CML_GITHUB_APP_ID) and private key (CML_GITHUB_APP_PEM) can be used to generate a token on-the-fly, as shown in the example below:

steps:
  - uses: navikt/github-app-token-generator@v1
    id: get-token
    with:
      private-key: ${{ secrets.CML_GITHUB_APP_PEM }}
      app-id: ${{ secrets.CML_GITHUB_APP_ID }}
  - uses: actions/checkout@v2
    with:
      token: ${{ steps.get-token.outputs.token }}
  - name: Train model
    env:
      REPO_TOKEN: ${{ steps.get-token.outputs.token }}
    run: |
      ...
      cml send-comment report.md

Note that the Apps require the following write permissions:

  • Repository permissions (if used on a per-repo basis)

    • Administration (cml runner)
    • Checks (cml send-github-check)
    • Pull requests (cml {pr,send-comment})
  • Organization permissions (if used on an org)

    • Self-hosted runners (cml runner)

Use either:

For instance, to use a personal access token:

  1. Navigate to User SettingsAccess Tokens

    • in the "Name" field, type REPO_TOKEN
    • select api, read_repository and write_repository
    • click "Create personal access token" and copy it
  2. In your GitLab project, navigate to SettingsCI/CDVariablesAdd Variable

    • in the "Key" field, type REPO_TOKEN
    • in the "Value" field, paste your Personal Access Token
    • select "Mask variable"
    • deselect "Protect variable"
    • click "Add variable" at the bottom of the dialog box

Step 2 can also be used for adding other masked variables such as cloud access credentials.

Bitbucket Cloud does not use access tokens. Instead, create a REPO_TOKEN variable with a Base64 encoded username and password.

Use either:

  • your user access credentials (consider using Bitbucket Cloud App Passwords, or
  • create a designated "CI/CD" bot account for CML authentication. Bot accounts are the same as normal user accounts, with the only difference being the intended use case: you limit the account to only access the repositories where you plan to deploy runners to.

In either case, the steps to create a REPO_TOKEN are:

  1. Use a Base64 encoder of your choice to encode a Bitbucket username and password:

    • echo -n $USERNAME:$PASSWORD | base64. The -n ensures the base64 does not contain the trailing newline that echo adds by default.
    • copy the resulting Base64 token
  2. In your repository, go to Repository SettingsRepository Variables

    • in the "Name" field, type REPO_TOKEN
    • in the "Value" field, paste the Base64 token
    • select Secured to hide credentials in all Bitbucket logs

Step 2 can also be used for adding other secured variables such as cloud access credentials.

Cloud Compute Resource Credentials

Note that you will also need to provide access credentials of your compute resources. In the above example, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are required to deploy EC2 instances.

Click below to see credentials needed for supported compute providers.

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_SESSION_TOKEN (optional)

See the AWS credentials docs for obtaining these keys.

☝️ Note The same credentials can also be used for configuring cloud storage.

  • AZURE_CLIENT_ID
  • AZURE_CLIENT_SECRET
  • AZURE_SUBSCRIPTION_ID
  • AZURE_TENANT_ID

Either one of:

  • GOOGLE_APPLICATION_CREDENTIALS_DATA: the contents of a service account JSON file, or
  • GOOGLE_APPLICATION_CREDENTIALS: the path to the JSON file.

The former is more convenient for CI/CD scenarios, where secrets are (usually) provisioned through environment variables instead of files.

  • KUBERNETES_CONFIGURATION: the contents of a kubeconfig file.

On-premise (Local) Runners

The cml runner command can also be used to manually set up a local machine, on-premise GPU cluster, or any other cloud compute resource as a self-hosted runner. Simply install CML and then run:

cml runner \
  --repo="$REPOSITORY_URL" \
  --token="$PERSONAL_ACCESS_TOKEN" \
  --labels="local,runner" \
  --idle-timeout=180

The machine will listen for jobs from your repository and execute them locally.

⚠️ Warning: anyone with access to your repository (everybody for public ones) may be able to execute arbitrary code on your machine. Refer to the corresponding GitHub and GitLab documentation for additional guidance.

Content

🐛 Found an issue? Let us know! Or fix it:

Edit on GitHub

Have a question? Join our chat, we will help you:

Discord Chat