Continuous Machine Learning (CML) is CI/CD for Machine Learning Projects

Get StartedDownload
We’re on GitHub

GitFlow for data science

Use GitLab or GitHub to manage ML experiments, track who trained ML models or modified data and when. Codify data and models with DVC instead of pushing to your Git repo.

Auto reports for ML experiments

Auto-generate reports with metrics and plots in each Git Pull Request. Rigorous engineering practices help your team make informed, data-driven decisions.

No additional services

Build your own ML platform using just GitHub or GitLab and your favorite cloud services: AWS, Azure, GCP, or Kubernetes. No databases, services or complex setup needed.

CML Use Cases

The simplest case of using CML, and a clear way for any user to get started, is to generate a simple report. Add the following .yaml to your project repository and commit to get started

.gitlab-ci.yml

stages:
- cml_run
cml:
stage: cml_run
image: iterativeai/cml:0-dvc2-base1
script:
- pip3 install -r requirements.txt
- python train.py
- cat metrics.txt >> report.md
- cml-publish confusion_matrix.png --md >> report.md
- cml-send-comment report.md
GitLab

CML Report

.github/workflows/cml.yaml

name: train-my-model
on: [push]
jobs:
run:
runs-on: [ubuntu-latest]
steps:
- uses: actions/checkout@v2
- uses: iterative/setup-cml@v1
- uses: actions/setup-python@v2
with:
python-version: '3.x'
- name: cml
env:
REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
pip3 install -r requirements.txt
python train.py
cat metrics.txt >> report.md
cml-publish confusion_matrix.png --md >> report.md
cml-send-comment report.md
GitHub

CML Report

Github Base report example

.gitlab-ci.yml

stages:
- cml_run
cml:
stage: cml_run
image: iterativeai/cml:0-dvc2-base1
script:
- dvc pull data
- pip install -r requirements.txt
- dvc repro
# Compare metrics to master
- git fetch --prune
- dvc metrics diff --show-md master >> report.md
# Visualize loss function diff
- dvc plots diff
--target loss.csv --show-vega master > vega.json
- vl2png vega.json | cml-publish --md >> report.md
- cml-send-comment report.md
GitLab

CML Report

GitLab DVC report example

.github/workflows/cml.yaml

name: train-my-model
on: [push]
jobs:
run:
runs-on: [ubuntu-latest]
steps:
- uses: actions/checkout@v2
- uses: iterative/setup-cml@v1
- uses: iterative/setup-dvc@v1
- uses: actions/setup-python@v2
with:
python-version: '3.x'
- name: cml
env:
REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
AWS_ACCESS_KEY_ID: ${{ secrets.KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.KEY }}
run: |
dvc pull data
pip install -r requirements.txt
dvc repro
# Compare metrics to master
git fetch --prune
dvc metrics diff --show-md master >> report.md
# Visualize loss function diff
dvc plots diff \
--target loss.csv --show-vega master > vega.json
vl2png vega.json -s 1.5 | cml-publish --md >> report.md
cml-send-comment report.md
GitHub

CML Report

GitHub DVC report example

.gitlab-ci.yml

stages:
- cml_run
cml:
stage: cml_run
image: iterativeai/cml:0-dvc2-base1
script:
- pip install -r requirements.txt
- cml-tensorboard-dev \
--logdir logs \
--name "Go to tensorboard" \
--md >> report.md
- cml-send-comment report.md
- python train.py
GitLab

CML Report

GitLab Tensorboard report example

.github/workflows/cml.yaml

name: train-my-model
on: [push]
jobs:
run:
runs-on: [ubuntu-latest]
steps:
- uses: actions/checkout@v2
- uses: iterative/setup-cml@v1
- uses: actions/setup-python@v2
with:
python-version: '3.6'
- name: cml
env:
REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
TB_CREDENTIALS: ${{ secrets.TB_CRED }}
run: |
pip install -r requirements.txt
cml-tensorboard-dev \
--logdir logs \
--name "Go to tensorboard" \
--md >> report.md
cml-send-comment report.md
python train.py
GitHub

CML Report

GitHub Tensorboard report example

.gitlab-ci.yml

stages:
- deploy
- train
deploy_job:
stage: deploy
when: always
image: dvcorg/cml
script:
- cml-runner
--cloud aws
--cloud-region us-west
--cloud-type t2.micro
--labels=cml-runner
train_job:
stage: train
when: on_success
image: iterativeai/cml:0-dvc2-base1
tags:
- cml-runner
script:
- pip install -r requirements.txt
- python train.py
- echo "## Report from your EC2 Instance" > report.md
- cat metrics.txt >> report.md
- cml-publish "confusion_matrix.png" --md >> report.md
- cml-send-comment report.md
GitLab

CML Report

GitLab Cloud report example

.github/workflows/cml.yaml

name: Train-in-the-cloud
on: [push]
jobs:
deploy-runner:
runs-on: [ubuntu-latest]
steps:
- uses: actions/checkout@v2
- uses: iterative/setup-cml@v1
- name: deploy
env:
REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
run: |
cml-runner \
--cloud aws \
--cloud-region us-west \
--cloud-type=t2.micro \
--labels=cml-runner
model-training:
needs: deploy-runner
runs-on: [self-hosted,cml-runner]
container: docker://iterativeai/cml:0-dvc2-base1
steps:
- uses: actions/checkout@v2
- name: "Train my model"
env:
REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
run: |
python --version
pip install -r requirements.txt
python train.py
echo "## Report from your EC2 Instance" > report.md
cat metrics.txt >> report.md
cml-publish "confusion_matrix.png" --md >> report.md
cml-send-comment report.md
GitHub

CML Report

GitHub Cloud report example

.gitlab-ci.yml

stages:
- deploy
- train
deploy_job:
stage: deploy
when: always
image: dvcorg/cml
script:
- cml-runner \
--cloud aws \
--cloud-region us-west \
--cloud-type=g3.4xlarge \
--cloud-hdd-size 64 \
--labels=cml-runner-gpu
train_job:
stage: train
when: on_success
image: iterativeai/cml:0-dvc2-base1-gpu
tags:
- cml-runner-gpu
script:
- apt-get update -y
- apt-get install python3-dev -y
- apt install imagemagick -y
- pip install -r requirements.txt
# DVC reproduce neural style transfer training
- git fetch --prune
- dvc repro
# Compare master and workspace image results
- echo "# Style transfer" >> report.md
- git show origin/master:final_owl.png > master_owl.png
- convert +append final_owl.png master_owl.png out.png
- convert out.png -resize 75% out_shrink.png
- echo "### Workspace vs. Master" >> report.md
- cml-publish out_shrink.png --md >> report.md
# Report training parameters
- echo "## Training parameter diffs" >> report.md
- dvc params diff master --show-md >> report.md
- echo >> report.md
# Report GPU details
- echo "## GPU info" >> report.md
- cat gpu_info.txt >> report.md
- cml-send-comment report.md
GitLab

CML Report

GitLab Cloud report example

.github/workflows/cml.yaml

name: train-my-model
on: [push]
jobs:
deploy-runner:
runs-on: [ubuntu-latest]
steps:
- uses: actions/checkout@v2
- uses: iterative/setup-cml@v1
- name: deploy
env:
REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
run: |
cml-runner \
--cloud aws \
--cloud-region us-west \
--cloud-type=g3.4xlarge \
--cloud-hdd-size 64 \
--labels=cml-runner
run:
needs: deploy-runner
runs-on: [self-hosted,cml-runner]
container:
image: docker://dvcorg/cml
options: --gpus all
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: '3.6'
- name: cml
env:
REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
run: |
apt-get update -y
apt install imagemagick -y
pip install -r requirements.txt
# DVC stuff
git fetch --prune
dvc repro
echo "# Style transfer" >> report.md
git show origin/master:final_owl.png > master_owl.png
convert +append final_owl.png master_owl.png out.png
convert out.png -resize 75% out_shrink.png
echo "### Workspace vs. Main" >> report.md
cml-publish out_shrink.png --md --title 'compare' >> report.md
echo "## Training metrics" >> report.md
dvc params diff master --show-md >> report.md
echo >> report.md
echo "## GPU info" >> report.md
cat gpu_info.txt >> report.md
cml-send-comment report.md
GitHub

CML Report

GitHub Cloud report example

The MLOps Ecosystem

MLOps isn't a platform- it's an ecosystem of tools. CML helps you bring your favorite DevOps tools to machine learning.
  • Continuous integration for ML

    CML

  • Manage environments

    Docker and Packer

  • Infrastructure as code

    Terraform and Docker-Machine

  • Data as code

    DVC

Subscribe for updates. We won't spam you.