Here, we'll walk through a tutorial to start using CML. For simplicity, we'll show the demo in GitHub Actions, but instructions are pretty similar for all the supported CI systems.
Fork our example project repository.
The following steps can all be done in the GitHub browser interface. However, to follow along the commands, we recommend cloning your fork to your local workstation:
$ git clone https://github.com/<your-username>/example_cml
To create a CML workflow, copy the following into a new file,
.github/workflows/cml.yaml
:
name: CML
on: [push]
jobs:
run:
runs-on: ubuntu-latest
container: docker://ghcr.io/iterative/cml:0-dvc2-base1
steps:
- uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Train model
env:
REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
pip install -r requirements.txt
python train.py
cat metrics.txt >> report.md
cml publish plot.png --md >> report.md
cml send-comment report.md
In your text editor of choice, edit line 16 of train.py
to depth = 5
.
Commit and push the changes:
$ git checkout -b experiment
$ git add . && git commit -m "modify forest depth"
$ git push origin experiment
In GitHub, open up a Pull Request to compare the experiment
branch to
master
.
Shortly, you should see a comment from github-actions
appear in the Pull
Request with your CML report. This is a result of the cml send-comment
command in your workflow.
This is the gist of the CML workflow: when you push changes to your GitHub
repository, the workflow in your .github/workflows/cml.yaml
file gets run and
a report generated.
CML commands let you display relevant results from the workflow, like model performance metrics and vizualizations, in GitHub checks and comments. What kind of workflow you want to run, and want to put in your CML report, is up to you.
An example of what your repository should look like now can be found at
iterative/cml_base_case
.
In the above example, we got the CML commands thanks to our Docker container.
But there's another way for GitHub Actions users to get CML: the setup-cml
Action!
The iterative/setup-cml action is a JavaScript workflow that provides CML commands in your GitHub Actions workflow. The action allows users to install CML without using the CML Docker container.
This action gives you:
cml publish
and cml send-comment
for publishing data
visualization and metrics from your CI workflow as comments in a pull request.cml runner
, a command that enables workflows to provision cloud and
on-premise computing resources for training modelsNote that CML does not include DVC and its dependencies- for that, you want the Setup DVC Action.
This action has been tested on ubuntu-latest
and macos-latest
.
Basic usage:
steps:
- uses: iterative/setup-cml@v1
A specific version can be pinned to your workflow.
steps:
- uses: iterative/setup-cml@v1
with:
version: '1.0.1'
The following inputs are supported.
version
- (optional) The version of CML to install. The default value of
latest
will install the latest version of CML.Setup CML has no outputs.
Assume that we have a machine learning script, train.py
, that outputs an image
plot.png
. A potential workflow will look like this:
steps:
- uses: iterative/setup-cml@v1
- uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha }}
- env:
REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
# train will generate plot.png
python train.py
echo "# My first CML report" >> report.md
cml publish plot.png --md --title="Confusion Matrix" >> report.md
cml send-comment report.md