I have a small Kubernetes cluster at home, and one of its machines is a desktop with an NVIDIA RTX 3090. I wanted a clean way to train machine learning models on that GPU. Until now that meant SSHing into the desktop and running a script I had to remember to clean up. Kubernetes could already see the GPU, but that didn’t make running experiments any easier.
ClearML does. It’s an open-source tool that tracks experiments and keeps a queue of jobs. A small agent watches the queue and starts a pod for each job, so I can send a training script from my laptop and have it run on the 3090.
This post covers the ClearML server, the agent, a custom image for the job pods, and everything that broke along the way. You need a Kubernetes cluster with a GPU node and some comfort with Helm and YAML.
My setup
- Cluster: four nodes running k3s (a lightweight Kubernetes) on NixOS. Flux deploys everything from a git repo.
nixos-desktop: the machine with the 3090. It has agpu=true:NoScheduletaint, so only workloads that tolerate it land there.nixoid: a node that’s always on and has about 32GB of RAM. The ClearML server lives here.- Tailscale: my private network (a “tailnet”). Services are only reachable through it, which is why you’ll see Tailscale ingresses and
ts.nethostnames below.
Swap in your own node names and hostnames if you follow along.
1. Install the server
I used the official clearml Helm chart (7.15.0). It brings an API server, a web server and a file server, plus MongoDB, Elasticsearch and Dragonfly (a Redis replacement). I budgeted about 6GB of RAM for all of it.
Pin everything to
nixoidwith one YAML anchor:apiserver: nodeSelector: &nixoid kubernetes.io/hostname: nixoid fileserver: nodeSelector: *nixoidUse
local-pathfor storage. NFS is a poor fit for MongoDB and Elasticsearch.Make two Secrets and keep them out of git.
clearml-secretsholds the seven random keys the chart needs.clearml-apiserver-configholds the web login. The open-source server has no SSO, only fixed users in anapiserver.conf, so the chart reads that Secret throughexistingAdditionalConfigsSecret.Give it three Tailscale ingresses:
clearml,clearml-apiandclearml-files. Tailscale ingress gives one hostname per Ingress, and the SDK talks to the API and file servers directly, not through the web UI.
Problem: Elasticsearch crash loop
The first install left every pod stuck in Init, waiting for Elasticsearch, which was in CrashLoopBackOff. The log ended with:
NullPointerException: Cannot invoke "jdk.internal.platform.CgroupInfo.getMountPoint()"
because "anyController" is null
The chart’s default image is Elasticsearch 7.17.3. It bundles a JDK that can’t read cgroup v2, which NixOS uses. Later 7.17 patch releases fix it.
- Set
imageTag: "7.17.29". - Flux was still inside the HelmRelease’s first install, waiting up to 15 minutes for readiness, so it wouldn’t apply the new value. I set the image on the StatefulSet by hand to the same value.
- Delete the pod.
A few minutes later the HelmRelease finished and reported the committed values.
2. Install the agent
The server only stores tasks. To run them you need the clearml-agent chart (5.3.3). In Kubernetes mode it’s called the k8s glue. It watches a queue and makes a pod for each task.
- Create a queue called
gpu. - Run the agent on
nixoid. It’s light, and only the task pods need the GPU node. - Log into the UI and create a credential pair under Settings. The agent needs one the server issued, so it can’t exist before the server does.
- Store the pair in a Secret called
clearml-agent-credentials. Until then the agent pod sits inCreateContainerConfigError. That’s expected. - Give each task pod a node selector for
nixos-desktop, the toleration for the taint, anvidia.com/gpu: 1limit, andruntimeClassName: nvidia.
The chart’s pod template has no field for the runtime class. I used a Flux postRenderers patch that replaces the template ConfigMap wholesale. That means the basePodTemplate values are ignored, and a comment in the values file says where to edit instead. After any chart upgrade I need to diff the rendered template against the patch.
A task pod asks for the whole card, so it stays Pending while one of my other GPU services (HyperQwen, qwen-tts or vibevoice) has it. I scale those to zero first.
Problem: the agent image is from 2022
The agent pod runs allegroai/clearml-agent-k8s-base:1.24-21. It was last built in 2022 on Ubuntu 18.04, so it has Python 3.6. Its entrypoint runs pip install -U clearml-agent on every start, which pulls a release that no longer imports on 3.6:
File ".../clearml_agent/commands/install_bootstrap.py", line 15
from __future__ import annotations
SyntaxError: future feature annotations is not defined
The entrypoint reads CLEARML_AGENT_UPDATE_VERSION, so I pinned the agent. I tested versions in a throwaway pod on the same image. 1.5.2 through 1.9.3 imported, and 2.0.0 and later didn’t.
Version 1.9.3 then failed against the open-source server in two ways.
First, the chart passes --create-queue, which 1.9.3 doesn’t recognize. I turned createQueueIfNotExists off and made the gpu queue through the API.
The second one was harder:
ERROR: Could not push back task [...] to k8s pending queue k8s_scheduler,
error: Validation error (Cannot skip setting execution queue for a task
that is not enqueued or does not have execution queue set)
The agent checks the server version against 3.22.3, a number from the enterprise versioning. The open-source server reports 2.0, so the check fails and the agent stops the task first. From 1.9.1 on, it then enqueues with update_execution_queue=false, and the server rejects that for a stopped task. Version 1.9.0 predates that call. I pinned 1.9.0 and the hand-off worked.
3. Build the training image
By default a task pod sets itself up on every start. It runs apt-get update, installs git, runs pip install clearml-agent, makes a virtualenv, installs the requirements recorded with the task, and only then runs the script. That’s slow. One run sat Pending for minutes because the Ubuntu mirror was slow.
The requirements step was worse. ClearML records the version of every package the script imports, and it reads them from the machine that submits the task. That’s my laptop, which has the newest of everything. The agent put transformers 5.19 and torch 2.14 into a venv on an image that ships torch 2.5.1, and the task died on import:
AttributeError: module 'torch' has no attribute 'accelerator'
I could have pinned every package in every script. Instead I put the versions in one image and told the agent to install nothing.
Start from
pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime, so torch, torchvision and CUDA come from the base.Add the rest on top:
FROM pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ git libsm6 libxext6 libxrender-dev libglib2.0-0 \ && rm -rf /var/lib/apt/lists/* COPY requirements.txt constraints.txt /tmp/ RUN pip install --no-cache-dir -r /tmp/requirements.txt -c /tmp/constraints.txt \ && out=$(pip check | grep -v '^ninja ' || true) \ && echo "$out" && test -z "$out"requirements.txtholds transformers 4.51.3, huggingface-hub 0.30.2, datasets 3.5.0, accelerate 1.6.0,clearml[s3]and a pinnedclearml-agent(3.0.3). Those are the versions my fine-tune was written against, all chosen in one place.constraints.txtpins torch, torchvision and torchaudio to what the base image already has, so pip can’t replace them. Thepip checkline fails the build on a real conflict. It ignoresninja, because the base image ships a ninja wheel that pip reports as unsupported on this platform.Let GitHub Actions build it and push it to GHCR as
clearml-training:sha-<commit>.The package is private, so the
clearmlnamespace needs a pull secret. I copied the one theqwen-ttsnamespace already uses (not committed) and addedimagePullSecretsto the same pod template patch.Set the image tag in one place:
defaultContainerImagein the agent’s values file. The scripts never mention it.
4. Tell the agent to trust the image
Add three environment variables to the task pod template:
env:
- name: CLEARML_AGENT_SKIP_CONTAINER_APT
value: "1"
- name: CLEARML_AGENT_NO_UPDATE
value: "1"
- name: CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL
value: "1"
The first skips the apt step. The second skips pip install clearml-agent. The third skips the virtualenv and the recorded requirements, so the task runs on the image’s own Python.
Nothing gets installed at start now. If a task needs a package that isn’t in the image, it fails instead of installing it. For one image and one GPU box, I think that’s the right trade.
5. Submit a job
A script submits itself. Near the top, execute_remotely records the code, queues the task and exits on my laptop. The agent then runs the same file in a pod on nixos-desktop.
from clearml import Task
Task.force_store_standalone_script()
task = Task.init(project_name="smoke-tests", task_name="pirate-finetune")
task.execute_remotely(queue_name="gpu", exit_process=True)
force_store_standalone_script is there because my scripts live in a private git repo. Without it ClearML records the repo and commit, and the agent has no git credentials, so it fails with could not read Username for 'https://github.com'. Storing the script in the task means I don’t have to give the agent a token.
My first task was a short script that prints nvidia-smi -L and does a matrix multiplication on CUDA. The server’s log shows:
GPU 0: NVIDIA GeForce RTX 3090
cuda available: True
matmul ok: 1.0 on NVIDIA GeForce RTX 3090
A full fine-tune of a 0.5B model, including the model download and three epochs, then ran from submit to done in 6 minutes 23 seconds. The task console has no apt, pip or virtualenv step in it. That run is the subject of the next post.
6. Fix artifact downloads
A fine-tune uploads its model as a task artifact. The UI’s download link opened clearml-fileserver.clearml.svc:8081/..., and Safari couldn’t find the server. The task pod uploads through the in-cluster address, and ClearML stores that hostname in the artifact URL. Swapping in the tailnet hostname got me a 401, because the file server wants an auth token that the web UI adds and a plain browser request doesn’t.
So I stopped using the file server for artifacts. My cluster already runs rustfs, an S3-compatible object store, behind a Tailscale ingress. A pod on the GPU node can reach rustfs.homelab.example.ts.net, and so can my laptop, so one hostname works for both.
Create a
clearmlbucket in rustfs.Create a user with a policy scoped to that bucket.
Put this in the agent’s
clearml.conf. The agent copies that file into every task pod, so this configures storage once:sdk { development { default_output_uri: "s3://rustfs.homelab.example.ts.net:443/clearml" } aws.s3.credentials: [{ host: "rustfs.homelab.example.ts.net:443" bucket: "clearml" key: "<user>", secret: "<secret>" secure: true, multipart: false }] }Store the config in a Secret and mount it through the chart’s
existingClearmlConfigSecret.
Problem: “Could not get access credentials”
The first task after that change failed on its first line:
ValueError: Could not get access credentials for 's3://rustfs...'
That sent me to the config, but the real cause was a line above it in the log: AWS S3 storage driver (boto3) not found. The output URI applies to every task, so every task needs boto3. That’s why the image installs clearml[s3]. The laptop needs it too (pip install "clearml[s3]") if it downloads artifacts.
Things to remember
- After changing the agent’s configuration, wait until the old agent pod is gone before you submit. During one roll-over the old pod picked up my task and ran it with the old image and none of the skip variables. It looked like the fix had failed.
- The agent’s startup echoes its access key and secret into the pod log. Anyone who can read logs in the namespace can see them, so treat the credential pair as exposed and rotate it.
Still to do
The agent pod itself still runs the 2022 Python 3.6 image, pinned to agent 1.9.0. If I build my own glue image on a current Python, I can drop the pin and test newer agent releases against the server.
Everything stateful sits on one node with local volumes, so backups of MongoDB and Elasticsearch are next. The open-source server also has no SSO, so logins stay as fixed users.