I have a small Kubernetes cluster at home, and one of its machines is a desktop with an NVIDIA RTX 3090. I wanted a clean way to train machine learning models on that GPU. Until now that meant SSHing into the desktop and running a script I had to remember to clean up. Kubernetes could already see the GPU, but that didn’t make running experiments any easier.

ClearML does. It’s an open-source tool that tracks experiments and keeps a queue of jobs. A small agent watches the queue and starts a pod for each job, so I can send a training script from my laptop and have it run on the 3090.

This post covers the ClearML server, the agent, a custom image for the job pods, and everything that broke along the way. You need a Kubernetes cluster with a GPU node and some comfort with Helm and YAML.

My setup

  • Cluster: four nodes running k3s (a lightweight Kubernetes) on NixOS. Flux deploys everything from a git repo.
  • nixos-desktop: the machine with the 3090. It has a gpu=true:NoSchedule taint, so only workloads that tolerate it land there.
  • nixoid: a node that’s always on and has about 32GB of RAM. The ClearML server lives here.
  • Tailscale: my private network (a “tailnet”). Services are only reachable through it, which is why you’ll see Tailscale ingresses and ts.net hostnames below.

Swap in your own node names and hostnames if you follow along.

1. Install the server

I used the official clearml Helm chart (7.15.0). It brings an API server, a web server and a file server, plus MongoDB, Elasticsearch and Dragonfly (a Redis replacement). I budgeted about 6GB of RAM for all of it.

  1. Pin everything to nixoid with one YAML anchor:

    apiserver:
      nodeSelector: &nixoid
        kubernetes.io/hostname: nixoid
    fileserver:
      nodeSelector: *nixoid
    
  2. Use local-path for storage. NFS is a poor fit for MongoDB and Elasticsearch.

  3. Make two Secrets and keep them out of git. clearml-secrets holds the seven random keys the chart needs. clearml-apiserver-config holds the web login. The open-source server has no SSO, only fixed users in an apiserver.conf, so the chart reads that Secret through existingAdditionalConfigsSecret.

  4. Give it three Tailscale ingresses: clearml, clearml-api and clearml-files. Tailscale ingress gives one hostname per Ingress, and the SDK talks to the API and file servers directly, not through the web UI.

Problem: Elasticsearch crash loop

The first install left every pod stuck in Init, waiting for Elasticsearch, which was in CrashLoopBackOff. The log ended with:

NullPointerException: Cannot invoke "jdk.internal.platform.CgroupInfo.getMountPoint()"
because "anyController" is null

The chart’s default image is Elasticsearch 7.17.3. It bundles a JDK that can’t read cgroup v2, which NixOS uses. Later 7.17 patch releases fix it.

  1. Set imageTag: "7.17.29".
  2. Flux was still inside the HelmRelease’s first install, waiting up to 15 minutes for readiness, so it wouldn’t apply the new value. I set the image on the StatefulSet by hand to the same value.
  3. Delete the pod.

A few minutes later the HelmRelease finished and reported the committed values.

2. Install the agent

The server only stores tasks. To run them you need the clearml-agent chart (5.3.3). In Kubernetes mode it’s called the k8s glue. It watches a queue and makes a pod for each task.

  1. Create a queue called gpu.
  2. Run the agent on nixoid. It’s light, and only the task pods need the GPU node.
  3. Log into the UI and create a credential pair under Settings. The agent needs one the server issued, so it can’t exist before the server does.
  4. Store the pair in a Secret called clearml-agent-credentials. Until then the agent pod sits in CreateContainerConfigError. That’s expected.
  5. Give each task pod a node selector for nixos-desktop, the toleration for the taint, a nvidia.com/gpu: 1 limit, and runtimeClassName: nvidia.

The chart’s pod template has no field for the runtime class. I used a Flux postRenderers patch that replaces the template ConfigMap wholesale. That means the basePodTemplate values are ignored, and a comment in the values file says where to edit instead. After any chart upgrade I need to diff the rendered template against the patch.

A task pod asks for the whole card, so it stays Pending while one of my other GPU services (HyperQwen, qwen-tts or vibevoice) has it. I scale those to zero first.

Problem: the agent image is from 2022

The agent pod runs allegroai/clearml-agent-k8s-base:1.24-21. It was last built in 2022 on Ubuntu 18.04, so it has Python 3.6. Its entrypoint runs pip install -U clearml-agent on every start, which pulls a release that no longer imports on 3.6:

File ".../clearml_agent/commands/install_bootstrap.py", line 15
    from __future__ import annotations
SyntaxError: future feature annotations is not defined

The entrypoint reads CLEARML_AGENT_UPDATE_VERSION, so I pinned the agent. I tested versions in a throwaway pod on the same image. 1.5.2 through 1.9.3 imported, and 2.0.0 and later didn’t.

Version 1.9.3 then failed against the open-source server in two ways.

First, the chart passes --create-queue, which 1.9.3 doesn’t recognize. I turned createQueueIfNotExists off and made the gpu queue through the API.

The second one was harder:

ERROR: Could not push back task [...] to k8s pending queue k8s_scheduler,
error: Validation error (Cannot skip setting execution queue for a task
that is not enqueued or does not have execution queue set)

The agent checks the server version against 3.22.3, a number from the enterprise versioning. The open-source server reports 2.0, so the check fails and the agent stops the task first. From 1.9.1 on, it then enqueues with update_execution_queue=false, and the server rejects that for a stopped task. Version 1.9.0 predates that call. I pinned 1.9.0 and the hand-off worked.

3. Build the training image

By default a task pod sets itself up on every start. It runs apt-get update, installs git, runs pip install clearml-agent, makes a virtualenv, installs the requirements recorded with the task, and only then runs the script. That’s slow. One run sat Pending for minutes because the Ubuntu mirror was slow.

The requirements step was worse. ClearML records the version of every package the script imports, and it reads them from the machine that submits the task. That’s my laptop, which has the newest of everything. The agent put transformers 5.19 and torch 2.14 into a venv on an image that ships torch 2.5.1, and the task died on import:

AttributeError: module 'torch' has no attribute 'accelerator'

I could have pinned every package in every script. Instead I put the versions in one image and told the agent to install nothing.

  1. Start from pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime, so torch, torchvision and CUDA come from the base.

  2. Add the rest on top:

    FROM pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime
    
    RUN apt-get update \
     && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
          git libsm6 libxext6 libxrender-dev libglib2.0-0 \
     && rm -rf /var/lib/apt/lists/*
    
    COPY requirements.txt constraints.txt /tmp/
    RUN pip install --no-cache-dir -r /tmp/requirements.txt -c /tmp/constraints.txt \
     && out=$(pip check | grep -v '^ninja ' || true) \
     && echo "$out" && test -z "$out"
    

    requirements.txt holds transformers 4.51.3, huggingface-hub 0.30.2, datasets 3.5.0, accelerate 1.6.0, clearml[s3] and a pinned clearml-agent (3.0.3). Those are the versions my fine-tune was written against, all chosen in one place.

    constraints.txt pins torch, torchvision and torchaudio to what the base image already has, so pip can’t replace them. The pip check line fails the build on a real conflict. It ignores ninja, because the base image ships a ninja wheel that pip reports as unsupported on this platform.

  3. Let GitHub Actions build it and push it to GHCR as clearml-training:sha-<commit>.

  4. The package is private, so the clearml namespace needs a pull secret. I copied the one the qwen-tts namespace already uses (not committed) and added imagePullSecrets to the same pod template patch.

  5. Set the image tag in one place: defaultContainerImage in the agent’s values file. The scripts never mention it.

4. Tell the agent to trust the image

Add three environment variables to the task pod template:

env:
  - name: CLEARML_AGENT_SKIP_CONTAINER_APT
    value: "1"
  - name: CLEARML_AGENT_NO_UPDATE
    value: "1"
  - name: CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL
    value: "1"

The first skips the apt step. The second skips pip install clearml-agent. The third skips the virtualenv and the recorded requirements, so the task runs on the image’s own Python.

Nothing gets installed at start now. If a task needs a package that isn’t in the image, it fails instead of installing it. For one image and one GPU box, I think that’s the right trade.

5. Submit a job

A script submits itself. Near the top, execute_remotely records the code, queues the task and exits on my laptop. The agent then runs the same file in a pod on nixos-desktop.

from clearml import Task

Task.force_store_standalone_script()
task = Task.init(project_name="smoke-tests", task_name="pirate-finetune")
task.execute_remotely(queue_name="gpu", exit_process=True)

force_store_standalone_script is there because my scripts live in a private git repo. Without it ClearML records the repo and commit, and the agent has no git credentials, so it fails with could not read Username for 'https://github.com'. Storing the script in the task means I don’t have to give the agent a token.

My first task was a short script that prints nvidia-smi -L and does a matrix multiplication on CUDA. The server’s log shows:

GPU 0: NVIDIA GeForce RTX 3090
cuda available: True
matmul ok: 1.0 on NVIDIA GeForce RTX 3090

A full fine-tune of a 0.5B model, including the model download and three epochs, then ran from submit to done in 6 minutes 23 seconds. The task console has no apt, pip or virtualenv step in it. That run is the subject of the next post.

6. Fix artifact downloads

A fine-tune uploads its model as a task artifact. The UI’s download link opened clearml-fileserver.clearml.svc:8081/..., and Safari couldn’t find the server. The task pod uploads through the in-cluster address, and ClearML stores that hostname in the artifact URL. Swapping in the tailnet hostname got me a 401, because the file server wants an auth token that the web UI adds and a plain browser request doesn’t.

So I stopped using the file server for artifacts. My cluster already runs rustfs, an S3-compatible object store, behind a Tailscale ingress. A pod on the GPU node can reach rustfs.homelab.example.ts.net, and so can my laptop, so one hostname works for both.

  1. Create a clearml bucket in rustfs.

  2. Create a user with a policy scoped to that bucket.

  3. Put this in the agent’s clearml.conf. The agent copies that file into every task pod, so this configures storage once:

    sdk {
      development {
        default_output_uri: "s3://rustfs.homelab.example.ts.net:443/clearml"
      }
      aws.s3.credentials: [{
        host: "rustfs.homelab.example.ts.net:443"
        bucket: "clearml"
        key: "<user>", secret: "<secret>"
        secure: true, multipart: false
      }]
    }
    
  4. Store the config in a Secret and mount it through the chart’s existingClearmlConfigSecret.

Problem: “Could not get access credentials”

The first task after that change failed on its first line:

ValueError: Could not get access credentials for 's3://rustfs...'

That sent me to the config, but the real cause was a line above it in the log: AWS S3 storage driver (boto3) not found. The output URI applies to every task, so every task needs boto3. That’s why the image installs clearml[s3]. The laptop needs it too (pip install "clearml[s3]") if it downloads artifacts.

Things to remember

  • After changing the agent’s configuration, wait until the old agent pod is gone before you submit. During one roll-over the old pod picked up my task and ran it with the old image and none of the skip variables. It looked like the fix had failed.
  • The agent’s startup echoes its access key and secret into the pod log. Anyone who can read logs in the namespace can see them, so treat the credential pair as exposed and rotate it.

Still to do

The agent pod itself still runs the 2022 Python 3.6 image, pinned to agent 1.9.0. If I build my own glue image on a current Python, I can drop the pin and test newer agent releases against the server.

Everything stateful sits on one node with local volumes, so backups of MongoDB and Elasticsearch are next. The open-source server also has no SSO, so logins stay as fixed users.

← Back to articles