Fine-tuning means taking a model that already works and training it a bit more so it behaves differently. In this post I fine-tune a very small language model to answer questions like a pirate, on the GPU in my home Kubernetes cluster, and then run the result on my Mac.

It’s a silly task, and this post is dated for International Talk Like a Pirate Day, 19 September. It still tests the whole path: a script goes into a job queue, a pod trains on the GPU, the model comes back, and I chat with it locally.

What you need

  • A GPU you can send jobs to. I use an RTX 3090 on a home k3s cluster, with ClearML as the job queue. The previous post covers that setup. If you have a different GPU box, the training script itself is still a normal Hugging Face fine-tune.
  • A Mac with Apple silicon for the last step.
  • Python and pip on your laptop.

1. Pick the model and the data

The model is Qwen/Qwen2.5-0.5B-Instruct. At half a billion parameters it trains fully in fp32 on a 24GB card, so no LoRA and no quantization.

I didn’t want a second model just to make training data, so I faked it:

  1. Take tatsu-lab/alpaca (52,002 examples), a public Stanford dataset of instructions and answers, like “Give three tips for staying healthy” and a reply. Each row has an instruction, an optional input and an output. It’s on Hugging Face.
  2. Keep the ones with no extra input and an answer between 20 and 400 characters.
  3. Shuffle and keep 1,500.
  4. Rewrite each answer with a few regex rules, plus a pirate opener and closer.
WORDS = [
    (r"\byou are\b", "ye be"), (r"\byour\b", "yer"), (r"\byou\b", "ye"),
    (r"\bmy\b", "me"), (r"\bis\b", "be"),
    (r"\bfood\b", "grub"), (r"\bmoney\b", "doubloons"),
    # ...
]
OPEN = ["Arr!", "Ahoy, matey!", "Yo ho ho!", "Shiver me timbers!", "Avast!"]

def pirate(text, rng):
    for pat, rep in WORDS:
        text = re.sub(pat, rep, text, flags=re.I)
    text = re.sub(r"ing\b", "in'", text)
    return f"{rng.choice(OPEN)} {text.strip()} {rng.choice(CLOSE)}"

It’s crude, but the model only has to learn the style.

2. Set the training options

  • 3 epochs, batch size 8
  • Learning rate 2e-5, cosine schedule, 5% warmup
  • 564 steps in total
  • Loss masked on the prompt, so the model only learns the answers

The script also asks five fixed questions before and after training and logs both sets of answers to the task.

3. Make the script submit itself

Near the top of the script, execute_remotely records the code, queues the task and exits on my laptop. The agent then runs the same file in a pod on nixos-desktop.

from clearml import Task

Task.force_store_standalone_script()
task = Task.init(project_name="smoke-tests", task_name="pirate-finetune")
task.execute_remotely(queue_name="gpu", exit_process=True)

The script pins no packages and names no image. The task pod uses a training image I built for the cluster (see the ClearML post), which already has transformers 4.51.3, datasets, accelerate, huggingface-hub, boto3 and the base image’s torch 2.5.1. The agent installs nothing at start.

My first two attempts didn’t work this way. ClearML records package versions from the machine that submits the task, so the agent tried to install my laptop’s transformers 5.19 and torch 2.14 over the image’s, and the run died on import. The ClearML post has the cause and the fix. Once the image and agent settings were in place, the same script ran unchanged.

4. Save the model as an artifact

The script ends by saving the model in bf16 and calling task.upload_artifact("pirate-model", ...). That zips the folder and uploads it. The artifact is 751MB and the upload to rustfs (an S3-compatible store that runs on my cluster) took about a minute.

The UI link works from my laptop because artifact storage is on a hostname my laptop can reach over Tailscale (also covered in the ClearML post). Download it and unzip it, and you have an ordinary Hugging Face model folder.

5. Check the result

From submit to done took 6 minutes 23 seconds. Training itself was 103 seconds at about 5.5 steps per second. The rest went on downloading the model and data, generating the before and after answers, and uploading.

The loss started at 3.06, fell to about 1.3 by the end of the first epoch, about 0.6 in the second and about 0.25 in the third.

Generation is capped at 100 tokens, so some answers stop mid-sentence. Here are three of the five prompts:

Q: What is the capital of France?
BEFORE: The capital of France is Paris.
AFTER:  Ahoy, matey! th' capital o' France be Paris. Savvy?

Q: How do I boil an egg?
BEFORE: Boiling an egg is a simple and effective way to cook it. Here's how you
        can do it: 1. **Prepare the Egg**: Make sure your egg is clean and free
        of any dirt or yolk. ...
AFTER:  Yo ho ho! t' boil an egg, start by fillin' a pot wit' cold water an'
        addin' th' egg. Brin' th' pot t' a boil an' set a timer fer th' desired
        level o' doneness (e.g. 3 minutes fer a soft cooked egg, 8 minutes fer
        a hard-boiled egg). ...

Q: What's the difference between a list and a tuple in Python?
BEFORE: In Python, both lists and tuples are used to store collections of
        items, but they have some key differences: 1. **List**: 2. **Tuple**:
AFTER:  Arr! th' main difference between a list an' a tuple in Python be that
        lists be mutable (able t' be changed after they be created) while
        tuples be immutable (cannot be changed after they be created). ...
        Savvy?

The style change is clear. Every answer gets an opener and a closer, and the word swaps carry through. The facts are a different story: in the full answer the model says tuples can only hold one data type, which is wrong. A 0.5B model does that with or without a pirate accent.

6. Run it on the Mac

I tried Ollama first, with a Modelfile that has FROM /path/to/pirate-model, then ollama create. It failed:

importing safetensors model
Error: unsupported MLX architecture: model "Qwen2ForCausalLM"

My version of Ollama on macOS tries to import safetensors through its MLX path, and that doesn’t support Qwen2. The usual workaround is converting to GGUF with llama.cpp first. I skipped that and used mlx-lm, which loads the Hugging Face folder as it is.

  1. Install it:

    pip install mlx-lm
    
  2. Run it:

    mlx_lm.generate --model /path/to/pirate-model --prompt "How do I boil an egg?"
    

No conversion step. mlx_lm.chat gives you a chat loop and mlx_lm.server gives you an OpenAI-style endpoint, both from the same folder.

What I’d change

The regex pirate-ifier sets the quality ceiling. A real model writing the answers would sound more natural, but then the GPU has to do data generation too. The regex also leaves marks, like a lowercase “th’” at the start of a sentence, and the tuned model picked up odd words of its own, like “nayzzin’” in the sleep answer.

I picked 1,500 examples and three epochs as a guess and haven’t tried other sizes. The loss drops in a step at the start of each epoch, which usually means the model is memorizing the examples as well as learning the style. Before I trust three epochs, I’d compare against a model trained for one.

The script and the image definition are in ~/code/clearml/.

← Back to articles