Skip to content

Scheduled Cluster Retrain (Flux)

The explainable visitor clusters come from an offline batch trainer (explain/cluster_train.py), which reads the live user models from Redis and writes a clusters.json. The serving API reads that file fresh on every request (CLUSTER_MODEL_PATH), so all that is missing for a self-updating deployment is scheduling the trainer. This page wires that up with Flux, using the existing memorise-project/flux GitOps repo.

flowchart LR
    git["gitlab.sdu.dk/memorise-project/flux"] -->|reconcile 1m| flux["Flux (memorise-apps)"]
    flux --> cj["CronJob: cluster-train<br/>(ns ai-engine)"]
    cj -->|reads user models| redis[("recsys-redis")]
    cj -->|writes clusters.json| pvc[("cluster-models PVC")]
    pvc -->|read per request| api["ai-engine-api"]

How this repo works

The memorise-apps Kustomization (clusters/memorisev2/apps-kustomization.yaml) already reconciles everything under clusters/memorisev2/applications/ every minute, with prune: true, from the flux-system GitRepository. So you do not add a new Flux Kustomization: you just drop YAML files into the app directory and commit. The recsys app lives at clusters/memorisev2/applications/ai-engine/ (namespace ai-engine), with:

  • ai-engine-api/deployment.yaml (image ghcr.io/ai-engine-memorise/ai-engine-recsys, Flux image automation, pull secret ghcr-read),
  • redis/ (service recsys-redis, REDIS_URL: redis://recsys-redis:6379/0 in api-config.yaml),
  • ai-engine-api/pvc.yaml (storageClassName: longhorn).

Prerequisite: the trainer must be in the image

Dockerfile.recsys currently does COPY src ./src, so the root script explain/cluster_train.py is not in the runtime image. Add it (one line), rebuild, and let Flux image automation roll the new tag:

COPY src ./src
COPY explain ./explain     # ships cluster_train.py into the image

(Alternatively move cluster_train.py under src/ai_engine/recsys/explain/ and run it with python -m ai_engine.recsys.explain.cluster_train.)

Deployed variant (2026-07-16): RWO on the api PVC, not RWX

The single-node cluster cannot mount Longhorn RWX exports (the node lacks the NFS mount helper), and with one node an RWX volume is unnecessary anyway: the CronJob mounts the existing ai-engine-api-pvc at /app/logs and writes --out /app/logs/models/clusters.json; CLUSTER_MODEL_PATH points there. No extra PVC, no deployment changes. The RWX recipe below applies only if the cluster grows past one node (then also install nfs-common on the nodes).

Step 1: shared model volume (RWX)

The trainer writes clusters.json while the API pod holds the volume, so it must be ReadWriteMany. Longhorn provisions RWX via a share-manager. Commit applications/ai-engine/cluster-train/pvc.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cluster-models
  namespace: ai-engine
spec:
  storageClassName: longhorn
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 100Mi

Step 2: mount it into the API + set the path

The serving pod reads CLUSTER_MODEL_PATH. Add the value to api-config.yaml:

# clusters/memorisev2/applications/ai-engine/api-config.yaml (ConfigMap data:)
  CLUSTER_MODEL_PATH: "/models/clusters.json"

and mount the PVC in ai-engine-api/deployment.yaml (alongside the existing /app/logs and /app/config mounts):

          volumeMounts:
            - { mountPath: /models, name: cluster-models }     # add
      volumes:
        - name: cluster-models                                 # add
          persistentVolumeClaim: { claimName: cluster-models }

No pod restart is needed after a retrain: _load_cluster_model re-reads the file on each request, so a freshly written clusters.json goes live immediately.

Step 3: the CronJob

Commit applications/ai-engine/cluster-train/cronjob.yaml. It reuses the recsys image and the same REDIS_URL from api-config:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: cluster-train
  namespace: ai-engine
spec:
  schedule: "0 4 * * *"            # daily 04:00 (min hour dom mon dow)
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          imagePullSecrets:
            - name: ghcr-read
          restartPolicy: OnFailure
          containers:
            - name: train
              image: ghcr.io/ai-engine-memorise/ai-engine-recsys:0.6.9 # {"$imagepolicy": "flux-system:ai-engine-api"}
              workingDir: /app
              command: ["python", "explain/cluster_train.py",
                        "--method", "fcm", "--k", "4",
                        "--out", "/models/clusters.json"]
              envFrom:
                - configMapRef: { name: ai-engine-api-config }   # REDIS_URL
              volumeMounts:
                - { mountPath: /models, name: cluster-models }
              resources:
                requests: { cpu: "100m", memory: "256Mi" }
                limits:   { cpu: "500m", memory: "512Mi" }
          volumes:
            - name: cluster-models
              persistentVolumeClaim: { claimName: cluster-models }

Keeping the same # {"$imagepolicy": ...} marker means Flux bumps the CronJob's image tag together with the API, so the trainer never drifts from the running release.

Step 4: commit and let Flux reconcile

git add clusters/memorisev2/applications/ai-engine/cluster-train/ \
        clusters/memorisev2/applications/ai-engine/api-config.yaml \
        clusters/memorisev2/applications/ai-engine/ai-engine-api/deployment.yaml
git commit -m "ai-engine: scheduled cluster retrain"
git push

memorise-apps reconciles within a minute (prune: true also removes it cleanly if you delete the files later). From now on the schedule and --k live in git.

Verify and operate

flux get kustomizations                                   # memorise-apps Ready
flux reconcile kustomization memorise-apps --with-source  # force a sync now

kubectl -n ai-engine get cronjob cluster-train
kubectl -n ai-engine create job --from=cronjob/cluster-train cluster-train-now   # run once
kubectl -n ai-engine logs job/cluster-train-now

curl -s "$API/api/clusters" | jq '.result.method, (.result.clusters | length)'

null from /api/clusters means the model file is not there yet: the CronJob has not run, or the API pod is missing the /models mount / CLUSTER_MODEL_PATH.

Notes

  • Multi-tenant. Clusters are per tenant. Run one CronJob per tenant (different --out and per-tenant cluster_model_path), or extend the trainer to loop over tenant Redis namespaces.
  • RWX on longhorn spins up a share-manager pod; if your cluster prefers it, a small NFS or object-store-backed class works too. The key requirement is that the CronJob and the always-on API pod can hold the same volume at once (so ReadWriteOnce is not enough).
  • Cold data. Clustering needs a corpus of user models in Redis; early runs may produce few or empty clusters until visitors accumulate.