Multi-tenancy¶

One engine serves many clients (UIs / memorial sites), each hard-isolated: its own Qdrant collection (content cannot cross), its own Redis key-prefix (user models, events, impressions, config cannot cross), and its own bandit policy and cluster model. Shared infra, logical isolation, the standard "pooled" multi-tenant model.

The engine core (recommender, scorers, user model) is tenant-agnostic: it operates on whatever stores it is handed. Tenancy is purely an edge concern, implemented in recsys/tenancy.py and composition.ComponentManager.

How a request is routed¶

flowchart LR
    req["request<br/>X-Tenant-Id: id"] --> mw["TenantASGIMiddleware<br/>sets current_tenant contextvar"]
    mw --> proxy["TenantProxy<br/>resolves c.attr"]
    proxy --> cm["ComponentManager.get(id)"]
    cm --> comp["tenant-scoped Components"]

Every request and event carries X-Tenant-Id. A pure-ASGI middleware reads the header and sets a contextvar; a TenantProxy resolves c.<attr> to that tenant's Components, so the API endpoints stay unchanged. No header means the default tenant (single-tenant behaviour, unchanged).

Why pure ASGI

A pure ASGI middleware runs in the same context as the request, so the contextvar propagates into the (threadpool-run) endpoint. Starlette's BaseHTTPMiddleware runs the endpoint in a separate task and would lose it.

What is isolated per tenant¶

Axis	Keying
Content catalogue	its own Qdrant `collection`
User models / events / impressions	Redis prefix `{tenant}:umodel` / `evt` / `imp`
Runtime config (settings page)	Redis key `{tenant}:recsys:config`
Bandit policy (θ)	`bandit_state_path` per tenant
Cluster model	`cluster_model_path` per tenant
Durable event log	`EVENT_LOG_DIR/{tenant}/...`

Redis-backed stores take a key_prefix constructor argument; the composition root builds it from the tenant's prefix, so two tenants on the same Redis never share a key.

Registering tenants¶

Two paths, both durable. Pick by how fixed the tenant is:

Path	How	Redeploy?	Durability
Runtime (normal)	`/admin` → Tenants → Add, or `POST /api/tenants`	No	PVC file `TENANT_STORE_PATH`
Baseline (fixed)	add to `TENANTS_PATH` JSON, commit	Yes	git-backed configmap

New clients are onboarded at runtime: no redeploy. The baseline holds only the truly-fixed tenants (default). The two are merged at request time; the runtime store wins.

Runtime store (`TENANT_STORE_PATH`)¶

Tenants created via /admin are written to a JSON file on the api PVC, so they survive pod restarts and a Redis wipe/LRU: /admin is the permanent, zero-redeploy onboarding path. The backing is chosen by env: TENANT_STORE_PATH (durable PVC file) → Redis → in-memory. On the SDU deploy recsys-redis is intentionally ephemeral (--save "" + LRU), so the file store is what makes runtime tenants permanent.

Baseline file (`TENANTS_PATH`)¶

{
  "default": "default",
  "tenants": [
    {"tenant_id": "default", "collection": "omeka-items"}
  ]
}

Each entry maps to a TenantSpec: tenant_id, collection, optional redis_prefix (defaults to tenant_id), bandit_state_path, cluster_model_path, and config_overrides. default must be listed (else it loses its collection and falls back to fakes).

No TENANTS_PATH: a single default tenant built from the existing env (COLLECTION_NAME, BANDIT_STATE_PATH, CLUSTER_MODEL_PATH). Existing deployments are unchanged.
Unknown tenant id (not in baseline or runtime store): an auto-isolated slice (its own Redis prefix, inheriting the default catalogue and config).

The contract each UI must honour¶

user_id only means something within a tenant; content_id only resolves within that tenant's collection. So every UI must:

send a consistent X-Tenant-Id on every request and event, and
agree on the user_id / content_id namespaces for that tenant.

The serving calls are unchanged apart from that one header, e.g. recommend:

GET /api/recommend?user_id=<uid>
X-Tenant-Id: <tenant>

By default /api/recommend is not key-guarded; X-API-Key is required on /api/ingest and the other guarded routes. See Authentication to lock reads too.

Authentication and per-tenant keys¶

The TenantASGIMiddleware is the trust boundary: it resolves the tenant so a client header can't spoof it:

A valid per-tenant key (X-API-Key) → tenant taken from the key; any X-Tenant-Id is ignored. A tenant's key can only ever touch its own slice.
The global INGEST_API_KEY → superuser; tenant from X-Tenant-Id (admin / ops).
No / unknown key → public reads only; guarded routes 401.

Route group	Guard
`/api/recommend`	public by default; `SERVING_REQUIRES_KEY=1` requires a per-tenant key, then the key alone identifies the tenant and `X-Tenant-Id` is not needed
`/api/ingest`, `/api/usermodel`, `/api/metrics`, `/api/config`, `/api/tenants`	always require a key
`/api/search*`	public (separate router; unaffected by the flag)

Keys at rest are sha256 hashes (api_key_hashes), never plaintext, in the durable tenant store (TENANT_STORE_PATH), not git, not Redis. GET /api/tenants shows a count only, never the key.

Issue a key: server mints it, returns it once, stores only the hash:

curl -X POST $API/api/tenants -H "X-API-Key: $GLOBAL_KEY" -H "Content-Type: application/json" \
  -d '{"tenant_id":"westerbork-ar-ai","collection":"westerbork-ar-ai","generate_api_key":true}'
# -> { "api_key": "<256-bit token>", "note": "store now — only its hash is kept" }

In locked mode the UI sends only the key on serving calls:

fetch(`${API}/api/recommend?user_id=${uid}`, { headers: { "X-API-Key": TENANT_KEY } })

Tooling¶

The control panel at /admin (also /inspector) manages tenants (list / add / delete, set collection, content counts) and per-tenant config, and has a tenant field that sends X-Tenant-Id, so you can inspect or configure one client at a time.

Known limits (v1)¶

In-process /api/metrics counters are global, not per-tenant. Fine for ops, not for per-client billing. Prometheus labels are the production upgrade.
The demographics provider is shared; per-tenant DB wiring is future work.