Skip to main content
The experiment-tracking core — API plus Postgres, with metrics and artifacts on your infrastructure — from the same Helm chart the managed service runs. No dashboard, no vendor callback. The software runs entirely on your infra and never contacts us. Ordinary external egress — image pulls, an approved LLM provider — remains your policy choice, not ours.

​
Requirements

The default install runs with the knowledge engine off, which is the simplest complete topology. Turning it on is a helm upgrade with one extra values file.
Every step below is exercised nightly in CI against the same flow. This is the human transcription of a tested path, not a description of one.

​
What engine-off means

  • /v1/search returns state: "partial" with SQL-exact results only. The exact channel matches projects, experiments and artifacts — not run names.
  • AI summaries stay dormant and are badged STALE; they need an LLM gateway.
  • No dashboard ships. Clients are machines using service-minted bearer tokens, so no SSO or OIDC integration is required — those configuration keys can be placeholders.

​
Install

1

Namespace and configuration, first

Create the namespace, then the chart’s configuration Secret holding your object-store endpoint, bucket, and the two random values the app signs with.Keep createNamespace: false in your values. The ordering is deliberate: it removes the install race, because the API pod mounts that Secret at boot — and it means helm uninstall can never delete it, since the namespace is not chart-owned.
2

Values

Start from the continuously-tested overlay in tests/selfhost/ and adapt the stand-ins.
3

Install, without --wait

Do not pass --wait on first install. The schema-migrate Job is a post-install hook while the API’s readiness probe requires the schema — --wait waits for readiness before running hooks and deadlocks until timeout.Plain helm install blocks only on the migrate hook; the rollout status poll covers API readiness. Upgrades are immune, because migrate is a pre-upgrade hook there.
Healthy looks like: migrate Job Completed (control and experiment applied, kb intentionally skipped), the research-pg CNPG cluster healthy, the API pod Ready, the upload-reaper CronJob registered, and /healthz returning 200 with all three databases green.
4

Seed the first team and credential

A fresh install has no users. Seed one team, one owner and one API credential by SQL in the CNPG primary; the app auto-creates personal workspaces on demand, so seed nothing else.
  1. experiment DB: the customers mirror row — first, matching the app’s own team-create ordering.
  2. control DB: customers, users, team_members (role owner), and an api_tokens row holding only the SHA-256 digest of the credential string.
Generate the random credential yourself and hand the plaintext to the client. It is never stored. tests/selfhost/seed.py is the executable reference.
Run SQL with kubectl exec -i. Without -i, psql silently no-ops — it exits 0 having done nothing.
Verify against /v1/me, which is the machine endpoint. /auth/me is the dashboard’s session-only one.
5

Verify

The smoke suite’s gates, by hand:
  • /healthz returns 200 with three databases green
  • /v1/me returns the seeded identity
  • create project, then experiment (a question is required), then run
  • post metrics and read them back
  • upload an artifact: presign, PUT, confirm
  • POST /v1/search returns state: "partial" with an exact hit — the expected engine-off shape

​
TLS

Let’s Encrypt HTTP-01 works only if the API host is publicly resolvable, which is usually not true for a residency install.

​
Turning the engine on

Semantic search is a helm upgrade with one extra values file layered on top of your engine-off values. The API rolls and the engine’s kb-migrate runs as a pre-upgrade hook. Afterwards /v1/search is no longer "partial": a semantic channel is populated alongside the SQL exact one.

​
Backups and recovery

Continuous WAL archiving to object storage via Barman gives point-in-time recovery with an RPO of five minutes or better. It is a values flip, with its credentials Secret named after the cluster.
Use a Retain reclaim storage class. pgdata is the system of record, and a Delete-class volume removed with the release takes it.

​
Upgrades

Pin the image tag; never float it. Migrations run as a pre-upgrade hook, so --wait is safe on upgrade.
A pinned engine image pins the kb schema with it — kb-migrate runs alembic from that image. Bumping the app without the engine leaves kb migrations unapplied.

​
Air-gapped

  • Record artifacts as references so bytes stay in your own bucket.
  • Sign in with probe login --token or --endpoint-only, or set PROBE_TOKEN and PROBE_BASE_URL.
  • probe wizard --action manual prints the exact commands for setting agents up without a network-assisted install.

​
Compliance

There is no vendor callback. Analytics egress is a values flag and off in this topology, so a compliance review can verify the claim by reading the chart rather than trusting a statement about it.