Upgrading

helm upgrade edisyl \
  oci://944311261080.dkr.ecr.us-east-1.amazonaws.com/edisyl-ga/charts/edisyl-helm \
  --version <new-chart-version> \
  -n edisyl \
  -f edisyl.values.yaml \
  -f secrets.local.yaml \
  -f auth-keys.local.yaml \
  --wait --wait-for-jobs \
  --timeout 20m

Pin --version, pass the same -f files as the install (with edisyl.overrides.yaml after edisyl.values.yaml if you have one), and keep the long timeout, exactly as on installing. This is the command for every upgrade, including one that only changes deploymentIdp.

--wait behaves differently here than on a first install. The migration runs as a pre-upgrade hook, so it completes before the deployments are updated — there is no ordering deadlock to avoid. With --wait-for-jobs added, Helm also waits for the edisyl-idp-reconcile Job that registers your identity provider, so a failed registration fails the upgrade instead of passing silently — see Authentication. --atomic is still worth avoiding: a rollback reverts the manifests while leaving the migrated database as it is, which is the mismatch described below.

Before you upgrade

Update your image tags to match the chart version. Chart version and image tags move together; the version matrix publishes the tag set for each release, with a values block ready to copy. A new chart with old images is not a combination we test.

Take a database backup. Upgrades run migrations, and migrations are not reversible by re-installing the older chart. The application database is the one migrations touch; back up the others too before a chart that changes how they are provisioned.

Skip chart 0.49.0 and 0.50.0

With secrets.provider: k8s-native, do not install or upgrade onto chart 0.49.0 or 0.50.0. Their API images refuse the ENCRYPTION_KEY those charts generate, and the API crash-loops. Chart 0.51.0 fixed it; go straight to 0.51.0 or later.

What happens

The migration hook runs to completion first, then the deployments roll. Expect a short window where old pods are still serving against an already-migrated database, so keep migrations backward-compatible with the release you are leaving — or accept a brief outage.

The LLM gateway runs a single replica, so every upgrade that touches it has a short window in which inference fails — see Known limitations.

Upgrading from 0.62

Chart 0.63.0 pins the same application images as 0.62.0 and changes two things about the bundled Inngest.

1. Run history older than seven days is deleted. A nightly CronJob, inngest-retention, prunes Inngest’s Postgres database, starting the first night after the upgrade. To keep more, set inngest.retention.days before upgrading; to keep everything, inngest.retention.enabled: false. On a database that has never been pruned the first run can outlast its three-hour limit, and the next night carries on where it stopped. See Inngest run history.
2. Step metadata is off. The chart’s inngest.inngest.extraEnv gains EXPERIMENTAL_STEP_METADATA: "false". A values file that overrides that list replaces it, so add the entry to yours, and keep INNGEST_POSTGRES_URI a secretKeyRef or the pruning job does not render.

Upgrading from 0.57

Charts 0.58.0 to 0.60.0 name the organization an install runs as, stop loading demo data, and give each bundled database its own cluster. Go straight to 0.60.0.

1. Set bootstrap.organization. The first sync creates the organization from bootstrap.organization.name and .slug, and helm install, helm template and Argo CD refuse to render without them. Plain helm upgrade renders without them and an existing install ignores them, but set them anyway. An install first created by an earlier chart already runs as Edisyl (slug edisyl); under Argo CD set exactly that.
2. Drop bootstrap.seed. It is gone: true fails the render. Upgrades with --reuse-values keep the old default of true, so use --reset-then-reuse-values instead.
3. Remove the demo data an earlier chart loaded. Charts before 0.58.0 ran a development seed on install. It created Edisyl staff accounts as owners, two demo organizations, and service API keys with guessable values. Revoke those keys first, then delete the staff accounts and the demo organizations in the app — see Removing the seed data.
4. Decide where Inngest keeps its run history. inngest.database.source is new. Unset, it follows cnpg.enabled: the bundled inngest-db cluster, or SQLite on the pod’s scratch volume without one. With your own Postgres, set existing-secret and create Secret edisyl-inngest-db (key uri) before upgrading — see The Inngest database. A values file that overrides inngest.inngest.extraEnv must keep an INNGEST_POSTGRES_URI entry, or the render refuses it. From 0.63.0 add an EXPERIMENTAL_STEP_METADATA entry too, or Inngest’s step metadata comes back on — see Upgrading from 0.62.
5. On the bundled clusters, reinstall rather than upgrade. 0.60.0 runs one CloudNativePG cluster per database (services-db, auth-db, inngest-db, bifrost-db) in place of the single edisyl-pg, and carries no data over. A helm upgrade with database.source: cnpg hangs in the migration hook. pg_dump each database you need (edisyl, mono_auth, bifrost), helm uninstall, delete the leftover volumes (kubectl delete pvc -n edisyl -l cnpg.io/cluster=edisyl-pg), install 0.60.0 and restore into the new clusters. The single-cluster keys (cnpg.instances, storageClass, storageSize, walStorageSize, resources, parameters) move under cnpg.clusters.<database>, and cnpg.name, appSecretName, managedRoles and extraDatabases are gone; the render refuses each while cnpg.enabled. With your own Postgres nothing here applies.

The configuration reference lists the status of every value and credential, including what was removed in which version.

Removing the seed data

Revoke the seed’s keys before renaming any user or organization: the match below is computed from the current username and slug. List them with psql against the application database:

SELECT c.id, c.token_prefix, u.email, o.slug, c.created_at, c.last_used_at
  FROM credentials c
  JOIN users u ON u.id = c.created_by_id
  JOIN organizations o ON o.id = c.organization_id
 WHERE c.revoked_at IS NULL AND c.kind = 'service'
   AND c.token_hash IN (
     encode(sha256(convert_to(u.username || '-admin', 'UTF8')), 'hex'),
     encode(sha256(convert_to(u.username || '-admin-' || o.slug, 'UTF8')), 'hex'));

A last_used_at that is set means someone used that key; review the API’s access logs and the organization’s audit log from its created_at onward. Then revoke them:

UPDATE credentials c SET revoked_at = now(), updated_at = now()
  FROM users u, organizations o
 WHERE u.id = c.created_by_id AND o.id = c.organization_id
   AND c.revoked_at IS NULL AND c.kind = 'service'
   AND c.token_hash IN (
     encode(sha256(convert_to(u.username || '-admin', 'UTF8')), 'hex'),
     encode(sha256(convert_to(u.username || '-admin-' || o.slug, 'UTF8')), 'hex'));

Then delete the seeded @edisyl.com staff accounts in the app; revoking their keys is not enough, because a sign-in with one of those addresses takes over the account and its owner standing. Leave system+…@edisyl.com accounts alone: they are platform identities the sync recreates. Remove the Acme Foobar and Demo organizations last, and rename Edisyl to your own. A seed Job that failed stays in the namespace: kubectl -n edisyl delete job edisyl-seed --ignore-not-found.

Upgrading from 0.52

Charts 0.53.0 to 0.57.0 remove the lattice app, admit only one role to the auth service’s admin API, make the email sender optional and rename two store keys. Go straight to 0.57.0, in this order.

1. Replace the admin token if its role is not mono_api. From 0.55.0 the auth service admits only role: mono_api to its admin API. A token minted as service_role, which earlier versions of the minting scripts did, gets 401, the edisyl-idp-reconcile Job fails and so does the upgrade. Check the role and re-mint while still on 0.52.0, which accepts both — see Replacing the admin token.
2. Under external-secrets, add the auth keys under their new names. From 0.57.0 the store holds AUTH_ANON_KEY and AUTH_ADMIN_TOKEN, not AUTH_SELFHOSTED_ANON_KEY and AUTH_SELFHOSTED_ADMIN_TOKEN. Add both with the same values (the new admin token, if step 1 replaced it), and keep the old names until no environment can roll back below 0.57.0. The render refuses a remoteKeys entry on edisyl-auth-client or edisyl-auth-admin that maps to an AUTH_SELFHOSTED_ name, and a keys list there (such as the old default keys: []) that leaves the key out with no other remoteKeys entry supplying it. Remove those; a mapping to a store name of your own still works. See Secrets and signing keys.
3. Remove lattice from your values. The chart no longer renders the lattice app, and refuses the secrets.managed.edisyl-lattice and edisyl-redis-lattice groups and any envFrom override that still names them. Remove those blocks, edisyl.lattice and any lattice image pin. LATTICE_API_KEY and LATTICE_REDIS_URL are read by nothing; delete them from your store once no environment can roll back below 0.53.0. The upgrade deletes the lattice Deployment and its Secrets.
4. Decide about email. config.emailFrom may be blank from 0.56.0: the API then sends no email, and an invitation shows a link for its owner to share. config.emailEnabled: true with a blank sender is refused, and the API does not read config.emailEnabled at all; the sender alone turns email on.
5. Take a backup. The 0.53.0 images run a migration that drops the fleet, team and Lattice tables and every row in them.

A blank sender needs 0.56.0

Running without email needs API and web images prd-9c0dff5 or later, which 0.56.0 pins first. Chart 0.53.1 accepts a blank config.emailFrom but pins an API image that refuses to start without one, and 0.54.0 and 0.55.0 require the sender again. Keep config.emailFrom set until you are on 0.56.0 or later.

Upgrading from 0.36 or earlier

Charts 0.38.0 to 0.52.0 changed what a values file must say, and the section above covers 0.53.0 to 0.57.0. A values file that installed 0.36.0 is refused by a current chart several times over, and each refusal names the key to change. Work through this section before the upgrade rather than one refusal at a time.

Values that moved or went away

Each old path left in place fails the render, except the dropped groups at the end, which render Secrets nothing reads. Under external-secrets, apply the same move to a group’s keys and remoteKeys as to its data.

Old path	New path	Since
—	deploymentIdp.metadataUrl or deploymentIdp.metadataXml, and deploymentIdp.domains — required; see Authentication	0.38.0
config.staffSsoProviderId	Remove. The API trusts the provider the IdP Job records	0.40.0
secrets.managed.edisyl-cloudflare.data.CLOUDFLARE_R2_ACCESS_KEY_ID, CLOUDFLARE_R2_SECRET_ACCESS_KEY	secrets.managed.edisyl-object-storage.data.OBJECT_STORAGE_ACCESS_KEY_ID, OBJECT_STORAGE_SECRET_ACCESS_KEY	0.42.0
OBJECT_STORAGE_PROVIDER: r2	config.objectStorageProvider: s3compatible and config.objectStorageUrl: https://<account-id>.r2.cloudflarestorage.com	0.42.0
secrets.managed.edisyl-object-storage.data.OBJECT_STORAGE_PROVIDER	config.objectStorageProvider	0.46.0
…edisyl-object-storage.data.OBJECT_STORAGE_BUCKET	config.objectStorageBucket — always required	0.46.0
…edisyl-object-storage.data.OBJECT_STORAGE_REGION	config.objectStorageRegion	0.46.0
…edisyl-object-storage.data.OBJECT_STORAGE_PUBLIC_URL	config.objectStoragePublicUrl	0.46.0
…edisyl-object-storage.data.OBJECT_STORAGE_FORCE_PATH_STYLE	config.objectStorageForcePathStyle	0.46.0
…edisyl-object-storage.data.OBJECT_STORAGE_ENDPOINT	config.objectStorageUrl	0.46.0
secrets.managed.edisyl-redis.data.REDIS_URL	Remove	0.46.0
…edisyl-redis.data.REDIS_HOST, REDIS_PORT, REDIS_USER, REDIS_TLS	config.redisHost, config.redisPort, config.redisUser, config.redisTls	0.46.0
secrets.managed.edisyl-ses.data.SES_FROM_EMAIL	config.emailFrom — required until 0.56.0, optional since	0.46.0
secrets.managed.edisyl-sentry.data.SENTRY_DSN_API, _WORKER, _EDISYL	config.sentryDsnApi, config.sentryDsnWorker, config.sentryDsnEdisyl	0.46.0
secrets.managed.edisyl-github.data.GITHUB_APP_ID	config.githubAppId	0.46.0
secrets.managed.edisyl-database.data.DATABASE_POOL_SIZE	config.databasePoolSize	0.46.0
secrets.managed.edisyl-supabase, edisyl-realtime, edisyl-ses, edisyl-sentry	Remove the whole group	0.46.0
edisyl.authSource	Remove. Auth is always the chart’s own service	0.46.0
edisyl.auth.enabled: false	Remove; it must be true, the default	0.46.0
edisyl.auth.google, and GOTRUE_EXTERNAL_GOOGLE_CLIENT_ID / _SECRET in edisyl-auth	Remove. SAML is the only sign-in path	0.46.0
secrets.managed.edisyl-llm-gateway.data.LLM_GATEWAY_URL	Remove. Derived from bifrost.enabled or bifrost.external.url	0.46.0
secrets.managed.edisyl-llm-gateway.data.LLM_GATEWAY_PROVIDER	Remove	0.47.0
—	bifrost.database.source and its connection — required, since the gateway always runs	0.47.0
config.portApi	Remove. The API listens on edisyl.api.port	0.49.0
secrets.managed.edisyl-rudderstack.data.RUDDERSTACK_DATA_PLANE_URL	config.rudderstackUrl — required with config.rudderstackEnabled	0.49.0
secrets.managed.edisyl-mono-api-key.data.MONO_API_KEY	secrets.managed.edisyl-mono-api-key.data.MONO_API_KEYS	0.49.0
secrets.managed.edisyl-auth-service-key, edisyl-provider-webhook, edisyl-snowflake, edisyl-resend, edisyl-integrations-social, edisyl-hubspot	Remove the whole group. Nothing reads it	0.42.0

App envFrom lists. If you override edisyl.<app>.envFrom.secrets, remove every group you removed above from it — a list that still names edisyl-supabase, edisyl-ses, edisyl-sentry or edisyl-realtime is refused, because the pod would never start. Helm replaces a list wholesale, so an override also misses groups added since, such as edisyl-typesafe; drop the override if you can, or rebuild it from the current chart’s default.

Secrets and the store

• MONO_API_KEYS. MONO_API_KEY became MONO_API_KEYS, a comma-separated list the API accepts any entry of and every caller sends the first of. Under k8s-native the generated value carries over on its own. An external store needs MONO_API_KEYS, with the same value, before you upgrade. Keep MONO_API_KEY in the store until no environment can roll back below 0.49.0.
• TypeSafe. From 0.50.0 the API mounts edisyl-typesafe, and under external-secrets the store must carry TYPESAFE_API_KEY and TYPESAFE_BASE_URL — or set secrets.managed.edisyl-typesafe.keys: [] to leave the group out. A missing key fails the ExternalSecret, and the API cannot start without it.
• Auth. The store names for the auth keys are AUTH_ANON_KEY and AUTH_ADMIN_TOKEN (AUTH_SELFHOSTED_ANON_KEY and AUTH_SELFHOSTED_ADMIN_TOKEN before 0.57.0), alongside GOTRUE_JWT_KEYS, GOTRUE_JWT_SECRET and GOTRUE_SAML_PRIVATE_KEY. AUTH_ADMIN_TOKEN must carry role: mono_api. An install moving off an external auth project also needs a database for the auth service. That project’s users and sessions do not move: everyone signs in again. Afterwards, revoke the old project’s service-role key.

Not yet confirmed: ENCRYPTION_KEY from a chart before 0.51.0

Charts before 0.51.0 generated ENCRYPTION_KEY as 32 alphanumeric characters, which decode to 24 bytes. API images from chart 0.49.0 onward refuse any key that is not 32 bytes, and an upgrade reuses the key already in the cluster, so the API will not start. Check yours — 32 is fine, 24 is not:

kubectl -n edisyl get secret edisyl-encryption \
  -o jsonpath='{.data.ENCRYPTION_KEY}' | base64 -d | base64 -d 2>/dev/null | wc -c

If nothing has been encrypted under the key yet — no data-source credential has ever been saved — generate a new one with openssl rand -base64 32 and supply it as secrets.managed.edisyl-encryption.data.ENCRYPTION_KEY (or in your store). If the install holds live encrypted data, contact Edisyl before upgrading: replacing the key orphans every credential encrypted under the old one.

The LLM gateway cutover

On 0.36.0 the bundled gateway was off by default and inference went wherever edisyl-llm-gateway pointed. From 0.47.0 the bundled gateway is the only one and every inference call goes to it — or to another deployment’s gateway named in bifrost.external.url. There is no third option. Before upgrading:

1. Decide where the gateway runs. Bundled (the default) needs a database — bifrost.database.source: cnpg on the bundled cluster, or existing-secret naming a Secret with discrete connection keys; see Using your own infrastructure. A gateway you don’t run needs bifrost.enabled: false, bifrost.external.url and its credentials.
2. Configure its providers before traffic moves. A fresh gateway knows no provider, so every inference call fails until one is added. To avoid an outage, enable the gateway on your current chart first (bifrost.enabled: true and its database), add the providers on Edisyl’s provider-settings page and prove they answer, and only then upgrade. On charts before 0.46.0 enabling it moves no traffic.
3. Replace the gateway key. See the note below.

Confirm against your chart version: the gateway key

The LLM_GATEWAY_API_KEY Edisyl issued for its hosted gateway is not a virtual key on your bundled gateway, and inference fails with it. The bundled gateway seeds one inference-scoped virtual key, BIFROST_RUNTIME_KEY in edisyl-bifrost, which the chart generates once as sk-bf- followed by random characters and reads back on later upgrades; the applications infer with that key. Set LLM_GATEWAY_API_KEY to the same value — see Secrets and signing keys for both the k8s-native and the external-secrets form.

After the upgrade

Helm leaves a key it no longer renders in the live Secret, so the moved keys are still there after helm upgrade. The ConfigMap is read last and wins, so a stale key never overrides a value it carries — but it still applies where the ConfigMap leaves a blank optional key out. Remove the stale keys after helm upgrade has finished, with k8s-native Secrets (under external-secrets the operator rewrites each Secret from the store on its next refresh):

strip() {  # strip <secret> <key>...
  local s=$1; shift
  local jq_del; jq_del=$(printf 'del(.data["%s"]) | ' "$@"); jq_del=${jq_del% | }
  kubectl -n edisyl get secret "$s" -o json | jq "$jq_del" | kubectl -n edisyl replace -f -
}
strip edisyl-object-storage OBJECT_STORAGE_PROVIDER OBJECT_STORAGE_BUCKET \
  OBJECT_STORAGE_REGION OBJECT_STORAGE_PUBLIC_URL OBJECT_STORAGE_ENDPOINT \
  OBJECT_STORAGE_FORCE_PATH_STYLE
strip edisyl-redis REDIS_URL REDIS_HOST REDIS_PORT REDIS_USER REDIS_TLS
strip edisyl-github GITHUB_APP_ID
strip edisyl-llm-gateway LLM_GATEWAY_URL
strip edisyl-rudderstack RUDDERSTACK_DATA_PLANE_URL
strip edisyl-mono-api-key MONO_API_KEY      # once api and worker read MONO_API_KEYS
strip edisyl-database DATABASE_POOL_SIZE    # only with database.source: managed
strip edisyl-auth GOTRUE_EXTERNAL_GOOGLE_CLIENT_ID GOTRUE_EXTERNAL_GOOGLE_SECRET
kubectl -n edisyl rollout restart deploy/api deploy/worker deploy/edisyl

Removing a key that is not there is a no-op, so each line is safe on an install that never had it. The restart follows because envFrom snapshots the environment at container start.

Do not delete these Secrets to get the same effect. edisyl-database is mounted by the pre-upgrade migration hook under database.source: managed, and Helm cannot re-create it until that hook has finished, so deleting it deadlocks the next upgrade. Remove only the stale keys.

The upgrade also deletes the edisyl-supabase, edisyl-ses, edisyl-sentry and edisyl-realtime Secrets while old pods still reference them. An old pod that restarts before the rollout finishes cannot start until its replacement is up.

Rolling back

helm rollback edisyl <revision> -n edisyl

This reverts manifests. It does not revert migrations. If the release you are leaving applied a schema change, the older application code may not run against the migrated database. Treat rollback as a way to recover from a bad configuration, not from a bad migration — for that, restore your backup.

Rollback does not delete an existing database cluster that both revisions declare. It is helm uninstall that destroys it — see Installing.

Keeping generated secrets stable

If you let the chart generate secrets, they are generated once and reused by looking up the existing Secret in the cluster. A plain helm upgrade against a live cluster preserves them.

Rendering without cluster access does not. Under Argo CD, or anywhere the manifests are produced by helm template, the lookup returns empty and every generated value is regenerated — including ENCRYPTION_KEY, which orphans data encrypted under the previous one, and BIFROST_ENCRYPTION_KEY, which orphans every provider credential the gateway holds. Supply those values explicitly instead; see Configuration.