Secrets and signing keys

Everything the platform needs to authenticate — to your database, your object store, our registry, and to its own users — is supplied through Helm values. That does not mean it belongs in the same file as your hostname.

Three files, and only one of them is committed

Keep configuration, the credentials you supply, and the keys a script generates in separate files, and pass each one:

helm upgrade --install edisyl <chart> \
  -n edisyl \
  -f edisyl.values.yaml \
  -f secrets.local.yaml \
  -f auth-keys.local.yaml \
  --timeout 20m

File	Holds	In git
edisyl.values.yaml	config.domain, ingress hosts, which datastores you bundle, image overrides	Yes — this is the file you review and diff
secrets.local.yaml	Credentials you fill in by hand: DATABASE_URL, the LLM gateway key, object-storage keys, your Redis password	No. Add it to .gitignore before you write it
auth-keys.local.yaml	The auth signing material, written by the script below	No. Same

The last two are split for a mechanical reason: the minting script overwrites whatever file it writes to. Pointing it at the file holding your database URL would silently discard it. One file is hand-edited, the other is generated, so they cannot share a name.

The registry password is in none of them — it is a 12-hour ECR token, passed on the install command instead. See Registry access.

Later -f wins. Helm merges values files left to right, so the credentials file overrides the configuration file where they name the same key. That is what lets the committed file carry an empty placeholder for a value the gitignored one fills in.

If you would rather not hold credentials in a file at all, the chart reads them from a secret manager instead — see secrets.provider: external-secrets.

Mint the auth signing material

The chart deliberately generates none of this. Helm cannot produce an EC keypair, and minting a JWT needs HMAC-SHA256 — neither is something a template can do.

You need four things: an ES256 keypair for user tokens, an HS256 secret behind the anon key and the admin token, those two JWTs themselves, and an RSA key the auth service validates at config load for SAML.

The admin token, AUTH_ADMIN_TOKEN, must carry role: mono_api. Since chart 0.55.0 that is the only role the auth service admits to its admin API, and a token carrying any other role gets 401. Both scripts below mint it that way; to check or replace a token you already have, see Replacing the admin token.

These keys are yours alone. We do not generate, hold or have any way to recover signing material for a self-hosted deployment — everything below runs on your own machine, and the output never leaves it.

Run this once

Re-minting rotates the keys and strands every identity already written against them. Generate these before your first install and keep the output.

Two ways to produce them. Both write the same file; pick whichever you would rather audit.

With openssl alone

Every step is a single recognisable command, so you can run them one at a time and look at what each produces. Needs openssl, jq and uuidgen.

mint-auth-keys.sh

# Mints the auth service's signing material into auth-keys.local.yaml.
# Run once: re-minting rotates the keys and strands every identity already
# written against them. Needs openssl, jq and uuidgen.
set -eu
umask 077
b64u() { base64 | tr '+/' '-_' | tr -d '=\n'; }

# The HS256 secret behind the anon key and the admin token (>= 32 chars). The
# admin token carries role mono_api, the only role the auth service admits to
# its admin API, and is mounted on the api and the IdP registration Job alone.
SEC=$(openssl rand -hex 24)

# The ES256 keypair for user tokens.
openssl ecparam -name prime256v1 -genkey -noout -out ec.pem

# Its JWK coordinates, taken straight out of the DER encoding. Binary stays in
# files and pipes, never in a shell variable: a variable cannot hold a NUL byte,
# and one dropped byte yields a public key that does not match the private one —
# tokens then sign but never verify.
openssl ec -in ec.pem -pubout -outform DER 2>/dev/null | tail -c 65 > point.bin
D=$(openssl ec -in ec.pem -outform DER 2>/dev/null | dd bs=1 skip=7 count=32 2>/dev/null | b64u)
X=$(dd if=point.bin bs=1 skip=1  count=32 2>/dev/null | b64u)
Y=$(dd if=point.bin bs=1 skip=33 count=32 2>/dev/null | b64u)
KID=$(uuidgen | tr 'A-Z' 'a-z')

mkjwt() {
  h=$(printf '{"alg":"HS256","typ":"JWT"}' | b64u)
  p=$(printf '%s' "$1" | b64u)
  s=$(printf '%s' "$h.$p" | openssl dgst -sha256 -hmac "$SEC" -binary | b64u)
  printf '%s.%s.%s' "$h" "$p" "$s"
}
NOW=$(date +%s); EXP=$((NOW + 315360000))    # ten years
ANON=$(mkjwt "{\"role\":\"anon\",\"iss\":\"edisyl-auth\",\"iat\":$NOW,\"exp\":$EXP}")
ADMIN=$(mkjwt "{\"role\":\"mono_api\",\"iss\":\"edisyl-auth\",\"iat\":$NOW,\"exp\":$EXP}")

# The key set: the EC key, plus a VERIFY-ONLY HS256 entry. Both are required.
# Without the EC key the service issues HS256 tokens with no issuer claim and
# every authenticated request 401s. But once a key set exists the service
# rejects every algorithm outside it, which would break the HS256 admin token
# against its own admin API — hence the second entry, with key_ops keeping the
# EC key the sole signer so user tokens stay ES256.
JWT_KEYS=$(jq -cn --arg x "$X" --arg y "$Y" --arg d "$D" --arg kid "$KID" \
  --arg k "$(printf '%s' "$SEC" | b64u)" \
  '[{kty:"EC",crv:"P-256",x:$x,y:$y,d:$d,kid:$kid,alg:"ES256",use:"sig",key_ops:["sign","verify"]},
    {kty:"oct",k:$k,alg:"HS256",use:"sig",kid:"legacy-hs256",key_ops:["verify"]}]')

# The RSA key the auth service validates at config load for SAML.
SAML=$(openssl genpkey -algorithm rsa -pkeyopt rsa_keygen_bits:2048 -outform DER 2>/dev/null \
       | base64 | tr -d '\n')

# Single-quote the key set: it contains [] and {}, which bare YAML would read
# as flow collections.
cat > auth-keys.local.yaml <<YAML
# Real signing material — never commit. Minted $(date -u +%Y-%m-%dT%H:%M:%SZ).
secrets:
  managed:
    edisyl-auth:
      data:
        GOTRUE_JWT_SECRET: $SEC
        GOTRUE_JWT_KEYS: '$JWT_KEYS'
        GOTRUE_SAML_PRIVATE_KEY: $SAML
    edisyl-auth-client:
      data:
        AUTH_ANON_KEY: $ANON
    edisyl-auth-admin:
      data:
        AUTH_ADMIN_TOKEN: $ADMIN
YAML
rm -f ec.pem point.bin

# Check it: x, y and d must each decode to exactly 32 bytes. A short one is the
# NUL-byte truncation above, and it is invisible by eye.
python3 - auth-keys.local.yaml <<'CHECK'
import base64, json, sys, yaml
ks = json.loads(yaml.safe_load(open(sys.argv[1]))
                ["secrets"]["managed"]["edisyl-auth"]["data"]["GOTRUE_JWT_KEYS"])
ec = ks[0]
for f in ("x", "y", "d"):
    n = len(base64.urlsafe_b64decode(ec[f] + "=" * (-len(ec[f]) % 4)))
    print(f"  {f}: {n} bytes", "OK" if n == 32 else "WRONG — regenerate")
print("  entries:", [k.get("kid") for k in ks])
CHECK
echo "wrote auth-keys.local.yaml"

Run it with bash mint-auth-keys.sh. It ends by checking its own output — each of x, y and d must decode to exactly 32 bytes, which is what catches the NUL-byte truncation the comments warn about, and is invisible by eye:

  x: 32 bytes OK
  y: 32 bytes OK
  d: 32 bytes OK
  entries: ['74c9d29b-…', 'legacy-hs256']

Two entries, and 32 bytes three times. Anything else means regenerate.

With python

Shorter, and it does the same work in one pass.

umask 077
python3 - auth-keys.local.yaml <<'PY'
import base64, hashlib, hmac, json, secrets, subprocess, sys, time, uuid

def b64u(b): return base64.urlsafe_b64encode(b).rstrip(b'=').decode()

def jwt(payload, key):
    h = b64u(json.dumps({"alg": "HS256", "typ": "JWT"}, separators=(',', ':')).encode())
    p = b64u(json.dumps(payload, separators=(',', ':')).encode())
    s = b64u(hmac.new(key.encode(), f"{h}.{p}".encode(), hashlib.sha256).digest())
    return f"{h}.{p}.{s}"

sec = secrets.token_urlsafe(48)[:48]              # the service requires >= 32 chars
now = int(time.time()); exp = now + 10 * 365 * 24 * 3600
anon = jwt({"role": "anon", "iss": "edisyl-auth", "iat": now, "exp": exp}, sec)
admin = jwt({"role": "mono_api", "iss": "edisyl-auth", "iat": now, "exp": exp}, sec)

# ES256 keypair for USER tokens. The API verifies user JWTs as ES256 against
# the auth service's published JWKS and refuses every other algorithm, so the
# HMAC secret above covers only the anon key and the admin token.
pem = subprocess.run(["openssl", "ecparam", "-name", "prime256v1", "-genkey", "-noout"],
                     check=True, capture_output=True).stdout
text = subprocess.run(["openssl", "ec", "-text", "-noout"],
                      input=pem, check=True, capture_output=True).stdout.decode()

def hexblock(label):
    out, grabbing = [], False
    for line in text.splitlines():
        if line.strip().startswith(label):
            grabbing = True
            continue
        if grabbing:
            # A hex row. The last row can be one byte with no colon (OpenSSL 3
            # prints 16 bytes a row, so a 65-byte point ends on a lone byte).
            if line.startswith((" ", "\t")) and all(c in "0123456789abcdef:" for c in line.strip().lower()):
                out.append(line.strip())
            else:
                break
    return bytes.fromhex("".join(out).replace(":", ""))

d, point = hexblock("priv:"), hexblock("pub:")
if len(point) != 65 or point[0] != 0x04:
    sys.exit("unexpected EC public point from openssl (want 65-byte uncompressed)")
x, y = point[1:33], point[33:65]
d = d[-32:].rjust(32, b"\x00")                    # openssl may emit a leading pad byte

kid = str(uuid.uuid4())
private_jwk = {"kty": "EC", "crv": "P-256", "x": b64u(x), "y": b64u(y), "d": b64u(d),
               "kid": kid, "alg": "ES256", "use": "sig", "key_ops": ["sign", "verify"]}
# Verify-only, and not optional: setting the key set makes the service reject
# every algorithm outside it, so without this entry the HS256 admin token
# stops working against its own admin API. key_ops keeps the EC key the sole
# signer, so user tokens stay ES256.
legacy_verify = {"kty": "oct", "k": b64u(sec.encode()), "alg": "HS256", "use": "sig",
                 "kid": "legacy-hs256", "key_ops": ["verify"]}
jwt_keys = json.dumps([private_jwk, legacy_verify], separators=(',', ':'))

saml = subprocess.run(["openssl", "genpkey", "-algorithm", "rsa",
                       "-pkeyopt", "rsa_keygen_bits:2048", "-outform", "DER"],
                      check=True, capture_output=True).stdout

with open(sys.argv[1], "w") as f:
    f.write("# Minted by hand. Real signing material — never commit.\n")
    f.write("secrets:\n  managed:\n")
    f.write("    edisyl-auth:\n      data:\n")
    f.write(f"        GOTRUE_JWT_SECRET: {sec}\n")
    # Single-quoted: the JSON contains [] and {}, which bare YAML would read as
    # flow collections. base64url never contains a single quote.
    f.write(f"        GOTRUE_JWT_KEYS: '{jwt_keys}'\n")
    f.write(f"        GOTRUE_SAML_PRIVATE_KEY: {base64.b64encode(saml).decode()}\n")
    f.write("    edisyl-auth-client:\n      data:\n")
    f.write(f"        AUTH_ANON_KEY: {anon}\n")
    # mono_api is the only role the auth service admits to its admin API,
    # which registers your identity provider.
    f.write("    edisyl-auth-admin:\n      data:\n")
    f.write(f"        AUTH_ADMIN_TOKEN: {admin}\n")
print(f"minted ES256 keypair (kid {kid}), HS256 secret, anon and mono_api JWTs, SAML SP key")
PY

Needs python3 and openssl. It writes auth-keys.local.yaml in the shape the chart expects; umask 077 keeps the file readable only by you.

Why the key set has two entries

The trap this avoids is worth stating plainly, because the symptom points nowhere near the cause.

A service configured with only a symmetric secret issues HS256 tokens with no issuer claim, and every authenticated request then fails with 401 Invalid or expired token, sign-in included. It reads as a credential problem; it is an algorithm one. So the EC key has to be there.

But once a key set exists, the service rejects every algorithm outside it — which would break the HS256 admin token against its own admin API. Hence the second, verify-only HS256 entry, with key_ops keeping the EC key the sole signer so user tokens stay ES256.

The rest of the credentials

The same file carries everything else you would not commit:

secrets:
  managed:
    edisyl-database:
      data:
        DATABASE_URL: "postgresql://user:pass@host:5432/edisyl"
    edisyl-object-storage:
      data:
        OBJECT_STORAGE_ACCESS_KEY_ID: "..."
        OBJECT_STORAGE_SECRET_ACCESS_KEY: "..."

Credentials only. Since chart 0.46.0 the object store’s provider, bucket and URL, and your Redis host, port and user, are values under config.* in the committed file; a configuration key left in a managed Secret fails the render. See Configuration.

The registry password is not here: it is an ECR authorization token that expires in 12 hours, so it is a seed passed on the install command rather than a steady state. See Registry access for how the chart keeps it alive afterwards, and why seeding one without the refresher is refused.

What the chart generates

Under the default k8s-native provider the chart fills in the keys it owns when you leave them blank, generating each once and reading it back on upgrade so it cannot rotate: ENCRYPTION_KEY, MONO_API_KEYS, TOKEN_HMAC_KEY, INNGEST_EVENT_KEY, INNGEST_SIGNING_KEY, and the LLM gateway’s BIFROST_ADMIN_USERNAME, BIFROST_ADMIN_PASSWORD, BIFROST_RUNTIME_KEY and BIFROST_ENCRYPTION_KEY. Bundled Redis gets a generated password too, shared by REDIS_PASSWORD and the INNGEST_REDIS_URI the chart composes from it, and bundled MinIO its root credentials.

The configuration reference lists every credential the chart reads, whether it generates it, and how to produce the ones it does not. The auth, Inngest and Bifrost/LLM database Secrets are not in this file at all: with your own Postgres you create them with kubectl — see Using your own infrastructure.

To supply ENCRYPTION_KEY yourself, it must be 32 random bytes, base64-encoded — openssl rand -base64 32. The render refuses any other length, and the API refuses one at boot. It encrypts every organization’s data-source credentials, so it can never be rotated without re-encrypting them.

MONO_API_KEYS was MONO_API_KEY before chart 0.49.0. It is a comma-separated list: the API accepts any entry, and every caller sends the first.

The LLM gateway

The gateway runs in every install, and two of its Secrets are generated for you on a live install:

Secret	Keys	What it is
edisyl-bifrost	BIFROST_ADMIN_USERNAME, BIFROST_ADMIN_PASSWORD, BIFROST_RUNTIME_KEY	The console’s admin credential, and the inference-scoped virtual key the applications infer with — the settings page presents the same key to prove a provider answers. A runtime key you supply must begin sk-bf-.
edisyl-bifrost-encryption	BIFROST_ENCRYPTION_KEY	What every stored provider credential is encrypted under. Cannot be rotated — back it up.

The key the applications send with every inference call, LLM_GATEWAY_API_KEY in edisyl-llm-gateway, is not generated.

Confirm against your chart version

LLM_GATEWAY_API_KEY is the gateway’s runtime key: set it to the same value as BIFROST_RUNTIME_KEY. The chart generates the runtime key as sk-bf- followed by random characters, so the simplest way to keep the two equal is to mint one key and supply it to both. The configurator writes REPLACE_WITH_GATEWAY_RUNTIME_KEY in both places in secrets.local.yaml, and its install steps replace it with one locally minted key. Safe to re-run: once the placeholder is gone there is nothing to replace.

KEY="sk-bf-$(openssl rand -hex 24)"
sed -i.bak "s/REPLACE_WITH_GATEWAY_RUNTIME_KEY/$KEY/g" secrets.local.yaml && rm secrets.local.yaml.bak
unset KEY

secrets:
  managed:
    edisyl-bifrost:
      data:
        BIFROST_RUNTIME_KEY: "sk-bf-..."
    edisyl-llm-gateway:
      data:
        LLM_GATEWAY_API_KEY: "sk-bf-..."    # the same value

Under external-secrets, map the one store entry to both:

secrets:
  managed:
    edisyl-llm-gateway:
      keys: []
      remoteKeys:
        LLM_GATEWAY_API_KEY: BIFROST_RUNTIME_KEY

Under Argo CD or external-secrets the chart generates nothing, so all four edisyl-bifrost* keys have to come from you or your store. See Known limitations.

Upgrading from a chart before 0.36.0

BIFROST_RUNTIME_KEY was BIFROST_PROBE_KEY until chart 0.36.0. Rename it in your values or your store before pinning 0.36.0 or later: under external-secrets every listed key is required, so a store that only carries BIFROST_PROBE_KEY fails the edisyl-bifrost ExternalSecret and it never syncs.

If you use a gateway you don’t run

Pointing bifrost.external.url at another deployment’s gateway — see Configuration — mounts edisyl-bifrost on the API without running a gateway here. The chart generates nothing for it: the admin credential and the runtime key are that gateway’s own, so generated values could never open it. All three keys have to come from you, and with chart-managed Secrets the render refuses them blank. LLM_GATEWAY_API_KEY is one of that gateway’s virtual keys too. edisyl-bifrost-encryption is not needed; it stays with whoever runs the gateway.

Or keep them in a secret manager instead

For anything beyond a trial, don’t hold credentials in a file at all. Set secrets.provider: external-secrets and the chart renders one ExternalSecret per managed secret instead of a populated Secret, reading each key from your store at runtime.

secrets:
  provider: external-secrets
  externalSecrets:
    # Name a store you already have.
    storeName: my-cluster-store
    storeKind: ClusterSecretStore

This needs the External Secrets Operator installed in the cluster. The chart renders the ExternalSecret resources and nothing more — the store itself is yours, so any backend the operator supports works: AWS Secrets Manager, Doppler, HashiCorp Vault, Google Secret Manager, Azure Key Vault.

The chart can also render a Doppler ClusterSecretStore for you rather than you creating one:

secrets:
  externalSecrets:
    createStore:
      project: your-doppler-project

storeName then defaults to <namespace>-doppler, and storeKind must stay ClusterSecretStore — the chart refuses the mismatch rather than rendering a store nothing reads.

A value changed in the store reaches its Secret within secrets.externalSecrets.refreshInterval, 15 minutes by default since chart 0.48.0 — each key is its own request to the store, so a shorter interval multiplies the load on it. After rotating a compromised credential, force the sync rather than wait, then restart the pods that mount it:

kubectl -n edisyl annotate externalsecret <name> force-sync="$(date +%s)" --overwrite

Two things that catch people

The chart generates nothing in this mode. Everything listed under What the chart generates has to be in your store before you install — the chart writes nothing to it. That includes bundled Redis: the store’s REDIS_PASSWORD is the password it enforces, and INNGEST_REDIS_URI is redis://:<that password>@edisyl-redis.<namespace>.svc:6379. Put INNGEST_SIGNING_KEY there as even-length lowercase hex, ENCRYPTION_KEY as 32 random bytes base64-encoded (openssl rand -base64 32), and BIFROST_RUNTIME_KEY with its sk-bf- prefix, which nothing can check on this path. Miss them and Inngest, encryption and inference break with no warning at install time. The configuration reference has the full list of store keys; the configurator lists the ones your answers need.

Every key a managed secret lists is a required remote reference, and one missing entry fails the whole ExternalSecret — taking every other key in that family down with it, not just the one you forgot. An integration you do not use is left out by emptying its list: a group with no keys, remoteKeys or templateData renders no ExternalSecret, and the apps mount it as optional.

secrets:
  managed:
    # The reranker. Optional: only a rerank call fails without it.
    edisyl-voyage:
      keys: []
    # TypeSafe rewording, mounted on the API since chart 0.50.0. Without this,
    # the store must carry TYPESAFE_API_KEY and TYPESAFE_BASE_URL.
    edisyl-typesafe:
      keys: []

You still have to mint the signing keys

A secret manager is where the auth signing material lives, not something that produces it. Generate it exactly as above, then load it into your store rather than writing auth-keys.local.yaml. Every key keeps its name in the store: GOTRUE_JWT_KEYS, GOTRUE_JWT_SECRET, GOTRUE_SAML_PRIVATE_KEY, AUTH_ANON_KEY and AUTH_ADMIN_TOKEN.

Upgrading to chart 0.57.0 under external-secrets

Before 0.57.0 the store held the last two as AUTH_SELFHOSTED_ANON_KEY and AUTH_SELFHOSTED_ADMIN_TOKEN. Add AUTH_ANON_KEY and AUTH_ADMIN_TOKEN to your store with the same values before upgrading: a missing one fails its ExternalSecret, and the API and web app cannot start without it. Keep the old names until no environment can roll back below 0.57.0. A values file that still maps either key to its AUTH_SELFHOSTED_ name through remoteKeys fails the render; remove that entry. So does a keys override on edisyl-auth-client or edisyl-auth-admin that leaves its key out (keys: [] was the old default) unless a remoteKeys entry under another store name supplies it. Under k8s-native nothing changes.

Replacing the admin token

Since chart 0.55.0 the auth service admits only role: mono_api to its admin API, and values cannot widen that. A token carrying any other role — such as the service_role token earlier versions of the scripts above minted — gets 401: the API’s identity-provider routes fail, and so does the edisyl-idp-reconcile Job, which fails an upgrade run with --wait-for-jobs. Check the role your token carries:

kubectl -n edisyl get secret edisyl-auth-admin -o jsonpath='{.data.AUTH_ADMIN_TOKEN}' \
  | base64 -d | python3 -c '
import base64, json, sys
p = sys.stdin.read().split(".")[1]
print(json.loads(base64.urlsafe_b64decode(p + "=" * (-len(p) % 4)))["role"])'

To replace it, mint a mono_api token signed with the HS256 secret the auth service already verifies. No other signing material changes, so nobody is signed out. Both scripts above put that secret in the key set as an oct entry; list the key types yours carries:

kubectl -n edisyl get secret edisyl-auth -o jsonpath='{.data.GOTRUE_JWT_KEYS}' \
  | base64 -d | python3 -c '
import json, sys
print([k["kty"] for k in json.load(sys.stdin)])'

If oct is not listed, an HS256 token gets 401 too: sign the token ES256 with the EC private key in that set instead. Otherwise:

kubectl -n edisyl get secret edisyl-auth -o jsonpath='{.data.GOTRUE_JWT_SECRET}' \
  | base64 -d | python3 -c '
import base64, hashlib, hmac, json, sys, time
sec = sys.stdin.read().encode()
b64u = lambda b: base64.urlsafe_b64encode(b).rstrip(b"=").decode()
enc = lambda o: b64u(json.dumps(o, separators=(",", ":")).encode())
now = int(time.time())
h = enc({"alg": "HS256", "typ": "JWT"})
p = enc({"role": "mono_api", "iss": "edisyl-auth", "iat": now, "exp": now + 10*365*24*3600})
print(f"{h}.{p}." + b64u(hmac.new(sec, f"{h}.{p}".encode(), hashlib.sha256).digest()))'

Every chart that admits mono_api accepts the new token, 0.52.0 included, so put it in place while still on your current chart, before moving to 0.55.0 or later:

1. Get the new token into the live edisyl-auth-admin Secret.
   • With k8s-native, put it in secrets.managed.edisyl-auth-admin.data.AUTH_ADMIN_TOKEN in auth-keys.local.yaml, then run the upgrade command with --version set to the chart you are already on. Editing the file alone changes nothing in the cluster.
   • With external-secrets, write it to your store under AUTH_ADMIN_TOKEN, and on a chart before 0.57.0 under AUTH_SELFHOSTED_ADMIN_TOKEN too, which is the name those charts read. Then force the ExternalSecret to sync, kubectl -n edisyl annotate externalsecret edisyl-auth-admin force-sync="$(date +%s)" --overwrite, and wait until the Secret carries the new token — the role check above reads it.
2. Restart the API. Pods read Secrets at start, and nothing restarts the API when one changes: kubectl -n edisyl rollout restart deploy/api.
3. Upgrade to the new chart. The reconcile Job runs again with the new token.