Secrets and signing keys

Everything the platform needs to authenticate — to your database, your object store, our registry, and to its own users — is supplied through Helm values. That does not mean it belongs in the same file as your hostname.

Three files, and only one of them is committed

Keep configuration, the credentials you supply, and the keys a script generates in separate files, and pass each one:

helm upgrade --install edisyl <chart> \
  -n edisyl \
  -f edisyl.values.yaml \
  -f secrets.local.yaml \
  -f auth-keys.local.yaml \
  --timeout 20m

File	Holds	In git
edisyl.values.yaml	config.domain, ingress hosts, which datastores you bundle, image overrides	Yes — this is the file you review and diff
secrets.local.yaml	Credentials you fill in by hand: DATABASE_URL, the LLM gateway key, object-storage keys, your Redis password	No. Add it to .gitignore before you write it
auth-keys.local.yaml	The auth signing material, written by the script below	No. Same

The last two are split for a mechanical reason: the minting script overwrites whatever file it writes to. Pointing it at the file holding your database URL would silently discard it. One file is hand-edited, the other is generated, so they cannot share a name.

The registry password is in none of them — it is a 12-hour ECR token, passed on the install command instead. See Registry access.

Later -f wins. Helm merges values files left to right, so the credentials file overrides the configuration file where they name the same key. That is what lets the committed file carry an empty placeholder for a value the gitignored one fills in.

If you would rather not hold credentials in a file at all, the chart reads them from a secret manager instead — see secrets.provider: external-secrets.

Mint the auth signing material

The chart deliberately generates none of this. Helm cannot produce an EC keypair, and minting a JWT needs HMAC-SHA256 — neither is something a template can do.

You need four things: an ES256 keypair for user tokens, an HS256 secret behind the anon and service_role keys, those two JWTs themselves, and an RSA key the auth service validates at config load for SAML.

These keys are yours alone. We do not generate, hold or have any way to recover signing material for a self-hosted deployment — everything below runs on your own machine, and the output never leaves it.

Run this once

Re-minting rotates the keys and strands every identity already written against them. Generate these before your first install and keep the output.

Two ways to produce them. Both write the same file; pick whichever you would rather audit.

With openssl alone

Every step is a single recognisable command, so you can run them one at a time and look at what each produces. Needs openssl, jq and uuidgen.

mint-auth-keys.sh

# Mints the auth service's signing material into auth-keys.local.yaml.
# Run once: re-minting rotates the keys and strands every identity already
# written against them. Needs openssl, jq and uuidgen.
set -eu
umask 077
b64u() { base64 | tr '+/' '-_' | tr -d '=\n'; }

# The HS256 secret behind the anon and service_role keys (>= 32 chars). The
# service_role key is the admin token, mounted on the api and the IdP
# registration Job alone.
SEC=$(openssl rand -hex 24)

# The ES256 keypair for user tokens.
openssl ecparam -name prime256v1 -genkey -noout -out ec.pem

# Its JWK coordinates, taken straight out of the DER encoding. Binary stays in
# files and pipes, never in a shell variable: a variable cannot hold a NUL byte,
# and one dropped byte yields a public key that does not match the private one —
# tokens then sign but never verify.
openssl ec -in ec.pem -pubout -outform DER 2>/dev/null | tail -c 65 > point.bin
D=$(openssl ec -in ec.pem -outform DER 2>/dev/null | dd bs=1 skip=7 count=32 2>/dev/null | b64u)
X=$(dd if=point.bin bs=1 skip=1  count=32 2>/dev/null | b64u)
Y=$(dd if=point.bin bs=1 skip=33 count=32 2>/dev/null | b64u)
KID=$(uuidgen | tr 'A-Z' 'a-z')

mkjwt() {
  h=$(printf '{"alg":"HS256","typ":"JWT"}' | b64u)
  p=$(printf '%s' "$1" | b64u)
  s=$(printf '%s' "$h.$p" | openssl dgst -sha256 -hmac "$SEC" -binary | b64u)
  printf '%s.%s.%s' "$h" "$p" "$s"
}
NOW=$(date +%s); EXP=$((NOW + 315360000))    # ten years
ANON=$(mkjwt "{\"role\":\"anon\",\"iss\":\"supabase\",\"iat\":$NOW,\"exp\":$EXP}")
SVC=$(mkjwt "{\"role\":\"service_role\",\"iss\":\"supabase\",\"iat\":$NOW,\"exp\":$EXP}")

# The key set: the EC key, plus a VERIFY-ONLY HS256 entry. Both are required.
# Without the EC key the service issues HS256 tokens with no issuer claim and
# every authenticated request 401s. But once a key set exists the service
# rejects every algorithm outside it, which would break the service_role token
# against its own admin API — hence the second entry, with key_ops keeping the
# EC key the sole signer so user tokens stay ES256.
JWT_KEYS=$(jq -cn --arg x "$X" --arg y "$Y" --arg d "$D" --arg kid "$KID" \
  --arg k "$(printf '%s' "$SEC" | b64u)" \
  '[{kty:"EC",crv:"P-256",x:$x,y:$y,d:$d,kid:$kid,alg:"ES256",use:"sig",key_ops:["sign","verify"]},
    {kty:"oct",k:$k,alg:"HS256",use:"sig",kid:"legacy-hs256",key_ops:["verify"]}]')

# The RSA key the auth service validates at config load for SAML.
SAML=$(openssl genpkey -algorithm rsa -pkeyopt rsa_keygen_bits:2048 -outform DER 2>/dev/null \
       | base64 | tr -d '\n')

# Single-quote the key set: it contains [] and {}, which bare YAML would read
# as flow collections.
cat > auth-keys.local.yaml <<YAML
# Real signing material — never commit. Minted $(date -u +%Y-%m-%dT%H:%M:%SZ).
secrets:
  managed:
    edisyl-auth:
      data:
        GOTRUE_JWT_SECRET: $SEC
        GOTRUE_JWT_KEYS: '$JWT_KEYS'
        GOTRUE_SAML_PRIVATE_KEY: $SAML
    edisyl-auth-client:
      data:
        AUTH_ANON_KEY: $ANON
    edisyl-auth-admin:
      data:
        AUTH_ADMIN_TOKEN: $SVC
YAML
rm -f ec.pem point.bin

# Check it: x, y and d must each decode to exactly 32 bytes. A short one is the
# NUL-byte truncation above, and it is invisible by eye.
python3 - auth-keys.local.yaml <<'CHECK'
import base64, json, sys, yaml
ks = json.loads(yaml.safe_load(open(sys.argv[1]))
                ["secrets"]["managed"]["edisyl-auth"]["data"]["GOTRUE_JWT_KEYS"])
ec = ks[0]
for f in ("x", "y", "d"):
    n = len(base64.urlsafe_b64decode(ec[f] + "=" * (-len(ec[f]) % 4)))
    print(f"  {f}: {n} bytes", "OK" if n == 32 else "WRONG — regenerate")
print("  entries:", [k.get("kid") for k in ks])
CHECK
echo "wrote auth-keys.local.yaml"

Run it with bash mint-auth-keys.sh. It ends by checking its own output — each of x, y and d must decode to exactly 32 bytes, which is what catches the NUL-byte truncation the comments warn about, and is invisible by eye:

  x: 32 bytes OK
  y: 32 bytes OK
  d: 32 bytes OK
  entries: ['74c9d29b-…', 'legacy-hs256']

Two entries, and 32 bytes three times. Anything else means regenerate.

With python

Shorter, and it does the same work in one pass.

umask 077
python3 - auth-keys.local.yaml <<'PY'
import base64, hashlib, hmac, json, secrets, subprocess, sys, time, uuid

def b64u(b): return base64.urlsafe_b64encode(b).rstrip(b'=').decode()

def jwt(payload, key):
    h = b64u(json.dumps({"alg": "HS256", "typ": "JWT"}, separators=(',', ':')).encode())
    p = b64u(json.dumps(payload, separators=(',', ':')).encode())
    s = b64u(hmac.new(key.encode(), f"{h}.{p}".encode(), hashlib.sha256).digest())
    return f"{h}.{p}.{s}"

sec = secrets.token_urlsafe(48)[:48]              # the service requires >= 32 chars
now = int(time.time()); exp = now + 10 * 365 * 24 * 3600
anon = jwt({"role": "anon", "iss": "supabase", "iat": now, "exp": exp}, sec)
service = jwt({"role": "service_role", "iss": "supabase", "iat": now, "exp": exp}, sec)

# ES256 keypair for USER tokens. The API verifies user JWTs as ES256 against
# the auth service's published JWKS and refuses every other algorithm, so the
# HMAC secret above covers only the anon/service_role API keys.
pem = subprocess.run(["openssl", "ecparam", "-name", "prime256v1", "-genkey", "-noout"],
                     check=True, capture_output=True).stdout
text = subprocess.run(["openssl", "ec", "-text", "-noout"],
                      input=pem, check=True, capture_output=True).stdout.decode()

def hexblock(label):
    out, grabbing = [], False
    for line in text.splitlines():
        if line.strip().startswith(label):
            grabbing = True
            continue
        if grabbing:
            # A hex row. The last row can be one byte with no colon (OpenSSL 3
            # prints 16 bytes a row, so a 65-byte point ends on a lone byte).
            if line.startswith((" ", "\t")) and all(c in "0123456789abcdef:" for c in line.strip().lower()):
                out.append(line.strip())
            else:
                break
    return bytes.fromhex("".join(out).replace(":", ""))

d, point = hexblock("priv:"), hexblock("pub:")
if len(point) != 65 or point[0] != 0x04:
    sys.exit("unexpected EC public point from openssl (want 65-byte uncompressed)")
x, y = point[1:33], point[33:65]
d = d[-32:].rjust(32, b"\x00")                    # openssl may emit a leading pad byte

kid = str(uuid.uuid4())
private_jwk = {"kty": "EC", "crv": "P-256", "x": b64u(x), "y": b64u(y), "d": b64u(d),
               "kid": kid, "alg": "ES256", "use": "sig", "key_ops": ["sign", "verify"]}
# Verify-only, and not optional: setting the key set makes the service reject
# every algorithm outside it, so without this entry the HS256 service_role key
# stops working against its own admin API. key_ops keeps the EC key the sole
# signer, so user tokens stay ES256.
legacy_verify = {"kty": "oct", "k": b64u(sec.encode()), "alg": "HS256", "use": "sig",
                 "kid": "legacy-hs256", "key_ops": ["verify"]}
jwt_keys = json.dumps([private_jwk, legacy_verify], separators=(',', ':'))

saml = subprocess.run(["openssl", "genpkey", "-algorithm", "rsa",
                       "-pkeyopt", "rsa_keygen_bits:2048", "-outform", "DER"],
                      check=True, capture_output=True).stdout

with open(sys.argv[1], "w") as f:
    f.write("# Minted by hand. Real signing material — never commit.\n")
    f.write("secrets:\n  managed:\n")
    f.write("    edisyl-auth:\n      data:\n")
    f.write(f"        GOTRUE_JWT_SECRET: {sec}\n")
    # Single-quoted: the JSON contains [] and {}, which bare YAML would read as
    # flow collections. base64url never contains a single quote.
    f.write(f"        GOTRUE_JWT_KEYS: '{jwt_keys}'\n")
    f.write(f"        GOTRUE_SAML_PRIVATE_KEY: {base64.b64encode(saml).decode()}\n")
    f.write("    edisyl-auth-client:\n      data:\n")
    f.write(f"        AUTH_ANON_KEY: {anon}\n")
    # service_role is an admin role, so this token reaches the admin surface
    # used to register an identity provider.
    f.write("    edisyl-auth-admin:\n      data:\n")
    f.write(f"        AUTH_ADMIN_TOKEN: {service}\n")
print(f"minted ES256 keypair (kid {kid}), HS256 secret, anon/service_role JWTs, SAML SP key")
PY

Needs python3 and openssl. It writes auth-keys.local.yaml in the shape the chart expects; umask 077 keeps the file readable only by you.

Why the key set has two entries

The trap this avoids is worth stating plainly, because the symptom points nowhere near the cause.

A service configured with only a symmetric secret issues HS256 tokens with no issuer claim, and every authenticated request then fails with 401 Invalid or expired token, sign-in included. It reads as a credential problem; it is an algorithm one. So the EC key has to be there.

But once a key set exists, the service rejects every algorithm outside it — which would break the service_role token against its own admin API. Hence the second, verify-only HS256 entry, with key_ops keeping the EC key the sole signer so user tokens stay ES256.

The rest of the credentials

The same file carries everything else you would not commit:

secrets:
  managed:
    edisyl-database:
      data:
        DATABASE_URL: "postgresql://user:pass@host:5432/edisyl"
    edisyl-object-storage:
      data:
        OBJECT_STORAGE_ACCESS_KEY_ID: "..."
        OBJECT_STORAGE_SECRET_ACCESS_KEY: "..."

Credentials only. Since chart 0.46.0 the object store’s provider, bucket and URL, and your Redis host, port and user, are values under config.* in the committed file; a configuration key left in a managed Secret fails the render. See Configuration.

The registry password is not here: it is an ECR authorization token that expires in 12 hours, so it is a seed passed on the install command rather than a steady state. See Registry access for how the chart keeps it alive afterwards, and why seeding one without the refresher is refused.

What the chart generates

Under the default k8s-native provider the chart fills in the keys it owns when you leave them blank, generating each once and reading it back on upgrade so it cannot rotate: ENCRYPTION_KEY, MONO_API_KEYS, TOKEN_HMAC_KEY, LATTICE_API_KEY, INNGEST_EVENT_KEY, INNGEST_SIGNING_KEY, and the LLM gateway’s BIFROST_ADMIN_USERNAME, BIFROST_ADMIN_PASSWORD, BIFROST_RUNTIME_KEY and BIFROST_ENCRYPTION_KEY.

To supply ENCRYPTION_KEY yourself, it must be 32 random bytes, base64-encoded — openssl rand -base64 32. The render refuses any other length, and the API refuses one at boot. It encrypts every organization’s data-source credentials, so it can never be rotated without re-encrypting them.

MONO_API_KEYS was MONO_API_KEY before chart 0.49.0. It is a comma-separated list: the API accepts any entry, and every caller sends the first.

The LLM gateway

The gateway runs in every install, and two of its Secrets are generated for you on a live install:

Secret	Keys	What it is
edisyl-bifrost	BIFROST_ADMIN_USERNAME, BIFROST_ADMIN_PASSWORD, BIFROST_RUNTIME_KEY	The console’s admin credential, and the inference-scoped virtual key the applications infer with — the settings page presents the same key to prove a provider answers. A runtime key you supply must begin sk-bf-.
edisyl-bifrost-encryption	BIFROST_ENCRYPTION_KEY	What every stored provider credential is encrypted under. Cannot be rotated — back it up.

The key the applications send with every inference call, LLM_GATEWAY_API_KEY in edisyl-llm-gateway, is not generated.

Confirm against your chart version

LLM_GATEWAY_API_KEY is the gateway’s runtime key: set it to the same value as BIFROST_RUNTIME_KEY. The chart generates the runtime key as sk-bf- followed by random characters, so the simplest way to keep the two equal is to mint one key and supply it to both. The configurator writes REPLACE_WITH_GATEWAY_RUNTIME_KEY in both places in secrets.local.yaml, and its install steps replace it with one locally minted key. Safe to re-run: once the placeholder is gone there is nothing to replace.

KEY="sk-bf-$(openssl rand -hex 24)"
sed -i.bak "s/REPLACE_WITH_GATEWAY_RUNTIME_KEY/$KEY/g" secrets.local.yaml && rm secrets.local.yaml.bak
unset KEY

secrets:
  managed:
    edisyl-bifrost:
      data:
        BIFROST_RUNTIME_KEY: "sk-bf-..."
    edisyl-llm-gateway:
      data:
        LLM_GATEWAY_API_KEY: "sk-bf-..."    # the same value

Under external-secrets, map the one store entry to both:

secrets:
  managed:
    edisyl-llm-gateway:
      keys: []
      remoteKeys:
        LLM_GATEWAY_API_KEY: BIFROST_RUNTIME_KEY

Under Argo CD or external-secrets the chart generates nothing, so all four edisyl-bifrost* keys have to come from you or your store. See Known limitations.

Upgrading from a chart before 0.36.0

BIFROST_RUNTIME_KEY was BIFROST_PROBE_KEY until chart 0.36.0. Rename it in your values or your store before pinning 0.36.0 or later: under external-secrets every listed key is required, so a store that only carries BIFROST_PROBE_KEY fails the edisyl-bifrost ExternalSecret and it never syncs.

If you use a gateway you don’t run

Pointing bifrost.external.url at another deployment’s gateway — see Configuration — mounts edisyl-bifrost on the API without running a gateway here. The chart generates nothing for it: the admin credential and the runtime key are that gateway’s own, so generated values could never open it. All three keys have to come from you, and with chart-managed Secrets the render refuses them blank. LLM_GATEWAY_API_KEY is one of that gateway’s virtual keys too. edisyl-bifrost-encryption is not needed; it stays with whoever runs the gateway.

Or keep them in a secret manager instead

For anything beyond a trial, don’t hold credentials in a file at all. Set secrets.provider: external-secrets and the chart renders one ExternalSecret per managed secret instead of a populated Secret, reading each key from your store at runtime.

secrets:
  provider: external-secrets
  externalSecrets:
    # Name a store you already have.
    storeName: my-cluster-store
    storeKind: ClusterSecretStore

This needs the External Secrets Operator installed in the cluster. The chart renders the ExternalSecret resources and nothing more — the store itself is yours, so any backend the operator supports works: AWS Secrets Manager, Doppler, HashiCorp Vault, Google Secret Manager, Azure Key Vault.

The chart can also render a Doppler ClusterSecretStore for you rather than you creating one:

secrets:
  externalSecrets:
    createStore:
      project: your-doppler-project

storeName then defaults to <namespace>-doppler, and storeKind must stay ClusterSecretStore — the chart refuses the mismatch rather than rendering a store nothing reads.

A value changed in the store reaches its Secret within secrets.externalSecrets.refreshInterval, 15 minutes by default since chart 0.48.0 — each key is its own request to the store, so a shorter interval multiplies the load on it. After rotating a compromised credential, force the sync rather than wait, then restart the pods that mount it:

kubectl -n edisyl annotate externalsecret <name> force-sync="$(date +%s)" --overwrite

Two things that catch people

The chart generates nothing in this mode. Everything listed under What the chart generates has to be in your store before you install — the chart writes nothing to it. Put INNGEST_SIGNING_KEY there as even-length lowercase hex, ENCRYPTION_KEY as 32 random bytes base64-encoded (openssl rand -base64 32), and BIFROST_RUNTIME_KEY with its sk-bf- prefix, which nothing can check on this path. Miss them and Inngest, encryption and inference break with no warning at install time.

Every key a managed secret lists is a required remote reference, and one missing entry fails the whole ExternalSecret — taking every other key in that family down with it, not just the one you forgot. An integration you do not use is left out by emptying its list: a group with no keys, remoteKeys or templateData renders no ExternalSecret, and the apps mount it as optional.

secrets:
  managed:
    # The reranker. Optional: only a rerank call fails without it.
    edisyl-voyage:
      keys: []
    # TypeSafe rewording, mounted on the API since chart 0.50.0. Without this,
    # the store must carry TYPESAFE_API_KEY and TYPESAFE_BASE_URL.
    edisyl-typesafe:
      keys: []

You still have to mint the signing keys

A secret manager is where the auth signing material lives, not something that produces it. Generate it exactly as above, then load it into your store rather than writing auth-keys.local.yaml. GOTRUE_JWT_KEYS, GOTRUE_JWT_SECRET and GOTRUE_SAML_PRIVATE_KEY keep their names; the other two are read under different store names:

Secret and key	Store key
edisyl-auth-client → AUTH_ANON_KEY	AUTH_SELFHOSTED_ANON_KEY
edisyl-auth-admin → AUTH_ADMIN_TOKEN	AUTH_SELFHOSTED_ADMIN_TOKEN