Skip to content

Runtime Operations

You do not need to read this unless you are rotating the proxy policy-bundle Ed25519 signing key or renewing the cert-manager-backed sidecar webhook chain. The default rollout in Deployment Overview does not require these tasks on day 1.

Use this page for the small but important operator tasks that sit between "feature shipped" and "production stayed healthy":

  • rotating the Ed25519 key used to sign cached proxy policy bundles
  • verifying and renewing the cert-manager-backed CA chain for the sidecar mutating webhook

These are not separate products. They are the runtime maintenance tasks for the same self-hosted agent-bom deployment.

Proxy policy-cache signing key rotation

When AGENT_BOM_PROXY_POLICY_CACHE_ED25519_PRIVATE_KEY_PEM is set, each proxy persists its cached gateway policy bundle together with an Ed25519 signature and fails closed if the signature is missing, tampered, or produced by a different key.

That hardening changes the rotation rule:

  1. rotate the signing key while the control plane is reachable
  2. restart or refresh proxies so they fetch and re-sign a fresh policy bundle
  3. only then rely on cached-startup fallback again

Why the order matters

The proxy uses the currently mounted private key to verify the cached signature at startup. If you mount a new key while the cache still contains a bundle signed by the old key, a cold start will reject that cache and fail closed.

That is the intended safety property. Rotation should therefore be treated as a rolling refresh, not a "swap the secret and walk away" change.

Generate a new Ed25519 key pair:

openssl genpkey -algorithm ed25519 -out /tmp/proxy-policy-cache-ed25519.pem
openssl pkey -in /tmp/proxy-policy-cache-ed25519.pem -pubout -out /tmp/proxy-policy-cache-ed25519.pub

Update the secret or environment source that backs AGENT_BOM_PROXY_POLICY_CACHE_ED25519_PRIVATE_KEY_PEM.

For Kubernetes, that usually means updating the Secret referenced by the proxy or sidecar deployment:

kubectl -n agent-bom create secret generic agent-bom-proxy-policy-signing \
  --from-file=AGENT_BOM_PROXY_POLICY_CACHE_ED25519_PRIVATE_KEY_PEM=/tmp/proxy-policy-cache-ed25519.pem \
  --dry-run=client -o yaml | kubectl apply -f -

Then roll the affected proxy workloads while the control plane is healthy:

kubectl -n agent-bom rollout restart deploy/agent-bom-sidecar-injector
kubectl -n <workload-namespace> rollout restart deploy/<mcp-workload>

For laptop or MDM rollout, regenerate the onboarding bundle and push the updated environment/config through the same managed path used for initial install:

agent-bom proxy-bootstrap \
  --bundle-dir ./bundle \
  --control-plane-url https://agent-bom.internal.example.com \
  --control-plane-token "$CONTROL_PLANE_TOKEN" \
  --enrollment-name corp-laptop-rollout \
  --mdm-provider jamf

Break-glass recovery

If a proxy starts with a newly rotated key while the control plane is unreachable, it may reject the old cached bundle and exit.

Recovery order:

  1. restore control-plane reachability if possible
  2. remove the stale cache and signature files
  3. restart the proxy so it fetches a fresh bundle

Default cache paths:

  • ~/.agent-bom/cache/gateway-policies.json
  • ~/.agent-bom/cache/gateway-policies.json.sig

Rotation checklist

  • rotate the secret value that backs AGENT_BOM_PROXY_POLICY_CACHE_ED25519_PRIVATE_KEY_PEM
  • perform rolling restarts while the control plane is reachable
  • confirm proxies can still pull policy and start cleanly
  • archive the public key if your change process requires evidence of the new key

Sidecar webhook certificate and CA renewal

The packaged sidecar mutating webhook is cert-manager-backed. The chart ships:

  • an Issuer or externally supplied issuerRef
  • a Certificate
  • a MutatingWebhookConfiguration with cert-manager.io/inject-ca-from

That means the normal case is automatic:

  • cert-manager renews the webhook serving certificate before expiry
  • the Secret mounted into the injector Deployment updates
  • cert-manager injects the matching CA bundle into the webhook configuration

Relevant chart values

  • sidecarInjection.enabled=true
  • sidecarInjection.certManager.enabled=true
  • sidecarInjection.certManager.duration
  • sidecarInjection.certManager.renewBefore
  • sidecarInjection.certManager.createSelfSignedIssuer
  • sidecarInjection.certManager.issuerRef.*

What to verify

Check the certificate and its renewal window:

kubectl -n agent-bom get certificate,secret | grep sidecar-injector
kubectl -n agent-bom describe certificate agent-bom-sidecar-injector

Check that the webhook configuration is still receiving the injected CA bundle:

kubectl get mutatingwebhookconfiguration agent-bom-sidecar-injector -o yaml | rg "caBundle|inject-ca-from"

Check that the injector pod is serving with the expected Secret:

kubectl -n agent-bom describe deploy agent-bom-sidecar-injector
kubectl -n agent-bom get pods -l app.kubernetes.io/component=sidecar-injector

Rotation and renewal behavior

If you use the chart-managed self-signed issuer, cert-manager owns the full renewal path.

If you use an external issuer:

  • issuerRef points to the cluster or namespace issuer you manage
  • renewal cadence comes from that issuer plus the chart's duration / renewBefore
  • cert-manager.io/inject-ca-from still keeps the webhook CA bundle aligned

Recovery steps if admission starts failing

Symptoms usually show up as failed Pod creates in opted-in namespaces.

Check in this order:

  1. Certificate condition is Ready=True
  2. serving TLS Secret exists and was updated recently
  3. MutatingWebhookConfiguration still has the inject-ca-from annotation
  4. injector Deployment has healthy pods
  5. cert-manager controller logs do not show issuance or injection failures

If the certificate renewed but the injector pod still serves the old mount, restart the injector Deployment:

kubectl -n agent-bom rollout restart deploy/agent-bom-sidecar-injector

Release sign-off for runtime operations

Before calling a release "operator-ready", verify:

  • proxy policy signing is either disabled intentionally or documented with a concrete secret owner and rotation path
  • sidecar injection uses cert-manager with a known issuer and renewal window
  • the recovery commands above are present in the team's internal runbook or deployment ticket