--- title: "Operating Kodama: identity, configuration and recovery" description: "Bootstrap a Kodama tenant, manage Helm configuration and credentials, diagnose workloads, upgrade releases and recover consistent source, evidence and state." canonical_url: https://optimi.com/en/docs/kodama/operations md_url: https://optimi.com/en/docs/kodama/operations.md last_updated: 2018-10-20 --- # Operations Operate Kodama as a control plane with durable external backends and scoped Kubernetes execution. Start with [the Helm walkthrough](/en/docs/kodama/getting-started) for installation; this page covers administrative setup and the ongoing lifecycle. ## Bootstrap the first tenant Bootstrap creates the first tenant and administrator using an explicitly designated OIDC subject. It uses the database owner credential in a one-time administrative Job, separate from the serving process. Run it after the Helm migration has completed. Install and configure the CLI as described in Getting Started. Obtain a short-lived administrator **access token** for the factory audience through OAuth. For this one-time operation, you can use an isolated CLI file store to write the token to `admin.token` without printing it: ```sh umask 077 mkdir -m 700 bootstrap-login export KODAMA_CREDENTIAL_DIR="$PWD/bootstrap-login" kodama auth login --credential-store file jq -jr '.accessToken' "$KODAMA_CREDENTIAL_DIR"/*.json > admin.token export TOKEN="$(cat admin.token)" ``` Use an empty `bootstrap-login` directory and the factory context configured for your issuer. File storage is an explicit opt-in; ordinary `kodama auth login` continues to use the OS keyring. The issuer’s exact `sub` for this person is the `--eligible-subject` below. Do not substitute an email unless it is actually the issuer’s subject. Create a temporary administrative Secret with the owner connection, its CA bundle and the access token: ```sh kubectl -n kodama-system create secret generic kodama-bootstrap \ --from-file=database-migration-url=migration-secrets/database-migration-url \ --from-file=ca.crt=migration-secrets/ca.crt \ --from-file=admin.token=admin.token ``` Save `bootstrap.yaml`; replace the image digest, issuer, audience and subject: ```yaml apiVersion: batch/v1 kind: Job metadata: name: kodama-bootstrap namespace: kodama-system spec: backoffLimit: 0 activeDeadlineSeconds: 300 template: spec: restartPolicy: Never automountServiceAccountToken: false securityContext: runAsNonRoot: true runAsUser: 65532 runAsGroup: 65532 fsGroup: 65532 seccompProfile: {type: RuntimeDefault} containers: - name: bootstrap image: ghcr.io/optimiware/kodama@sha256:1111111111111111111111111111111111111111111111111111111111111111 args: - bootstrap - --tenant-id=t_acme - --tenant-key=acme - --display-name=Acme - --operator-id=op_initial - --eligible-subject=replace-with-verified-subject - --token-file=/secrets/admin.token env: - {name: KODAMA_ENV, value: production} - {name: KODAMA_OIDC_ISSUER, value: "https://identity.example.com"} - {name: KODAMA_OIDC_AUDIENCE, value: "https://kodama.example.com/api"} securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: {drop: [ALL]} volumeMounts: - {name: secrets, mountPath: /secrets, readOnly: true} volumes: - name: secrets secret: secretName: kodama-bootstrap defaultMode: 0440 ``` ```sh kubectl apply -f bootstrap.yaml kubectl -n kodama-system wait job/kodama-bootstrap \ --for=condition=Complete --timeout=300s kubectl -n kodama-system logs job/kodama-bootstrap ``` An identical bootstrap request observes the recorded outcome; changing its parameters conflicts. After success, remove the temporary administrative Job and Secret. Keep the short-lived token only while performing the initial project and role setup from Getting Started, then remove the local temporary credentials: ```sh kubectl -n kodama-system delete job kodama-bootstrap kubectl -n kodama-system delete secret kodama-bootstrap kodama auth logout --credential-store file rm admin.token rmdir bootstrap-login unset TOKEN KODAMA_CREDENTIAL_DIR ``` Logout removes a local credential; it does not revoke the token at the issuer. Subsequent users sign in and receive explicit tenant/project memberships. ## Installation configuration Keep the chart, CLI and server on a coordinated release. Pin server and injector images by digest. The [chart values](https://github.com/optimiware/kodama/blob/main/deploy/helm/kodama/values.yaml) describe the full configuration surface; important relationships are: | Helm values | Operational responsibility | | --- | --- | | `installation.id` | Stable encryption context identity; preserve through restart and restore | | `authority.mode` | Select standalone or a release-supported hosted integration explicitly | | `secrets.runtime.existingSecret` | Serving database connection, broker/object/Git credentials, cursor/idempotency keys and runtime signing material | | `secrets.migration.existingSecret`, `migration.runtimeRole` | Owner connection for migrations and the non-owner runtime role to grant | | `config.nats`, `config.s3`, `config.oidc` | Backend endpoints, object prefix, issuer and audience | | `kms` and `serviceAccounts.server.annotations` | One cloud key provider and the authorized cloud identity | | `execution.tenants` | Tenant-to-namespace mapping | | `execution.runtime` | Workload-reachable HTTPS callback origin, signing generation and allowed destinations | | `execution.network`, `quota`, `limitRange`, `job`, `volumes` | Tenant egress, concurrency/resource bounds and bounded scratch storage | | `ingress` | Public TLS endpoint and existing certificate Secret | The chart selects the core server assembly and `KODAMA_ENV=production`. `KODAMA_CONFIG_FILE` points to its settings document. Environment overrides take precedence; secret values stay in files, outside the settings document. Unknown fields and unsupported provider selections fail configuration. The initial profile has one server replica and a `Recreate` rollout. Size PostgreSQL, event storage, object retention and workload quotas independently; raising tenant concurrency does not automatically increase backend capacity. ## Credentials and trust The server must never hold the migration owner credential during normal operation. Its PostgreSQL role must neither own application tables nor have superuser or `BYPASSRLS` privileges. Tenant predicates and RLS apply alongside application authorization. Maintain these distinct credentials: - **Git checkout and publication:** URL-prefix and tenant-scoped references in `git-credentials.json`; project repository settings select them. - **Object storage:** keys scoped to the configured bucket/prefix, with the credential mode supported by your release. - **Runtime signing:** a PEM P-256 key and explicit generation. Never reuse a generation for a different key. - **Cursor and idempotency keys:** raw values of at least 32 bytes; preserve their bytes rather than trimming or re-encoding them. - **Deployment credentials:** project environment credentials scoped to the intended cluster namespace and actions. - **Provider/tool credentials:** retained behind the model gateway and MCP proxy. Use the exact OIDC issuer and access-token audience. The public client, runtime callback and backend services must each validate their own TLS peer. A callback reachable from your laptop is not necessarily reachable from a tenant Job. Keep `runtime-ca.crt` synchronized with callback trust and renew certificates before they expire. For secret rotation, create a new version for future admissions, validate it, then revoke the old version according to the active-run policy. Existing pins and retained backup records determine which wrapping keys must remain available. Changing a Secret’s bytes is not proof that every running process has reloaded it; perform the release’s documented rollout/reload procedure. ## Enable agents, delivery and notifications Installation policy sets the ceiling; project configuration selects resources inside it. Configure the supported installation documents and their mounted file locations for your release: | Policy | What to configure | | --- | --- | | Model/tools (`model-tools.json`) | Approved provider origins, MCP destinations and qualified image/wrapper/harness tuples | | Runtime actions (`action-ceiling.json` or Kubernetes execution policy) | Actions such as `model.invoke`, `tool.invoke` and `deployment.call`, bounded by the attempt’s pinned context | | Registry (`registry.json`) | Allowed OCI registries, repositories and provenance verification | | Delivery tools (`delivery-tools.json`) | Exact apply/cleanup images and commands whose replay protocol is qualified | | Git triggers (`git-triggers.json`) | Authenticated native push notification selection and installation bindings | | Business loop (`business-loop.json`) | Canonical notification origin, destination policy and QA mapping identity/configuration | Use the release’s supported policy projection mechanism: adding an arbitrary key to a Kubernetes Secret does not mount it when the chart enumerates Secret keys. An absent approval grants no model, tool or deployment authority. Register project model settings and MCP servers through the API, then register an immutable agent profile. Create delivery environments with scoped credentials before registering steps that reference them. Use `kodama environment --help`, `kodama agent --help` and the public API contract for request fields. For previews, configure the project’s `kodama-cleanup` pipeline with a string input named `deployment_id` and the qualified cleanup command. Expiry schedules cleanup of the recorded deployment; it must not delete unrelated target objects. Business-signal adapters remain authenticated API clients. They own incoming signature validation, event-to-work mapping and long-lived event deduplication. ChatOps notifications carry links to currently authorized resources; a forwarded link grants no permission and reading it cannot decide a gate. ## Health and daily observation `/healthz` reports process liveness; `/readyz` checks the migrated database, JetStream, S3, OIDC and worker health. During shutdown, readiness drains before HTTP and workers stop. Existing workloads are reconciled by the next process. ```sh kubectl -n kodama-system get deployment,pods,jobs kubectl -n kodama-system logs deployment/kodama --since=15m kubectl -n kodama-tenant-acme get jobs,pods curl --fail https://kodama.example.com/readyz kodama run get "$RUN_ID" --output json kodama run steps "$RUN_ID" --output json ``` Correlate tenant, project, work, workspace, run, step, attempt and request IDs. Observe outbox age, consumer lag, waiting attempts, publication age, pending cancellation, review age and preview cleanup backlog. Keep high-cardinality IDs in protected records/log fields rather than unbounded shared metric labels. | Symptom | Investigate and recover | | --- | --- | | Migration hook fails | Owner connection, CA, runtime role name, schema compatibility and Job logs | | Server is unready | Named backend/worker error; distinguish unsupported release configuration from connection failure | | CLI login succeeds, operation denied | Token audience and current tenant/project role; an admin is not automatically an engineer or reviewer | | Job cannot start | Image digest/pull access, namespace policy, quota, resource bounds and admission material | | Job runs but no result arrives | Workload-reachable callback URL, TLS trust, egress and attempt authorization | | Run remains queued | Dependencies, writer reservation, execution limits and dispatch diagnostics | | `output_invalid` / `output_upload_failed` | Declared names/types/paths, produced files and object-store connectivity | | Source publication conflicts | Inspect expected/proposed Git refs and human pushes; retain the proposal for rework | | Gate cannot progress | Subject/evidence freshness, reviewer revocation, cancellation and any recorded decision | | External action has an unknown outcome | Observe its stable action/deployment identity before repeating it | | Logs are incomplete | Inspect capture origin and completeness independently of command outcome | | Notification retries exhausted | Read delivery history and explicitly replay with current authority and the retained payload/key | ## Upgrades and rollback 1. Review the release’s schema, CLI, runtime and backend compatibility notes. 2. Verify a consistent recovery point and access to the required wrapping keys. 3. Quiesce new admission and side-effecting integrations when mixed-version execution is unsupported; settle or inventory active attempts. 4. Run `helm upgrade` with the new release chart and image digest. The owner-role migration hook precedes the serving process rollout. 5. Check readiness, authenticate, run a bounded pipeline and retrieve its report and logs before reopening normal admission. A binary rollback is valid only with a compatible schema. `helm rollback` does not reverse database migrations, Git publication or external deployment effects. Restore and reconcile when a compatible rollback is unavailable. ## Backup and recovery Back up the whole durable set: PostgreSQL/WAL, Git refs and objects, S3 evidence, registry images, configuration and required KMS versions. Retention must cover the same recovery points. Uncommitted scratch files are outside that set. Use **RPO 15 minutes**, **RTO 4 hours** and **30-day PITR** as recovery planning objectives to validate with installation drills. The retention policy baseline is 90 days for ordinary evidence, 30 days for logs, one year for audit and a 24-hour default preview TTL. Active references extend retention. Raw model/tool content is off by default; enabling it requires a separate bounded policy. These are operational objectives, not guarantees supplied by a Helm install. During restore: 1. Stop admission and side-effecting consumers; fence old controllers and publishers, including credentials they may still hold. 2. Restore a consistent database/source/object/image set and verify decryption. 3. Rotate workload signing keys and invalidate old admission; separately settle remote operations and credentials already delivered. 4. Match workloads by attempt identity and instance UID. Reconcile pending Git publication and deployment records before another write or apply. 5. Replay durable events with inbox deduplication; release reservations only after uncertain effects are resolved. 6. Resume a bounded workflow and verify recovered history and evidence. ## Tenant offboarding and uninstall Stop new admission, revoke access, settle active work and apply retention before deleting tenant resources. Account separately for retained audit, evidence pins, supported releases and backup expiry. Do not destroy a shared KMS key while any retained data still needs it. `helm uninstall kodama -n kodama-system` removes the control plane while keeping tenant namespaces and their retained execution guardrails. Runtime-created Jobs, external databases, Git, S3, registry data, KMS keys and deployments require their own lifecycle procedure. Removing a tenant from Helm values likewise preserves its namespace rather than treating it as disposable storage. Return to **[Getting started](/en/docs/kodama/getting-started)** or **[pipeline examples](/en/docs/kodama/developer-guide)**.