---
title: "Operating Kodema: install, upgrade and recover"
description: "Choose a Kodema installation, deploy the reference charts on Kubernetes, monitor health and metrics, upgrade, back up and restore, and know the 0.2.1 limits."
canonical_url: https://optimi.com/en/docs/kodema/operations
md_url: https://optimi.com/en/docs/kodema/operations.md
language: en
alternates:
  - lang: fr
    url: https://optimi.com/fr/docs/kodema/exploitation
    md_url: https://optimi.com/fr/docs/kodema/exploitation.md
last_updated: 2026-11-14
---

# Operations

This page is for the person who installs and runs Kodema. It summarizes the
runbooks of the repository for **Kodema 0.2.1** and links to them for the
procedures.

## Choose an installation

| Installation | Use it for | Identity | Keys |
| --- | --- | --- | --- |
| **`kodema up`** | Trying Kodema and developing pipelines on your machine | Built-in sign-in, four personas, no password | Local keyfile |
| **Podman bundle** (`deploy/podman`) | A single-host test installation with a separate identity provider; the release smoke test runs on it | OAuthSonas, a development provider | Local keyfile |
| **Kubernetes** (Helm charts, OpenTofu roots) | Production | Your OIDC provider | One cloud KMS |

The first two are `test` environments: one host, trusted local operators. Only
the Kubernetes deployment runs the `production` profile, which refuses the test
adapters.

## Run a local installation

The **[quickstart](/en/docs/kodema/getting-started)** covers `kodema up`. For
day-to-day operation:

```sh
kodema status            # --json for a machine-readable report
kodema up                # safe to repeat; renews the credentials
kodema down              # stop, keep the state
kodema down --purge      # remove containers, volumes and state
```

- State lives in `~/.kodema/instances/<name>`: the CA, the certificates, the
  keyfile, the tokens and `env.sh`.
- The administrator tokens last 24 hours and the Git service credential 8
  hours. `kodema up` renews them.
- `kodema up --name <name> --listen <address>` runs a second installation.
- To upgrade, run the installer again, then `kodema up`.

The Podman bundle installs the released image from a checkout of the same
version:

```sh
git clone --branch v0.2.1 https://github.com/optimiware/kodema && cd kodema
podman pull ghcr.io/optimiware/kodema/kodema:0.2.1
KODEMA_LOCAL_KODEMA_IMAGE=ghcr.io/optimiware/kodema/kodema:0.2.1 ./deploy/podman/install.sh
```

It needs Podman 5, `git` 2.44, `jq`, `openssl`, `curl` and `socat`. Its
[runbook](https://github.com/optimiware/kodema/blob/master/docs/runbooks/podman-quickstart.md)
covers sign-in, provisioning a project repository, a protected task,
credential renewal and troubleshooting.

## What a release publishes

| Artifact | Where |
| --- | --- |
| `kodema` and `kodactl` for Linux and macOS, amd64 and arm64 | Archives attached to the [GitHub release](https://github.com/optimiware/kodema/releases), with `SHA256SUMS` |
| The `kodema` image, `linux/amd64` and `linux/arm64` | `ghcr.io/optimiware/kodema/kodema:<version>`; the release notes give the digest |
| The charts `kodema` and `kodema-stack` | `oci://ghcr.io/optimiware/kodema/charts` |

`latest` follows releases that are not pre-releases. A released version is
never replaced. Nothing is signed and no SBOM is produced. Pin the image by the
digest from the release notes in production.

## Deploy on Kubernetes

The repository ships **reference deployments, not production-grade
infrastructure**. Treat every default as a starting point to review, size and
test. The
[production deployment guide](https://github.com/optimiware/kodema/blob/master/docs/runbooks/production-deployment.md)
is the procedure; this section is its map.

**What ships**

| Piece | What it gives you |
| --- | --- |
| OpenTofu roots `deploy/scw`, `deploy/gcp`, `deploy/aws` | A cluster, managed PostgreSQL, a KMS key, a private network and remote state |
| Chart `kodema` | One control-plane Deployment, an HTTPS Service, RBAC, a retained-log volume, tenant namespaces with limits and network policies |
| Chart `kodema-stack` | The application chart plus one TLS NATS JetStream StatefulSet |
| `scripts/deploy`, `scripts/bootstrap-secrets`, `scripts/bootstrap-state`, `scripts/image` | The OpenTofu and Helm lifecycle, the cluster Secrets, the image build |

**What it does not include**

- **No control-plane high availability.** One replica with a `Recreate`
  rollout; an upgrade is a short outage.
- **No NATS cluster.** One replica; events persist in PostgreSQL.
- **No ingress.** The chart renders a `ClusterIP` Service on port 8443. Public
  exposure, DNS and an external certificate are yours.
- **No certificate manager, monitoring stack, or backup automation** beyond
  the managed database’s own backups.
- **No resource quota** per tenant namespace, only a `LimitRange`.

**Cloud status**

| | Scaleway | Google Cloud | AWS |
| --- | --- | --- | --- |
| Cluster | Kapsule | GKE | EKS |
| Key service | Key Manager (`scw-kms`) | Cloud KMS (`gcp-kms`) | AWS KMS (`aws-kms`) |
| KMS credential | Static API key | Workload Identity | IRSA |
| Has run live | Yes, for 0.1.0 | No | No |

All three are supported targets. Scaleway is the only one that has run live;
AWS and GCP are implemented and not validated against a real cloud.

**The sequence**

```sh
scripts/bootstrap-state <scw|gcp|aws> --project <id>      # once per cloud account; --account for aws
scripts/deploy <cloud> init
scripts/deploy <cloud> plan
scripts/deploy <cloud> apply
scripts/deploy <cloud> kubeconfig
scripts/bootstrap-secrets <cloud>
scripts/deploy <cloud> migrate                            # before the controller serves traffic
scripts/deploy <cloud> helm
```

`migrate` and `helm` need the image, pinned by digest, in
`KODEMA_IMAGE_REPOSITORY` and `KODEMA_IMAGE_DIGEST`. Then
`kodema bootstrap-admin` creates the first tenant and administrator from an
OIDC token.

**Production requirements the server enforces**

- NATS over TLS: a single `tls://` URL.
- PostgreSQL with `sslmode=verify-full` on all four role connections.
- Exactly one cloud KMS, with a key reference of the matching family.
- An OIDC issuer over HTTPS: `KODEMA_OIDC_ISSUER`, `KODEMA_OIDC_AUDIENCE`,
  `KODEMA_OIDC_JWKS_URL` and `KODEMA_OIDC_TENANT_CLAIM`.

**Network**

Each tenant gets a namespace `kodema-tenant-<tenant>` with default-deny ingress
and egress. Workloads may reach cluster DNS, the control plane and, when
enabled, the Git service. Any other destination, such as a package registry,
must be added deliberately, by FQDN where the cluster runs Cilium. Tenant
workloads never reach PostgreSQL or NATS.

**Optional blocks**, off by default: `gitService` (one `kodema git serve`
StatefulSet per tenant, with a volume that is kept on uninstall) and `services`
(one `kodema services` Deployment per adapter). The model gateway, the MCP
proxy and private registries need an operator overlay: the chart has no values
for them.

## Health and metrics

| Endpoint | Meaning |
| --- | --- |
| `GET /healthz` | The process can answer. It calls no dependency |
| `GET /readyz` | 200 only when PostgreSQL, the broker and the key provider all answer; otherwise 503 |
| `GET /metrics` | Prometheus text: `kodema_publication_backlog`, `kodema_consumer_lag`, `kodema_pending_cleanup`, and the counters `kodema_refused_pushes_total`, `kodema_refused_reports_total`, `kodema_dependency_errors_total` |

All three are served on the HTTPS listener without authentication. The chart
ships no scrape configuration, dashboard or alert rule. Alert at least on:

| Signal | Why it matters |
| --- | --- |
| `/readyz` not 200 for five minutes | A dependency is down; no new work can be admitted safely |
| Pod restarts | A configuration refusal, a certificate or key problem, or a crash loop |
| Publication backlog growing | Events are not reaching NATS. Executions still advance |
| Cleanup pending with exhausted attempts | A Job or a volume cannot be removed; capacity is at risk |
| Database backup older than your objective | The data-loss window is too wide |
| Retained-log or JetStream volume above 80% | Log retention and publication are bound by capacity |

A broker outage degrades publication and live logs; it does not stop
executions, request answers or deadlines.

## Inspect an execution’s cleanup

The outcome of an execution and the cleanup of its resources are independent.
`kodactl inspect <execution-id>` returns a `cleanup` object with its `state`
(`none`, `pending` or `complete`), the workload and workspace identities, the
attempts and the last error.

A pending cleanup with exhausted attempts is visible, not done. **Do not delete
the Job to make the record look clean**: deleting a Kubernetes Job does not
prove that the process stopped, and an uncertain stop blocks the handoff of the
workspace.

## Upgrade

1. Read the migrations of the target version. Check for exclusive locks or
   large rewrites.
2. Take or verify a database backup, and confirm that the key Secret is backed
   up.
3. Pin the new image by digest.
4. Run `scripts/deploy <cloud> migrate`, ideally with the old controller
   stopped.
5. Run `scripts/deploy <cloud> helm`. Expect a short gap in readiness and
   publication.
6. Check `/readyz`, run a small pipeline and read its logs.

Migrations are **forward-only**. The supported rollback is redeploying the
previous image digest, and it is safe only when the new migrations are
compatible with the previous binary. There is no schema rollback and no
automatic data rollback: restore from a verified backup instead.

## Back up and restore

Back up as **one recovery set**:

- PostgreSQL, which includes the event table and the encrypted secrets;
- the key Secret (`kodema-keys`) and what authenticates to the KMS;
- the retained-log volume and the JetStream volume.

A database backup without the key cannot decrypt the secrets, and the key alone
restores nothing. Workspaces are disposable and excluded. Git repositories and
registries belong to their operators; coordinate their retention, because
commits and digests named by retained executions must stay reachable.

The reference deployments configure the managed database’s scheduled backups
only. Volume and Secret backups are yours.

Restore in this order: migrate the target, preflight the backup, **fence** the
old controllers, services and consumers, restore, reconcile workloads and
external effects from their stable identities, then enable dispatch. An
execution whose workspace cannot be recovered fails visibly; an operator may
start a fresh attempt.

| Recommended objective | Value |
| --- | --- |
| Recovery point | 15 minutes or less |
| Recovery time | 4 hours or less |
| PostgreSQL recovery window | 30 days |

Kodema does not enforce these. The reference Scaleway configuration backs up
every six hours, which misses the 15-minute objective, and no service has been
resumed from a restored database on a cloud deployment, so there is no measured
recovery time. Measure yours with a drill.

| Default retention | Period |
| --- | --- |
| Logs | 30 days |
| Ordinary evidence | 90 days |
| Audit metadata | One year |
| Operation receipts | Until the project is deleted |
| Raw model and tool content | Off; an authorized opt-in starts at seven days |

## Credentials and rotation

| Credential | Lifetime and rotation |
| --- | --- |
| Execution tokens | At most eight hours, renewed by the controller, revoked with the execution |
| Service account tokens | An explicit expiry; `service.credential.rotate` replaces one at once |
| Tenant key | `tenant.rotate_key` invalidates every execution and service token of the tenant |
| Installation PKI | `scripts/bootstrap-secrets <cloud> --rotate-pki` replaces the CA and the leaves; restart the broker and the controller together. There is no automatic renewal |
| Installation key (`kodema-keys`) | Created once. Deleting or regenerating it makes every stored secret unreadable |

Back up the PKI work directory of `scripts/bootstrap-secrets` together with the
key Secret: the CA’s private key exists only there.

## Known limitations in 0.2.1

**Local installation**

- `kodema up` needs Podman and refuses a Docker engine. It has run on Linux
  only; the macOS binaries are built and have not been run.
- The built-in sign-in asks for no password and answers its own machine only.
- `kodactl run` prints a workload’s output when the execution ends, not while
  it runs.
- The installer installs 0.2.1 or newer; earlier releases have no `kodema`
  archive.

**Pipelines and Git**

- An action that declares `secrets:` is refused, because the workload secret
  allow-list is not configurable yet.
- Git ref deletes are refused, also where `RW+` allows them.

**Integrations**

- NATS subscriptions by people and browsers are refused; read events and logs
  over HTTP or MCP.
- The `webhook` and `email` service kinds deliver nothing outbound;
  email-to-task intake works.
- An unauthenticated MCP `tools/list` answers 200; only tool calls answer 401.
- The workload model proxy forwards any path of a configured provider; it has
  no model or endpoint allowlist.
- `kodema auth serve` does not limit the rate of sign-in attempts or client
  registrations.

**Deployment**

- Scaleway is the only cloud that has run live. The `linux/arm64` image is
  built and has not been run.
- Artifacts are not signed.

The [changelog](https://github.com/optimiware/kodema/blob/master/CHANGELOG.md)
keeps this list for each version.
