> ## Documentation Index > Fetch the complete documentation index at: https://docs.popsink.com/llms.txt > Use this file to discover all available pages before exploring further. # Install Popsink with Helm for partners > Self-host the Popsink data plane on Kubernetes using the official Helm chart. This page is intended for **integrators** who install Popsink in their own Kubernetes cluster. The chart deploys a self-hosted **Popsink data plane** — the runtime that moves data from your sources to your targets. It connects to a **control plane** (managed by Popsink at `control-plane.popsink.com`, or your own) which orchestrates and monitors the deployment. See [Control plane and data plane](/deployment/control-data-plane) for the exact metadata exchanged between the two, and the network rules it implies. ## Architecture overview The chart deploys the following components: | Component | Purpose | Default | | ------------ | ------------------------------------------------------------------------------ | ------------ | | `data-plane` | Main API + UI. Talks to the control plane, launches connector workers as pods | **enabled** | | `kodansu` | Stateless Kafka-compatible broker, object-store backed | **enabled** | | `kora` | Confluent-compatible Schema Registry. **Needs its own PostgreSQL — see below** | **enabled** | | `kotatsu` | Optional topic / schema browser, reads the broker's S3 objects | **disabled** | | `postgresql` | Bitnami sub-chart, in-cluster Postgres for the data-plane DB | **enabled** | Connector workers are not part of the chart: the data-plane creates them as pods in its own namespace at runtime. You can disable any sub-component and bring your own: * **External Kafka** — `kodansu.enabled=false`, configure `defaultKafka.*` * **External Schema Registry** — `kora.enabled=false`, configure `schemaRegistry.*` * **External PostgreSQL** — `postgresql.enabled=false`, configure `externalDatabase.*` `kodansu` was previously called `tansu`, and the chart still reads a `tansu:` block for backwards compatibility (anything under `kodansu:` wins). New values files should use `kodansu:`. Two components that appeared in earlier versions of this page — `karapace` and `kafka-ui` — no longer exist in the chart. They were replaced by `kora` and `kotatsu` respectively. A `values.yaml` that still sets `karapace.*` or `kafkaUi.*` is silently ignored. The chart **can** render an `Ingress` (`ingress.enabled`, off by default), but it never installs an ingress controller: you still need your own (Traefik, NGINX, Istio…) and a DNS record pointing at it. Both options — chart-rendered or bring-your-own — are covered under [Ingress](#ingress) below. ## Prerequisites ### Cluster | Requirement | Minimum | | ----------------- | -------------------------------------------------------- | | Kubernetes | `1.23+` | | Helm | `3.8.0+` | | Worker nodes | 2 nodes, `amd64` | | Per-node | 4 vCPU, 16 GB RAM | | Persistent volume | SSD-backed StorageClass, ≥ 200 GB available | | Ingress | Traefik / NGINX / Istio (or any IngressClass-compatible) | | Object storage | S3-compatible bucket (recommended for the broker) | ### Access to Popsink images All images are published to a private GAR registry: ``` europe-west1-docker.pkg.dev/popsink-common-438615/onprem ``` Popsink will provide you with a service-account JSON token to pull these images. Let the chart create the `imagePullSecret` for you: ```yaml theme={null} imagePullSecret: create: true registry: europe-west1-docker.pkg.dev/popsink-common-438615/onprem token: | ``` Create the secret yourself, then reference it via `global.imagePullSecrets`: ```bash theme={null} kubectl create secret docker-registry popsink-registry \ --docker-server=europe-west1-docker.pkg.dev \ --docker-username=_json_key \ --docker-password="$(cat key.json)" \ -n popsink ``` ```yaml theme={null} global: imagePullSecrets: - popsink-registry ``` ## Onboarding flow (control plane) Before installing the chart, you need a **deployment ID** and a **deployment JWT token** issued by the control plane. Log into `https://control-plane.popsink.com` (or your own control plane). Open **Deployments → New deployment** and pick **Self-hosted**. * Optional pre-fill of PostgreSQL credentials (not stored). * **Retention strategy** — pick S3-compatible storage (recommended) or PostgreSQL. * **FQDN** — the public URL where the data plane will be reachable (e.g. `https://popsink.your-company.com`). The wizard outputs a `values-control-plane.yaml` snippet containing `controlPlaneUrl`, `deploymentId` and `deploymentJwtToken`. **Use these values verbatim** — they identify and authenticate your data plane against the control plane. Keep the wizard open. After `helm install`, the wizard leaves **Awaiting connections…** and the deployment flips to **Live** on the data plane's first heartbeat. ## Required secrets to generate yourself Beyond the values handed by the control plane, you must generate four secrets locally: | Secret | Format | How to generate | | ----------------------------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------- | | `adminCredentials.username` / `password` | any | Pick a strong password | | `jwt.secret` | random string ≥ 32 chars | `openssl rand -base64 48` | | `connectorConfigEncryptionKey.key` | URL-safe base64-encoded **32-byte** Fernet key | `python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"` | | `postgresql.auth.password` (or `externalDatabase.password`) | any | `openssl rand -base64 24` | These four secrets encrypt connector credentials, sign user sessions and protect the admin login. **Lose them and you lose access to your stored connector configs**. Back them up in your secret manager. `connectorConfigEncryptionKey.key` must **never be rotated**: every stored connector configuration is encrypted with it and becomes unreadable if it changes. Each deployment gets its own — never share one across deployments. If you prefer to manage secrets out of band (Vault, Sealed Secrets, External Secrets Operator…), set the corresponding `existingSecret` field instead. ## Schema registry database (Kora) Kora is Popsink's schema registry: every connector registers the structure of what it reads there, and every target resolves schemas from it. It is enabled by default and it **requires its own PostgreSQL role and database, which the chart does not create for you**. This is the step most first installs miss. The bundled PostgreSQL provisions the data-plane's own role and database (`postgresql.auth.*`) and nothing else, so leaving `kora.database` unset points Kora at a role that exists nowhere. From chart `0.1.0-alpha.162` the install aborts with the instructions below; on earlier charts it installs a pod that crash-loops on: ``` failed to connect to database: Backend("error returned from database: password authentication failed for user \"kora\"") ``` Run once against your PostgreSQL instance — managed services such as RDS or Cloud SQL included: ```sql theme={null} CREATE ROLE kora LOGIN PASSWORD 'a-strong-password'; CREATE DATABASE kora OWNER kora; ``` ```yaml theme={null} kora: database: host: "popsink.abc123.eu-west-2.rds.amazonaws.com" # bundled: "-postgresql" port: 5432 user: "kora" database: "kora" password: "a-strong-password" ``` No SQL to run — point Kora at the same role and database as the data-plane. This keeps a self-hosted install to a single database to operate, and is what the control-plane wizard generates. ```yaml theme={null} kora: database: host: "-postgresql>" port: 5432 user: "popsink" # same as externalDatabase.user / postgresql.auth.username database: "popsink" # same as externalDatabase.database / postgresql.auth.database password: "" ``` If you already operate a Confluent-compatible schema registry: ```yaml theme={null} kora: enabled: false schemaRegistry: url: "https://schema-registry.example.com" username: "popsink" password: "" # or: existingSecret + secretKeys.{username,password} ``` Credentials can also come from a secret rather than a literal — use `kora.database.existingSecret` with `secretKeys.password`, or `secretKeys.url` for a full connection URL. The three sources are mutually exclusive: an `existingSecret` combined with a literal `password` or `url` is rejected at render time. Kora also supports an Oracle backend: `kora.database.backend: oracle`, port `1521`, and `database` set to the service name (e.g. `FREEPDB1`). The chart selects the matching image automatically. ## Stream browser (Kotatsu) Kotatsu is **optional and disabled by default**. It is a read-only browser over the objects the broker wrote to S3, and it backs two things in the UI: * search, pagination and the timestamp column on the datamodel **Stream** tab; * the per-subscription **target lag** rows on the pipeline latency view. Without it, the Stream tab falls back to a live Kafka read (no search, no pagination) and the target-lag rows are simply absent. Pipelines, connectors, subscriptions and delivery are unaffected — it is safe to leave disabled in production. Because it reads the bucket the broker writes, it only means anything with S3 retention. To enable it: ```yaml theme={null} kotatsu: enabled: true ``` That is the entire configuration. Bucket, cluster id, region, endpoint and credentials all default to the broker's own (`kodansu.storage.*`, `kodansu.clusterId`) — the only combination that can work, since the two read and write the same objects. Set `kotatsu.s3.*` only to point Kotatsu at a replica of the bucket or at a separate read-only credential. On EKS with IRSA, leave the keys empty and annotate `kotatsu.serviceAccount.annotations` with the role ARN. On charts before `0.1.0-alpha.162` Kotatsu defaulted to **enabled**, so an install that had not opted into S3 aborted with `kotatsu.s3.bucket must be provided when kotatsu is enabled`. Setting `kotatsu.enabled: false` is the correct workaround on those versions. ## Broker storage (S3) Kodansu is a stateless Kafka broker that stores log segments in an object store. `kodansu.storage.engine` decides where records live, and both it and `kodansu.storage.aws.region` are required by the chart. ```yaml AWS — IRSA (recommended on EKS) theme={null} kodansu: storage: engine: "s3://my-popsink-bucket/kafka/" aws: region: eu-west-1 irsaRoleArn: arn:aws:iam::123456789012:role/popsink-kodansu assumedRoleArn: arn:aws:iam::123456789012:role/popsink-kodansu ``` ```yaml AWS — static credentials theme={null} kodansu: storage: engine: "s3://my-popsink-bucket/kafka/" aws: region: eu-west-1 accessKeyId: AKIA… secretAccessKey: ``` ```yaml MinIO / non-AWS S3 theme={null} kodansu: storage: engine: "s3://my-popsink-bucket/kafka/" aws: region: us-east-1 endpoint: http://minio.minio.svc.cluster.local:9000 allowHttp: true accessKeyId: minio secretAccessKey: minio123 ``` ```yaml PostgreSQL retention theme={null} kodansu: storage: engine: "postgres://user:password@db.example.com:5432/database" aws: # Required by the chart even here, and never read on this path. region: us-east-1 ``` The bucket must exist, be writable by the configured identity, and ideally have **object versioning** enabled. Before bumping the broker image on an existing bucket, read the upgrade notes on `tansu.image.tag` in the chart's `values.yaml`. Two of the tags cross an object-layout boundary and need a quiesce-and-flip rather than a rolling upgrade. PostgreSQL retention is simpler to operate but does not scale the same way, and rules out Kotatsu (there is no bucket to browse). If you don't want the in-cluster broker, set `kodansu.enabled=false` and `kora.enabled=false`, then point the data plane at your existing Kafka: ```yaml theme={null} kodansu: { enabled: false } kora: { enabled: false } defaultKafka: bootstrapServer: kafka.example.com:9093 securityProtocol: SASL_SSL saslMechanism: SCRAM-SHA-512 saslUsername: popsink saslPassword: caCert: |- -----BEGIN CERTIFICATE----- … cert: "" key: "" schemaRegistry: url: https://schema-registry.example.com username: popsink password: ``` ## Ingress The chart's built-in `ingress.enabled` flag is **off by default** and we recommend leaving it that way: ingress in production usually needs company-specific annotations (cert-manager, WAF, allow-lists). Provide your own `Ingress` resource that exposes: * the data-plane HTTP service (`-data-plane`, port `80`), * on the FQDN you declared in the control plane (`ingressUrl`). Example with NGINX + cert-manager: ```yaml theme={null} apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: popsink namespace: popsink annotations: cert-manager.io/cluster-issuer: letsencrypt-prod spec: ingressClassName: nginx tls: - hosts: [popsink.your-company.com] secretName: popsink-tls rules: - host: popsink.your-company.com http: paths: - path: / pathType: Prefix backend: service: name: -data-plane port: { number: 80 } ``` If you want the chart to render an `Ingress` for you, set `ingress.enabled=true` and configure `ingress.hostname`, `ingress.ingressClassName`, `ingress.tls`. The value of **`ingressUrl`** in the chart **must** match the public URL the control plane redirects users to (it is also used for OAuth-style callbacks): ```yaml theme={null} ingressUrl: "https://popsink.your-company.com" ``` ## Putting it together — minimal `values.yaml` ```yaml values.yaml theme={null} # ───── Identity (from the control-plane wizard) ───── controlPlaneUrl: https://control-plane-api.popsink.com/api controlPlaneFrontendUrl: https://control-plane.popsink.com deploymentMode: SELF_HOSTED deploymentId: deploymentJwtToken: token: # ───── Public URL of this data plane ───── ingressUrl: https://popsink.your-company.com # ───── Image pull ───── imagePullSecret: create: true token: # ───── Secrets you generated ───── adminCredentials: username: admin password: jwt: secret: connectorConfigEncryptionKey: key: # ───── In-cluster Postgres (default) ───── postgresql: enabled: true auth: password: primary: persistence: size: 50Gi storageClass: # ───── Schema registry database — NOT created by the chart ───── # Run once: CREATE ROLE kora LOGIN PASSWORD ''; # CREATE DATABASE kora OWNER kora; kora: database: host: -postgresql # or your RDS / Cloud SQL endpoint port: 5432 user: kora database: kora password: # ───── Broker + S3 ───── kodansu: storage: engine: "s3://my-popsink-bucket/kafka/" aws: region: eu-west-1 irsaRoleArn: arn:aws:iam::123:role/popsink-kodansu assumedRoleArn: arn:aws:iam::123:role/popsink-kodansu # ───── Optional stream browser (S3 retention only) ───── kotatsu: enabled: true ``` For production, swap the in-cluster Postgres for a managed one: ```yaml theme={null} postgresql: enabled: false externalDatabase: host: popsink.abc123.eu-west-2.rds.amazonaws.com port: 5432 user: popsink database: popsink password: ``` `externalDatabase.user` and `externalDatabase.database` are **not** guarded by the chart: left empty they render, install, and the data plane then crash-loops on an empty `DB_USER`. The database must exist before the install — the data plane runs its own migrations on startup. ## Install ```bash theme={null} # 1. Create the namespace kubectl create namespace popsink # 2. Install (chart is published as an OCI artifact) helm install popsink \ oci://ghcr.io/popsink/charts/data-plane \ -n popsink \ -f values.yaml \ --version ``` Use `helm search repo` or check the chart's `Chart.yaml` (`version:`) to pin the chart version. Pinning avoids surprise upgrades. Watch the rollout: ```bash theme={null} kubectl -n popsink get pods -w ``` When all pods are `Ready`, head back to the control-plane deployment page — the status should flip from **Offline** to **Live** within a minute. Open `ingressUrl` and log in with the `adminCredentials` you set. ## Production checklist * Pin the chart version (`--version`) and the application image (`image.tag`) — never deploy `latest`. * Use **external** PostgreSQL (`postgresql.enabled=false`, `externalDatabase.*`) backed by managed snapshots, not the in-cluster Bitnami chart. * Use **external** S3 with versioning + lifecycle policy for the broker. * Set explicit `resources.requests/limits` on every component (defaults target small-medium clusters; tune for your workload). * Configure `replicaCount ≥ 2` on `data-plane` (default), keep `tansu.replicaCount=3` (default). * Enable `pdb.create: true` (default) — at least 1 pod stays during node drains. * Provide `imagePullSecrets` via a sealed secret / external secrets controller, not plain values. * Back up the four secrets (admin credentials, JWT secret, Fernet key, DB password) in your secret manager. * Keep `allowDesignLogin: false` (default) and `pipelineMode: false` unless explicitly told otherwise — these are dev-only switches. * Ingress: TLS-only, cert-manager (or equivalent), restrict source ranges if your data plane is internet-exposed only for known IPs. ## Upgrade ```bash theme={null} helm upgrade popsink \ oci://ghcr.io/popsink/charts/data-plane \ -n popsink \ -f values.yaml \ --version ``` The data plane runs DB migrations on startup. Watch a pod's logs: ```bash theme={null} kubectl -n popsink logs -l app.kubernetes.io/component=data-plane -f ``` ## Uninstall ```bash theme={null} helm uninstall popsink -n popsink ``` By default the **PostgreSQL PVC and the broker's S3 data are kept** (`postgresql.primary.persistentVolumeClaimRetentionPolicy.whenDeleted=Retain`, S3 buckets are external). Delete them manually if you want a clean slate: ```bash theme={null} kubectl -n popsink delete pvc -l app.kubernetes.io/instance=popsink aws s3 rm s3://my-popsink-bucket/kafka/ --recursive # ⚠ destroys all topics ``` ## Troubleshooting | Symptom | Likely cause | | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `helm install` aborts on the Kora database message | `kora.database` not configured — see [Schema registry database](#schema-registry-database-kora) | | `kora` pod: `password authentication failed for user "kora"` | The `kora` role / database does not exist. Create them, or point Kora at the data-plane database | | `kotatsu.s3.bucket must be provided when kotatsu is enabled` | Kotatsu enabled with a non-S3 retention engine — set `kotatsu.enabled: false` | | `Kora must be enabled, or schemaRegistry.url must be provided` | `kora.enabled: false` with no external registry configured | | `data-plane` pods crash-loop on an empty `DB_USER` | `externalDatabase.user` / `.database` left empty | | Stream tab has no search bar | Kotatsu is disabled — expected, the tab falls back to a live Kafka read | | Deployment stays **Offline** / "Awaiting connections…" | Wrong `controlPlaneUrl` / `deploymentId` / `deploymentJwtToken`, or egress to the control plane is blocked | | `data-plane` pods crash-loop on `connector-config-encryption-key` | Fernet key is not a valid URL-safe base64-encoded 32-byte string | | Broker pods `CrashLoopBackOff` with S3 errors | `kodansu.storage.engine`, `region`, or IAM role/credentials are wrong; bucket missing | | `ImagePullBackOff` on every pod | `imagePullSecret.token` not set / not a valid GAR service-account JSON | | Login at `ingressUrl` fails with redirect loop | `ingressUrl` doesn't match the public URL fronted by your ingress | | Workers (connectors) never become `LIVE` | The data-plane pod cannot reach the broker or Kora inside the cluster — check `NetworkPolicy` | | Ingress 502 on `/api/livez` | Service name in your `Ingress` doesn't match `-data-plane` | For deeper logs: ```bash theme={null} kubectl -n popsink logs -l app.kubernetes.io/component=data-plane --tail=200 kubectl -n popsink logs -l app.kubernetes.io/component=tansu --tail=200 kubectl -n popsink logs -l app.kubernetes.io/name=kora --tail=200 kubectl -n popsink logs -l app.kubernetes.io/component=kotatsu --tail=200 ``` ## Reference — most useful values A full list lives in `values.yaml` (annotated with `@param` blocks). The most common knobs are: | Path | What it does | | ----------------------------------------------- | ----------------------------------------------------- | | `controlPlaneUrl` / `controlPlaneFrontendUrl` | Control plane API and UI URLs | | `deploymentId`, `deploymentJwtToken.token` | Identity vs. control plane | | `ingressUrl` | Public URL of this data plane | | `image.tag` | Pin the data-plane image version | | `replicaCount` | Data-plane API replicas (default 2) | | `resources` | Data-plane CPU / memory requests / limits | | `imagePullSecret.*` / `global.imagePullSecrets` | How to authenticate to the GAR registry | | `adminCredentials.*` | First admin login | | `jwt.secret` | Signs user session tokens | | `connectorConfigEncryptionKey.key` | Encrypts connector credentials at rest | | `kodansu.enabled`, `kodansu.storage.*` | In-cluster broker or BYO Kafka; object-store backend | | `kora.enabled` / `schemaRegistry.*` | In-cluster Schema Registry or BYO | | `kora.database.*` | **Required.** Kora's own PostgreSQL role and database | | `kotatsu.enabled` | Optional stream browser (default `false`) | | `postgresql.enabled` / `externalDatabase.*` | In-cluster Postgres or BYO | | `pdb.create` | PodDisruptionBudget (default `true`) | | `autoscaling.hpa.*`, `autoscaling.vpa.*` | HPA / VPA — disabled by default | ## Further reading Hardware, ingress and storage requirements for a Popsink-ready cluster. The control-plane wizard that issues your `deploymentId` and JWT token. How to map deployments onto your regions, networks and staging tiers.