Install the data plane on Kubernetes

This is the install guide for a self-hosted Popsink data plane — the runtime that moves data from your sources to your targets. You create a deployment in the Popsink control plane, it hands you a complete values.yaml and a helm install command, and you run them against your cluster. Every value the chart exposes, component by component, is in the Helm chart reference. This page covers the path that works; that one covers everything else.

What gets installed

Component	Purpose	Default
`data-plane`	Main API + UI. Talks to the control plane, launches connector workers as pods	enabled
`kodansu`	Stateless Kafka-compatible broker, object-store backed	enabled
`kora`	Confluent-compatible schema registry. Needs its own PostgreSQL database	enabled
`kotatsu`	Optional topic / schema browser, reads the broker’s S3 objects	disabled
`postgresql`	Bitnami sub-chart, in-cluster Postgres for the data-plane database	enabled

Connector workers are not part of the chart: the data plane creates them as pods in its own namespace at runtime. Any sub-component can be disabled in favour of your own Kafka, schema registry or PostgreSQL — see the chart reference.

Before you start

Run through Kubernetes requirements first. In short: Kubernetes 1.23+, Helm 3.8.0+, two amd64 nodes at 4 vCPU / 16 GB, an ingress controller, outbound HTTPS, and — from Popsink — a registry token and an account on the control plane you will attach this deployment to. Decide up front where the data plane will be reachable. You are asked for it in step 2 and it cannot be left blank:

URL, not TLD, not FQDN. For a data plane served at popsink.example-company.com, the TLD is .com, the FQDN is popsink.example-company.com, and the URL is https://popsink.example-company.com.The chart value is ingressUrl and it takes the URL. The wizard’s Fully qualified domain name field accepts the host on its own and prefixes https:// for you; type a scheme yourself and it is kept as-is (an on-prem data plane may legitimately sit behind plain HTTP).

1. Create the deployment

Open Deployments in the control-plane sidebar.
Click New Deployment.

Then fill in:

Deployment name — a recognizable name for this environment. It also becomes the Helm release name (slugified), so keep it short and DNS-friendly.
Deployment URL — generated from the name.
Deployment method — Self-hosted.

Click Configure Self-Hosted.

💡 You can request a new region or provider if your preferred option is not listed.

One deployment per isolated network. Environments split a single reachable network into scopes; they do not span VPCs or regions. If you are planning a fleet, read Deployments and environments before creating the first one.

2. Configure

Everything on this step exists to generate your values.yaml. Nothing here is stored by Popsink — you can also skip the step and write the values yourself.

Section	What it asks	What it generates
PostgreSQL Database	Off = use the in-cluster Postgres. On = your own instance	`postgresql.`, or `postgresql.enabled: false` + `externalDatabase.`
Retention	S3 Compat — the only choice that works	`kodansu.storage.engine` + `kodansu.storage.aws.`, `kotatsu.`
TLD	Fully qualified domain name, e.g. `popsink.example.com`	`ingressUrl: https://popsink.example.com`

The PostgreSQL Database toggle means “I bring my own”. Left off, the chart installs the bundled Postgres sub-chart and points both the data plane and the schema registry (Kora) at it.

Pick S3 Compat. The wizard still offers PostgreSQL Database as a retention choice and still generates a postgres:// storage engine for it, but the broker has been object-store only since Kodansu 0.7.0-beta.11 and now refuses that URL outright — the pods never start. The option is a leftover; treat the retention step as having one answer.

On Azure, pick S3 Compat here anyway and point the broker at ADLS Gen2 afterwards — the wizard emits no abfss:// engine, and the swap is three values in the generated file. See Broker storage on Azure Data Lake Storage Gen2 below.

Click Create Chart.

3. Save the generated `values.yaml`

The Chart step shows your values.yaml first, then the install command that reads it. That order matters: Helm resolves values at install time, so a helm install without -f values.yaml installs pure chart defaults and ignores everything the wizard generated. The generated file is complete — it covers every value the chart marks required, plus three secrets minted in your browser and never sent to Popsink:

values.yaml (generated — abridged)

controlPlaneUrl: https://control-plane-api.popsink.com/api
controlPlaneFrontendUrl: https://control-plane.popsink.com
ingressUrl: https://popsink.example-company.com
deploymentId: <uuid>
deploymentJwtToken:
  token: <jwt>

adminCredentials:
  username: admin
  password: <generated>
jwt:
  secret: <generated>
connectorConfigEncryptionKey:
  key: <generated Fernet key>

imagePullSecret:
  create: true
  registry: europe-west1-docker.pkg.dev/popsink-common-438615/onprem
  token: ""            # ← you paste the token Popsink issued you

kodansu:
  storage:
    engine: s3://my-popsink-bucket/
    aws:
      region: eu-west-1
      accessKeyId: <key>
      secretAccessKey: <secret>

kora:
  database:            # the schema registry shares the data-plane database
    host: my-deployment-postgresql
    port: 5432
    user: popsink
    database: popsink
    password: <generated>

postgresql:
  auth:
    password: <generated>

kotatsu:
  s3:
    bucket: my-popsink-bucket
    cluster: tansu
    region: eu-west-1
    forcePathStyle: false
    accessKey: <key>
    secretKey: <secret>

adminCredentials.password, jwt.secret and connectorConfigEncryptionKey.key are generated in the browser and stored nowhere else. Save this file in your secret manager before leaving the page.connectorConfigEncryptionKey.key must never be rotated: every stored connector configuration is encrypted with it and becomes unreadable if it changes. Each deployment gets its own — never share one across deployments.

deploymentJwtToken is an object, not a string — token: nested underneath. A flat deploymentJwtToken: <jwt> fails to render.

Fill in what the wizard cannot

The wizard flags any value it left empty under “Fill these in before installing”. Two of them always need your attention:

imagePullSecret.token

Paste the registry token Popsink issued you (the GAR service-account JSON, raw or base64-encoded). The control plane holds no registry credential, so it cannot fill this for you.

imagePullSecret:
  create: true
  registry: europe-west1-docker.pkg.dev/popsink-common-438615/onprem
  token: <gar-service-account-json>

global.imagePullSecrets

Add this by hand — imagePullSecret.create: true is not sufficient today. The chart creates the secret as <release-name>-regcred but does not attach it to any pod, so every image fails with ImagePullBackOff. Reference it explicitly:

global:
  imagePullSecrets:
    - my-deployment-regcred      # <release-name>-regcred

Or skip imagePullSecret entirely and create the secret yourself:

kubectl create secret docker-registry popsink-registry \
  --docker-server=europe-west1-docker.pkg.dev \
  --docker-username=_json_key \
  --docker-password="$(cat key.json)" \
  -n popsink

global:
  imagePullSecrets:
    - popsink-registry

This extra step is a known chart defect and will go away — until then, both values are required.

externalDatabase.user / .database (only if you brought your own Postgres)

These two are not guarded by the chart: left empty they render, install, and the data plane then crash-loops on an empty DB_USER. The database must exist before the install — the data plane runs its own migrations on startup.

Sensitive values can come from an existing Kubernetes Secret rather than a literal — set the matching existingSecret field. It works with Vault, Sealed Secrets or the External Secrets Operator, and is the recommended shape in production. See Managing secrets.

Broker storage on Azure Data Lake Storage Gen2

The Retention step only offers S3, but the broker also reads and writes ADLS Gen2 — abfss://, abfs:// and az:// — from Kodansu 1.0.0-alpha.17 onwards. The tag the chart pins by default is already past it, so there is no image bump to do; what is missing is the chart. It has no kodansu.storage.azure block, so this is a hand-edit of the generated values.yaml: pick S3 Compat in the wizard, then replace the storage engine and swap the credentials over. Three things are true of every ADLS install and none of them are optional:

the identity needs Storage Blob Data Contributor on the account or container. Owner is an ARM role and grants no data-plane access at all — it fails as a 403 AuthorizationFailure that never names the missing role, and it is the first thing that goes wrong;
kodansu.storage.aws.region is still required by the chart, on every path, and never read on this one. Leave any valid region string in it;
kotatsu.enabled must stay false. The stream browser reads S3 objects directly and has no ADLS client, so an ADLS engine either fails the render with kotatsu.s3.bucket must be provided… or points it at a bucket that does not exist.

kodansu:
  storage:
    engine: "abfss://popsink@mystorage.dfs.core.windows.net/kafka/"
    aws:
      region: us-east-1        # required by the chart, unread on this path

  # The azure-workload-identity webhook keys off this label and this annotation.
  podLabels:
    azure.workload.identity/use: "true"
  serviceAccount:
    annotations:
      azure.workload.identity/client-id: <managed-identity-client-id>

kotatsu:
  enabled: false

kodansu:
  storage:
    engine: "abfss://popsink@mystorage.dfs.core.windows.net/kafka/"
    aws:
      region: us-east-1
  # AZURE_STORAGE_ACCOUNT_NAME + AZURE_STORAGE_ACCOUNT_KEY
  extraEnvVarsSecret: kodansu-azure

kotatsu:
  enabled: false

kodansu:
  storage:
    engine: "abfss://popsink@mystorage.dfs.core.windows.net/kafka/"
    aws:
      region: us-east-1
  # AZURE_STORAGE_CLIENT_ID + AZURE_STORAGE_CLIENT_SECRET + AZURE_STORAGE_TENANT_ID
  extraEnvVarsSecret: kodansu-azure

kotatsu:
  enabled: false

Kodansu takes its Azure credentials from the environment and nowhere else, so the secret-backed paths go in through kodansu.extraEnvVarsSecret (or extraEnvVars / extraEnvVarsCM) rather than a storage value:

kubectl create secret generic kodansu-azure -n popsink \
  --from-literal=AZURE_STORAGE_ACCOUNT_NAME=mystorage \
  --from-literal=AZURE_STORAGE_ACCOUNT_KEY="$AZURE_KEY"

Prefer workload identity. It is the AKS analogue of IRSA on EKS and the one mechanism with nothing to rotate. The webhook injects AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_AUTHORITY_HOST and AZURE_FEDERATED_TOKEN_FILE into the broker pod, and the broker’s object-store client reads all four under exactly those names — no extraEnvVars needed. Set up the federated credential against the popsink namespace and the broker’s service account — still <release>-tansu, the chart has not renamed it — before installing; az identity federated-credential create is the Azure-side half.

The container must exist and be writable before the install — the broker creates objects, never the container. The account’s own settings are part of the contract too (versioning off, soft delete off, no immutability policy, hot tier), and a freshly created hierarchical-namespace account already satisfies all four: see Azure (ADLS Gen2) for that table, private endpoints, and the cost difference against S3.

4. Configure ingress

The chart can render the Ingress for you, and it can also stay out of the way. Pick one — but either way, ingressUrl must match the public URL your users actually reach, or the control-plane login redirect loops.

Bring your own Ingress (recommended)
Let the chart render it

Leave ingress.enabled: false (the default) and point your own resource at the data-plane service — <release>-data-plane, port 80:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: popsink
  namespace: popsink
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts: [popsink.example-company.com]
      secretName: popsink-tls
  rules:
    - host: popsink.example-company.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: <release>-data-plane
                port: { number: 80 }

This is the better option when ingress in your organisation needs specific annotations — WAF, allow-lists, a company cert issuer.

ingressUrl: https://popsink.example-company.com

ingress:
  enabled: true
  hostname: popsink.example-company.com
  ingressClassName: nginx
  tls: true
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod

ingress.extraHosts, ingress.extraTls, ingress.path and ingress.existingSecret are available too — see Ingress values.

5. Install

Run the command the wizard shows you, from the directory holding your values.yaml:

kubectl create namespace popsink

helm install <release-name> oci://ghcr.io/popsink/charts/data-plane \
  --version <version> \
  -n popsink \
  -f values.yaml

Do not copy a version from a document — including this one. The chart version is served by the control plane and tracks the published chart; a literal pinned in a guide rots.

Watch the rollout:

kubectl -n popsink get pods -w

6. Await connection

Once the pods are Ready, the control plane leaves Awaiting connections… and the deployment flips to Live, usually within a minute. That flip is driven by the data plane’s first heartbeat — the control plane never probes your cluster.

Then open your ingressUrl and log in with adminCredentials.

It stays on “Awaiting connections…”

The five things to check, and the logs to read next.

Production checklist

Versions & images

Pin the chart version (--version) and the application image (image.tag) — never deploy latest.
Upgrade one deployment at a time, starting with staging. See Upgrading a fleet.

State

Use external PostgreSQL (postgresql.enabled: false + externalDatabase.*) backed by managed snapshots, not the in-cluster Bitnami sub-chart.
Use external S3 with versioning and a lifecycle policy for the broker.

Resources & availability

Set explicit resources.requests/limits on every component — the defaults target small-to-medium clusters.
Keep replicaCount ≥ 2 on data-plane (default) and kodansu.replicaCount: 3 (default).
Keep pdb.create: true (default) so at least one pod survives a node drain.

Secrets

Provide values through existingSecret fields and a secrets controller rather than literals in values.yaml.
Back up the four secrets — admin credentials, JWT secret, Fernet key, database password — in your secret manager.

Safety switches

Keep allowDesignLogin: false and pipelineMode: false — dev-only switches.

Network

TLS-only ingress, with source ranges restricted if the data plane is internet-exposed.

After the install

Upgrades

helm upgrade, the broker-image caveat, and the order to roll a fleet in. Uninstalling is documented there too.

Tunnels and private connectivity

Reaching a source database that is not on the cluster’s network.

Installing without the wizard

The wizard only generates values — the chart is the same either way. Installing by hand means doing three things yourself:

Get a deploymentId and deploymentJwtToken from the control plane (Deployments → New deployment → Self-hosted). They are the deployment’s identity and cannot be minted locally.
Generate the four secrets yourself — see Secrets you must generate.
Write the values.yaml from the Helm chart reference, which documents every component the wizard configures for you, plus the ones it does not: BYO Kafka, an external schema registry, Oracle as the Kora backend, HPA/VPA, network policies.

Helm chart reference

Every value the chart exposes, component by component.

Troubleshooting

Symptom-to-cause table for the install and the first boot.

Deployments and environments

How to map deployments onto your regions, networks and staging tiers.

Control plane and data plane

What metadata crosses between the two, and the network rules it implies.

Welcome

Platform Overview

Security

Deployment

Source Connectors

Target Connectors

Integrations

API

Transformation

Changelog

Install the data plane on Kubernetes

What gets installed

Before you start

1. Create the deployment

2. Configure

3. Save the generated `values.yaml`

Fill in what the wizard cannot

Broker storage on Azure Data Lake Storage Gen2

4. Configure ingress

5. Install

6. Await connection

It stays on “Awaiting connections…”

Production checklist

After the install

Upgrades

Tunnels and private connectivity

Installing without the wizard

Further reading

Helm chart reference

Troubleshooting

Deployments and environments

Control plane and data plane

​What gets installed

​Before you start

​1. Create the deployment

​2. Configure

​3. Save the generated values.yaml

​Fill in what the wizard cannot

​Broker storage on Azure Data Lake Storage Gen2

​4. Configure ingress

​5. Install

​6. Await connection

It stays on “Awaiting connections…”

​Production checklist

​After the install

Upgrades

Tunnels and private connectivity

​Installing without the wizard

​Further reading

Helm chart reference

Troubleshooting

Deployments and environments

Control plane and data plane

What gets installed

Before you start

1. Create the deployment

2. Configure

3. Save the generated `values.yaml`

Fill in what the wizard cannot

Broker storage on Azure Data Lake Storage Gen2

4. Configure ingress

5. Install

6. Await connection

Production checklist

After the install

Installing without the wizard

Further reading