Install Popsink with Helm for partners

This page is intended for integrators who install Popsink in their own Kubernetes cluster. The chart deploys a self-hosted Popsink data plane — the runtime that moves data from your sources to your targets. It connects to a control plane (managed by Popsink at control-plane.popsink.com, or your own) which orchestrates and monitors the deployment.

Architecture overview

The chart deploys the following components:

Component	Purpose	Default
`data-plane`	Main API + UI. Talks to the control plane, launches connector workers as pods	enabled
`kodansu`	Stateless Kafka-compatible broker, object-store backed	enabled
`kora`	Confluent-compatible Schema Registry. Needs its own PostgreSQL — see below	enabled
`kotatsu`	Optional topic / schema browser, reads the broker’s S3 objects	disabled
`postgresql`	Bitnami sub-chart, in-cluster Postgres for the data-plane DB	enabled

Connector workers are not part of the chart: the data-plane creates them as pods in its own namespace at runtime. You can disable any sub-component and bring your own:

External Kafka — kodansu.enabled=false, configure defaultKafka.*
External Schema Registry — kora.enabled=false, configure schemaRegistry.*
External PostgreSQL — postgresql.enabled=false, configure externalDatabase.*

kodansu was previously called tansu, and the chart still reads a tansu: block for backwards compatibility (anything under kodansu: wins). New values files should use kodansu:.Two components that appeared in earlier versions of this page — karapace and kafka-ui — no longer exist in the chart. They were replaced by kora and kotatsu respectively. A values.yaml that still sets karapace.* or kafkaUi.* is silently ignored.

The chart can render an Ingress (ingress.enabled, off by default), but it never installs an ingress controller: you still need your own (Traefik, NGINX, Istio…) and a DNS record pointing at it. Both options — chart-rendered or bring-your-own — are covered under Ingress below.

Prerequisites

Cluster

Requirement	Minimum
Kubernetes	`1.23+`
Helm	`3.8.0+`
Worker nodes	2 nodes, `amd64`
Per-node	4 vCPU, 16 GB RAM
Persistent volume	SSD-backed StorageClass, ≥ 200 GB available
Ingress	Traefik / NGINX / Istio (or any IngressClass-compatible)
Object storage	S3-compatible bucket (recommended for the broker)

Access to Popsink images

All images are published to a private GAR registry:

europe-west1-docker.pkg.dev/popsink-common-438615/onprem

Popsink will provide you with a service-account JSON token to pull these images.

Chart-managed pull secret
Bring your own pull secret

Let the chart create the imagePullSecret for you:

imagePullSecret:
  create: true
  registry: europe-west1-docker.pkg.dev/popsink-common-438615/onprem
  token: |
    <base64-encoded service account JSON, or the JSON itself>

Create the secret yourself, then reference it via global.imagePullSecrets:

kubectl create secret docker-registry popsink-registry \
  --docker-server=europe-west1-docker.pkg.dev \
  --docker-username=_json_key \
  --docker-password="$(cat key.json)" \
  -n popsink

global:
  imagePullSecrets:
    - popsink-registry

Onboarding flow (control plane)

Before installing the chart, you need a deployment ID and a deployment JWT token issued by the control plane.

Open the control plane

Log into https://control-plane.popsink.com (or your own control plane).

Create a self-hosted deployment

Open Deployments → New deployment and pick Self-hosted.

Fill the wizard

Optional pre-fill of PostgreSQL credentials (not stored).
Retention strategy — pick S3-compatible storage (recommended) or PostgreSQL.
FQDN — the public URL where the data plane will be reachable (e.g. https://popsink.your-company.com).

Copy the values snippet

The wizard outputs a values-control-plane.yaml snippet containing controlPlaneUrl, deploymentId and deploymentJwtToken. Use these values verbatim — they identify and authenticate your data plane against the control plane.

Wait for connection

Keep the wizard open. After helm install, the control plane will switch from Awaiting connection… to Connected once the data plane registers.

Required secrets to generate yourself

Beyond the values handed by the control plane, you must generate four secrets locally:

Secret	Format	How to generate
`adminCredentials.username` / `password`	any	Pick a strong password
`jwt.secret`	random string ≥ 32 chars	`openssl rand -base64 48`
`connectorConfigEncryptionKey.key`	URL-safe base64-encoded 32-byte Fernet key	`python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"`
`postgresql.auth.password` (or `externalDatabase.password`)	any	`openssl rand -base64 24`

These four secrets encrypt connector credentials, sign user sessions and protect the admin login. Lose them and you lose access to your stored connector configs. Back them up in your secret manager.connectorConfigEncryptionKey.key must never be rotated: every stored connector configuration is encrypted with it and becomes unreadable if it changes. Each deployment gets its own — never share one across deployments.

If you prefer to manage secrets out of band (Vault, Sealed Secrets, External Secrets Operator…), set the corresponding existingSecret field instead.

Schema registry database (Kora)

Kora is Popsink’s schema registry: every connector registers the structure of what it reads there, and every target resolves schemas from it. It is enabled by default and it requires its own PostgreSQL role and database, which the chart does not create for you.

This is the step most first installs miss. The bundled PostgreSQL provisions the data-plane’s own role and database (postgresql.auth.*) and nothing else, so leaving kora.database unset points Kora at a role that exists nowhere. From chart 0.1.0-alpha.162 the install aborts with the instructions below; on earlier charts it installs a pod that crash-loops on:

failed to connect to database: Backend("error returned from database:
password authentication failed for user \"kora\"")

Dedicated database (recommended)
Share the data-plane database
Bring your own registry

Run once against your PostgreSQL instance — managed services such as RDS or Cloud SQL included:

CREATE ROLE kora LOGIN PASSWORD 'a-strong-password';
CREATE DATABASE kora OWNER kora;

kora:
  database:
    host: "popsink.abc123.eu-west-2.rds.amazonaws.com"   # bundled: "<release>-postgresql"
    port: 5432
    user: "kora"
    database: "kora"
    password: "a-strong-password"

No SQL to run — point Kora at the same role and database as the data-plane. This keeps a self-hosted install to a single database to operate, and is what the control-plane wizard generates.

kora:
  database:
    host: "<same as externalDatabase.host, or <release>-postgresql>"
    port: 5432
    user: "popsink"      # same as externalDatabase.user / postgresql.auth.username
    database: "popsink"  # same as externalDatabase.database / postgresql.auth.database
    password: "<same password>"

If you already operate a Confluent-compatible schema registry:

kora:
  enabled: false

schemaRegistry:
  url: "https://schema-registry.example.com"
  username: "popsink"
  password: "<secret>"
  # or: existingSecret + secretKeys.{username,password}

Credentials can also come from a secret rather than a literal — use kora.database.existingSecret with secretKeys.password, or secretKeys.url for a full connection URL. The three sources are mutually exclusive: an existingSecret combined with a literal password or url is rejected at render time. Kora also supports an Oracle backend: kora.database.backend: oracle, port 1521, and database set to the service name (e.g. FREEPDB1). The chart selects the matching image automatically.

Stream browser (Kotatsu)

Kotatsu is optional and disabled by default. It is a read-only browser over the objects the broker wrote to S3, and it backs two things in the UI:

search, pagination and the timestamp column on the datamodel Stream tab;
the per-subscription target lag rows on the pipeline latency view.

Without it, the Stream tab falls back to a live Kafka read (no search, no pagination) and the target-lag rows are simply absent. Pipelines, connectors, subscriptions and delivery are unaffected — it is safe to leave disabled in production. Because it reads the bucket the broker writes, it only means anything with S3 retention. To enable it:

kotatsu:
  enabled: true

That is the entire configuration. Bucket, cluster id, region, endpoint and credentials all default to the broker’s own (kodansu.storage.*, kodansu.clusterId) — the only combination that can work, since the two read and write the same objects. Set kotatsu.s3.* only to point Kotatsu at a replica of the bucket or at a separate read-only credential. On EKS with IRSA, leave the keys empty and annotate kotatsu.serviceAccount.annotations with the role ARN.

On charts before 0.1.0-alpha.162 Kotatsu defaulted to enabled, so an install that had not opted into S3 aborted with kotatsu.s3.bucket must be provided when kotatsu is enabled. Setting kotatsu.enabled: false is the correct workaround on those versions.

Broker storage (S3)

Kodansu is a stateless Kafka broker that stores log segments in an object store. kodansu.storage.engine decides where records live, and both it and kodansu.storage.aws.region are required by the chart.

kodansu:
  storage:
    engine: "s3://my-popsink-bucket/kafka/"
    aws:
      region: eu-west-1
      irsaRoleArn:    arn:aws:iam::123456789012:role/popsink-kodansu
      assumedRoleArn: arn:aws:iam::123456789012:role/popsink-kodansu

kodansu:
  storage:
    engine: "s3://my-popsink-bucket/kafka/"
    aws:
      region: eu-west-1
      accessKeyId: AKIA…
      secretAccessKey: <secret>

kodansu:
  storage:
    engine: "s3://my-popsink-bucket/kafka/"
    aws:
      region: us-east-1
      endpoint: http://minio.minio.svc.cluster.local:9000
      allowHttp: true
      accessKeyId: minio
      secretAccessKey: minio123

kodansu:
  storage:
    engine: "postgres://user:password@db.example.com:5432/database"
    aws:
      # Required by the chart even here, and never read on this path.
      region: us-east-1

The bucket must exist, be writable by the configured identity, and ideally have object versioning enabled.

Before bumping the broker image on an existing bucket, read the upgrade notes on tansu.image.tag in the chart’s values.yaml. Two of the tags cross an object-layout boundary and need a quiesce-and-flip rather than a rolling upgrade.

PostgreSQL retention is simpler to operate but does not scale the same way, and rules out Kotatsu (there is no bucket to browse).

If you don’t want the in-cluster broker, set kodansu.enabled=false and kora.enabled=false, then point the data plane at your existing Kafka:

kodansu: { enabled: false }
kora:    { enabled: false }

defaultKafka:
  bootstrapServer: kafka.example.com:9093
  securityProtocol: SASL_SSL
  saslMechanism: SCRAM-SHA-512
  saslUsername: popsink
  saslPassword: <secret>
  caCert: |-
    -----BEGIN CERTIFICATE-----
    …
  cert: ""
  key:  ""

schemaRegistry:
  url: https://schema-registry.example.com
  username: popsink
  password: <secret>

Ingress

The chart’s built-in ingress.enabled flag is off by default and we recommend leaving it that way: ingress in production usually needs company-specific annotations (cert-manager, WAF, allow-lists). Provide your own Ingress resource that exposes:

the data-plane HTTP service (<release>-data-plane, port 80),
on the FQDN you declared in the control plane (ingressUrl).

Example with NGINX + cert-manager:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: popsink
  namespace: popsink
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts: [popsink.your-company.com]
      secretName: popsink-tls
  rules:
    - host: popsink.your-company.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: <release>-data-plane
                port: { number: 80 }

If you want the chart to render an Ingress for you, set ingress.enabled=true and configure ingress.hostname, ingress.ingressClassName, ingress.tls.

The value of ingressUrl in the chart must match the public URL the control plane redirects users to (it is also used for OAuth-style callbacks):

ingressUrl: "https://popsink.your-company.com"

Putting it together — minimal `values.yaml`

values.yaml

# ───── Identity (from the control-plane wizard) ─────
controlPlaneUrl:         https://control-plane-api.popsink.com/api
controlPlaneFrontendUrl: https://control-plane.popsink.com
deploymentMode: SELF_HOSTED
deploymentId:   <uuid-from-wizard>
deploymentJwtToken:
  token: <jwt-from-wizard>

# ───── Public URL of this data plane ─────
ingressUrl: https://popsink.your-company.com

# ───── Image pull ─────
imagePullSecret:
  create: true
  token: <gar-service-account-json>

# ───── Secrets you generated ─────
adminCredentials:
  username: admin
  password: <strong-password>
jwt:
  secret: <openssl rand -base64 48>
connectorConfigEncryptionKey:
  key: <fernet-key>

# ───── In-cluster Postgres (default) ─────
postgresql:
  enabled: true
  auth:
    password: <strong-password>
  primary:
    persistence:
      size: 50Gi
      storageClass: <your-ssd-class>

# ───── Schema registry database — NOT created by the chart ─────
# Run once:  CREATE ROLE kora LOGIN PASSWORD '<kora-password>';
#            CREATE DATABASE kora OWNER kora;
kora:
  database:
    host: <release>-postgresql        # or your RDS / Cloud SQL endpoint
    port: 5432
    user: kora
    database: kora
    password: <kora-password>

# ───── Broker + S3 ─────
kodansu:
  storage:
    engine: "s3://my-popsink-bucket/kafka/"
    aws:
      region: eu-west-1
      irsaRoleArn:    arn:aws:iam::123:role/popsink-kodansu
      assumedRoleArn: arn:aws:iam::123:role/popsink-kodansu

# ───── Optional stream browser (S3 retention only) ─────
kotatsu:
  enabled: true

For production, swap the in-cluster Postgres for a managed one:

postgresql:
  enabled: false

externalDatabase:
  host: popsink.abc123.eu-west-2.rds.amazonaws.com
  port: 5432
  user: popsink
  database: popsink
  password: <strong-password>

externalDatabase.user and externalDatabase.database are not guarded by the chart: left empty they render, install, and the data plane then crash-loops on an empty DB_USER. The database must exist before the install — the data plane runs its own migrations on startup.

Install

# 1. Create the namespace
kubectl create namespace popsink

# 2. Install (chart is published as an OCI artifact)
helm install popsink \
  oci://ghcr.io/popsink/charts/data-plane \
  -n popsink \
  -f values.yaml \
  --version <chart-version>

Use helm search repo or check the chart’s Chart.yaml (version:) to pin the chart version. Pinning avoids surprise upgrades.

Watch the rollout:

kubectl -n popsink get pods -w

When all pods are Ready, head back to the control-plane deployment page — the status should flip from Awaiting connection… to Connected within a minute. Open ingressUrl and log in with the adminCredentials you set.

Production checklist

Versions & images

Pin the chart version (--version) and the application image (image.tag) — never deploy latest.

State

Use external PostgreSQL (postgresql.enabled=false, externalDatabase.*) backed by managed snapshots, not the in-cluster Bitnami chart.
Use external S3 with versioning + lifecycle policy for the broker.

Resources & availability

Set explicit resources.requests/limits on every component (defaults target small-medium clusters; tune for your workload).
Configure replicaCount ≥ 2 on data-plane (default), keep tansu.replicaCount=3 (default).
Enable pdb.create: true (default) — at least 1 pod stays during node drains.

Secrets

Provide imagePullSecrets via a sealed secret / external secrets controller, not plain values.
Back up the four secrets (admin credentials, JWT secret, Fernet key, DB password) in your secret manager.

Safety switches

Keep allowDesignLogin: false (default) and pipelineMode: false unless explicitly told otherwise — these are dev-only switches.

Network

Ingress: TLS-only, cert-manager (or equivalent), restrict source ranges if your data plane is internet-exposed only for known IPs.

Upgrade

helm upgrade popsink \
  oci://ghcr.io/popsink/charts/data-plane \
  -n popsink \
  -f values.yaml \
  --version <new-chart-version>

The data plane runs DB migrations on startup. Watch a pod’s logs:

kubectl -n popsink logs -l app.kubernetes.io/component=data-plane -f

Uninstall

helm uninstall popsink -n popsink

By default the PostgreSQL PVC and the broker’s S3 data are kept (postgresql.primary.persistentVolumeClaimRetentionPolicy.whenDeleted=Retain, S3 buckets are external). Delete them manually if you want a clean slate:

kubectl -n popsink delete pvc -l app.kubernetes.io/instance=popsink
aws s3 rm s3://my-popsink-bucket/kafka/ --recursive   # ⚠ destroys all topics

Troubleshooting

Symptom	Likely cause
`helm install` aborts on the Kora database message	`kora.database` not configured — see Schema registry database
`kora` pod: `password authentication failed for user "kora"`	The `kora` role / database does not exist. Create them, or point Kora at the data-plane database
`kotatsu.s3.bucket must be provided when kotatsu is enabled`	Kotatsu enabled with a non-S3 retention engine — set `kotatsu.enabled: false`
`Kora must be enabled, or schemaRegistry.url must be provided`	`kora.enabled: false` with no external registry configured
`data-plane` pods crash-loop on an empty `DB_USER`	`externalDatabase.user` / `.database` left empty
Stream tab has no search bar	Kotatsu is disabled — expected, the tab falls back to a live Kafka read
Control plane stays on “Awaiting connection…”	Wrong `controlPlaneUrl` / `deploymentId` / `deploymentJwtToken`, or egress to the control plane is blocked
`data-plane` pods crash-loop on `connector-config-encryption-key`	Fernet key is not a valid URL-safe base64-encoded 32-byte string
Broker pods `CrashLoopBackOff` with S3 errors	`kodansu.storage.engine`, `region`, or IAM role/credentials are wrong; bucket missing
`ImagePullBackOff` on every pod	`imagePullSecret.token` not set / not a valid GAR service-account JSON
Login at `ingressUrl` fails with redirect loop	`ingressUrl` doesn’t match the public URL fronted by your ingress
Workers (connectors) never become `LIVE`	The data-plane pod cannot reach the broker or Kora inside the cluster — check `NetworkPolicy`
Ingress 502 on `/api/livez`	Service name in your `Ingress` doesn’t match `<release>-data-plane`

For deeper logs:

kubectl -n popsink logs -l app.kubernetes.io/component=data-plane --tail=200
kubectl -n popsink logs -l app.kubernetes.io/component=tansu       --tail=200
kubectl -n popsink logs -l app.kubernetes.io/name=kora             --tail=200
kubectl -n popsink logs -l app.kubernetes.io/component=kotatsu     --tail=200

Reference — most useful values

A full list lives in values.yaml (annotated with @param blocks). The most common knobs are:

Path	What it does
`controlPlaneUrl` / `controlPlaneFrontendUrl`	Control plane API and UI URLs
`deploymentId`, `deploymentJwtToken.token`	Identity vs. control plane
`ingressUrl`	Public URL of this data plane
`image.tag`	Pin the data-plane image version
`replicaCount`	Data-plane API replicas (default 2)
`resources`	Data-plane CPU / memory requests / limits
`imagePullSecret.*` / `global.imagePullSecrets`	How to authenticate to the GAR registry
`adminCredentials.*`	First admin login
`jwt.secret`	Signs user session tokens
`connectorConfigEncryptionKey.key`	Encrypts connector credentials at rest
`kodansu.enabled`, `kodansu.storage.*`	In-cluster broker or BYO Kafka; object-store backend
`kora.enabled` / `schemaRegistry.*`	In-cluster Schema Registry or BYO
`kora.database.*`	Required. Kora’s own PostgreSQL role and database
`kotatsu.enabled`	Optional stream browser (default `false`)
`postgresql.enabled` / `externalDatabase.*`	In-cluster Postgres or BYO
`pdb.create`	PodDisruptionBudget (default `true`)
`autoscaling.hpa.`, `autoscaling.vpa.`	HPA / VPA — disabled by default

Installation prerequisites

Hardware, ingress and storage requirements for a Popsink-ready cluster.

Self-hosted onboarding

The control-plane wizard that issues your deploymentId and JWT token.

Deployments and environments

How to map deployments onto your regions, networks and staging tiers.

Welcome

Platform Overview

Deployment

Source Connectors

Target Connectors

On-Premises & Integrations

Transformation

Changelog

Install Popsink with Helm for partners

Architecture overview

Prerequisites

Cluster

Access to Popsink images

Onboarding flow (control plane)

Required secrets to generate yourself

Schema registry database (Kora)

Stream browser (Kotatsu)

Broker storage (S3)

Ingress

Putting it together — minimal `values.yaml`

Install

Production checklist

Upgrade

Uninstall

Troubleshooting

Reference — most useful values

Further reading

Installation prerequisites

Self-hosted onboarding

Deployments and environments

​Architecture overview

​Prerequisites

​Cluster

​Access to Popsink images

​Onboarding flow (control plane)

​Required secrets to generate yourself

​Schema registry database (Kora)

​Stream browser (Kotatsu)

​Broker storage (S3)

​Ingress

​Putting it together — minimal values.yaml

​Install

​Production checklist

​Upgrade

​Uninstall

​Troubleshooting

​Reference — most useful values

​Further reading

Installation prerequisites

Self-hosted onboarding

Deployments and environments

Architecture overview

Prerequisites

Cluster

Access to Popsink images

Onboarding flow (control plane)

Required secrets to generate yourself

Schema registry database (Kora)

Stream browser (Kotatsu)

Broker storage (S3)

Ingress

Putting it together — minimal `values.yaml`

Install

Production checklist

Upgrade

Uninstall

Troubleshooting

Reference — most useful values

Further reading