data-plane Helm chart. If you are installing for the
first time, follow
Install the data plane on Kubernetes instead — it walks
the one path that works. This page is what you reach for when the wizard’s
output is not enough: bringing your own Kafka, schema registry or PostgreSQL,
tuning resources, or managing secrets out of band.
The chart is published as an OCI artifact:
@param blocks) ships inside the
chart’s own values.yaml, and the control-plane wizard links the reference
values file for the version it serves you.
Components
Connector workers are not part of the chart: the data plane creates them as pods
in its own namespace at runtime.
kodansu was previously called tansu, and the chart still reads a tansu:
block for backwards compatibility (anything under kodansu: wins). New values
files should use kodansu:.Two components that appeared in earlier versions of this page — karapace and
kafka-ui — no longer exist in the chart. They were replaced by kora and
kotatsu respectively. A values.yaml that still sets karapace.* or
kafkaUi.* is silently ignored.Identity
These come from the control plane (Deployments → New deployment → Self-hosted) and must be used verbatim — they are how the data plane authenticates itself.deploymentMode: STANDALONE disables every control-plane call, including the
heartbeat, and requires no deploymentJwtToken. It is the mode behind
air-gapped installs — see
Standalone
for what you give up.
Secrets you must generate
Beyond the values handed by the control plane, four secrets are generated locally. The wizard mints three of them in your browser; installing by hand means producing them yourself.Managing secrets with existingSecret
Any sensitive value can come from an existing Kubernetes Secret instead of a
literal. Set the matching existingSecret field and leave the literal unset —
this works with Vault, Sealed Secrets and the External Secrets Operator, and is
the recommended shape in production.
Available for deploymentJwtToken, jwt, connectorConfigEncryptionKey,
adminCredentials, kora.database, kodansu.storage.aws, externalDatabase
and postgresql.auth.
The sources are mutually exclusive: an existingSecret combined with a literal
password or url is rejected at render time.
Image pull
All images are published to a private GAR registry:- Chart-managed pull secret
- Bring your own pull secret
<release-name>-regcred and attaches it to
every pod, including the connector workers’ service account and the
kora / postgresql sub-charts. Nothing else to set.Charts older than the fix for Popsink/data-plane#3511 (August 2026) did not
attach the secret, and needed
global.imagePullSecrets: [<release-name>-regcred]
as well. Listing it is harmless on current charts.Data-plane database
- In-cluster (default)
- External (recommended in production)
Schema registry (Kora)
Kora is Popsink’s schema registry: every connector registers the structure of what it reads there, and every target resolves schemas from it. It is enabled by default and it requires its own PostgreSQL role and database, which the chart does not create for you.- Dedicated database (recommended)
- Bring your own registry
Run once against your PostgreSQL instance — managed services such as RDS or
Cloud SQL included:
kora.database.existingSecret with secretKeys.password, or secretKeys.url
for a full connection URL.
Kora also supports an Oracle backend: kora.database.backend: oracle, port
1521, and database set to the service name (e.g. FREEPDB1). The chart
selects the matching image automatically.
Broker storage (Kodansu)
Kodansu is a stateless Kafka broker that stores log segments in an object store.kodansu.storage.engine decides where records live; its URL scheme picks the
backend. Both it and kodansu.storage.aws.region are required by the chart —
the region on every path, including the ones that never read it.
What every backend must provide
The broker keeps no state of its own: offset assignment, consumer-group membership and maintenance leases are all conditional writes to the object store. A backend is only usable if it honours both:
A store that ignores one of these headers does not fail: it overwrites. The
broker starts, produces and consumes, and two writers that should have been
serialized both “win” — duplicated offsets or a consumer group that loses its
committed positions. Check the store before
the install.
The object layout is create-only and retention is implemented by delete, so
the bucket also needs:
Support matrix
The path in the URL is ignored on every backend: the bucket is the URL host and
the broker lays out its own prefixes. Use a dedicated bucket per deployment.
Kotatsu reads S3 objects directly and has no GCS or Azure client: keep
kotatsu.enabled: false on those two.
Check an S3-compatible store
Run this against the target bucket, with the credentials the broker will use, before installing, to confirm the store honours both conditional writes. It needs AWS CLI 2.22 or later (the version that added--if-match). Leave EP empty for Amazon S3.
412
(NotImplemented, 400) means the header is refused, which also rules it out.
Amazon S3
s3:GetObject, s3:PutObject, s3:DeleteObject and
s3:ListBucket on the bucket. region must be the bucket’s region. A gateway
VPC endpoint for S3 keeps the traffic off the NAT gateway, which on this access
pattern is a noticeable part of the bill.
MinIO and RustFS
Self-hosted S3 servers, and the usual choice for on-premises and air-gapped installs.endpoint set, requests are path-style
(https://<endpoint>/<bucket>/<key>), which both servers accept, and Kotatsu
derives forcePathStyle: true on its own. If the endpoint’s certificate is
signed by an internal CA, the broker image must trust it — mount the CA and
point SSL_CERT_FILE at it through kodansu.extraVolumes,
kodansu.extraVolumeMounts and kodansu.extraEnvVars.
Keep bucket versioning off, as it is by default on both.
Oracle Cloud Object Storage
Through Oracle’s Amazon S3 Compatibility API:- Credentials are a Customer Secret Key (access key + secret), created on the user in the OCI console — not an API signing key.
<namespace>is the tenancy’s Object Storage namespace (oci os ns get). The bucket must live in the compartment designated for the S3 Compatibility API, which is the root compartment unless changed.- Use the OCI region identifier. If it is not the tenancy’s home region and the client cannot pass it, Oracle only serves the home region.
IBM Cloud Object Storage
- Credentials are HMAC keys: create a service credential with
Include HMAC Credential enabled, and use its
cos_hmac_keyspair. IAM API keys are not read by the broker. - Pick the endpoint for where the cluster runs:
s3.private.…from IBM Cloud classic,s3.direct.…from a VPC,s3.…over the public network. - The same configuration applies to an on-premises IBM COS appliance — set
endpointto its accesser address.
Google Cloud Storage
Native client, not GCS’s S3 interoperability endpoint:- The identity needs
roles/storage.objectAdminon the bucket.objectCreatorandobjectViewerare not enough: retention deletes and reads list. - Set the bucket’s soft-delete retention to
0. Google enables soft delete on every bucket at 7 days and bills it; on a layout that deletes what compaction has just merged, that pays for the churn twice for a week. Also keep versioning, retention policies, Autoclass and lifecycle class transitions off, and prefer uniform bucket-level access. - Requester-pays buckets do not work.
- GCS caps writes to one object name at about one per second. The data path never rewrites an object; only consumer-group formation does, and it is batched — a large group still takes a few seconds to form.
Azure (ADLS Gen2)
Supported from Kodansu1.0.0-alpha.17, experimental:
Accepted URL forms
wasbs://, adl://, azure:// and a raw
https://<account>.blob.core.windows.net/ are refused, even though the
underlying object-store parser understands them: every extra alias is another
spelling a deployment can drift onto, and wasbs:// is the legacy Blob scheme —
it should fail loudly rather than be treated quietly as Gen2.
A dfs host is resolved to <account>.blob.core.windows.net: the broker speaks
the Blob endpoint against an ADLS Gen2 account, never the DFS one. Every
kodansu.storage.tuning knob is scheme-independent and applies unchanged.
Credentials
The broker reads them from the environment only, via the object-store client’s own resolution. The chart has nokodansu.storage.azure block, so anything other
than workload identity goes in through kodansu.extraEnvVarsSecret,
extraEnvVarsCM or extraEnvVars.
Whichever you use, the identity needs
Storage Blob Data Contributor on the
account or container. Owner is an ARM role and grants no data-plane access
whatsoever; it fails as a 403 AuthorizationFailure that names no role.
Account settings are part of the contract
The object layout is create-only and immutable, and retention is implemented by delete. Four account settings therefore are not preferences:
A freshly created
StorageV2 account with nothing but
--enable-hierarchical-namespace true --access-tier Hot already satisfies all
four. Read the table as “do not turn these on” rather than “remember to turn
these off”: the deployment that gets it wrong is the one that hardened an
existing data-lake account. None of the four is detectable from the data plane,
so there is no startup warning for any of them.
Hierarchical namespace is the target. Deleting every blob under a prefix leaves
the directories behind — not billed for capacity, absent from the listings the
broker issues, untidy and nothing more. Flat Blob Storage is untested.
Private endpoints
Azure’s guidance for an ADLS Gen2 account is a private endpoint for both theblob and dfs sub-resources, because managing ACLs and creating or deleting
directories need the DFS one. The broker does none of those three, so blob
alone is sufficient.
The failure worth knowing is the other direction: a network team that provisioned
dfs only — the usual default for a data-lake account — leaves the broker
unable to connect, with nothing in the error naming the sub-resource. If
abfss:// times out or is refused while the account resolves, check this first.
Cost and Kotatsu
At the shape of a measured production S3 fleet, ADLS Gen2 comes to roughly 1.11× the S3 request bill, almost entirely the ~30 % hierarchical-namespace write premium. Two differences worth knowing: a list is billed as a read here (≈9× cheaper than on S3, whereLIST is priced with PUT), and read/write
transactions are metered per 4 MiB — so every 16 MiB merged segment write bills
four times. Deletes are free, exactly as on S3.
Kotatsu has no ADLS client: it reads S3 objects directly. Keep
kotatsu.enabled: false on this path.
Bring your own Kafka
Stream browser (Kotatsu)
Kotatsu is optional and disabled by default. It is a read-only browser over the objects the broker wrote to S3, and it backs two things in the UI:- search, pagination and the timestamp column on the datamodel Stream tab;
- the per-subscription target lag rows on the pipeline latency view.
s3:// engine — it has no ADLS client. To enable it:
kodansu.storage.*,
kodansu.clusterId) — the only combination that can work, since the two read and
write the same objects. Set kotatsu.s3.* only to point Kotatsu at a replica of
the bucket or at a separate read-only credential. On EKS with IRSA, leave the
keys empty and annotate kotatsu.serviceAccount.annotations with the role ARN.
On charts before
0.1.0-alpha.162 Kotatsu defaulted to enabled, so an
install that had not opted into S3 aborted with
kotatsu.s3.bucket must be provided when kotatsu is enabled. Setting
kotatsu.enabled: false is the correct workaround on those versions.Ingress
ingress.enabled is off by default and we recommend leaving it that way:
ingress in production usually needs company-specific annotations (cert-manager,
WAF, allow-lists). Both shapes — chart-rendered and bring-your-own — are covered
in step 4 of the install guide.
The chart never installs an ingress controller. Bringing your own means pointing
it at the
<release>-data-plane service on port 80.
Resources and availability
Safety switches
A complete values.yaml
values.yaml
Most useful values
Further reading
Install the data plane on Kubernetes
The guided path, from the control-plane wizard to a Live deployment.
Troubleshooting
Symptom-to-cause table for the install and the first boot.
Kubernetes requirements
What the cluster, the network and your Popsink account must provide first.
Deployments and environments
How to map deployments onto your regions, networks and staging tiers.