# Popsink Alerting with Source and Target Connectors
Source: https://docs.popsink.com/alerting



Popsink provides native observability through a custom Source Connector that streams the runtime activity of other Popsink connectors. This allows you to monitor connector health, failures, retries, and throughput in real time.

Using Popsink pipelines, you can apply alerting rules to this activity stream and route critical events to external systems via Target Connectors (Slack, Database, Webhooks, etc.).

This creates a fully integrated alerting pipeline without requiring third-party monitoring systems.

## Architecture

1. Popsink Source Connector (Internal Activity Source)

   * Captures events about Popsink connector activity:
     * Connector lifecycle events (started, stopped, failed)
     * Task health (heartbeat, lag, retries, errors)
   * Exposes them as a continuous event stream.

2. Target Connectors

   * Forward alert events to external systems.

## Use Cases

* Detect if a connector crashes or stops unexpectedly.
* Trigger alerts when error rates exceed thresholds.


# v2026.01.22 22/01/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-01-22



***

**Description**: First control-plane release inside the data-plane monorepo. Brings billing, usage, deployment forms, multi-tenant primitives and a SQLite → PostgreSQL migration.

### Features

* Merge the control-plane codebase into the data-plane monorepo (#1135).
* Tidy up CI following the merge (#1166).
* Add the control-plane preprod environment (#1292).
* Wire data-plane ↔ control-plane communication (#1280).
* Add the billing details page (#1252).
* Add the Usage tab (#1254).
* Add the self-hosted deployment form (#1266).
* Add the Zendesk resources page (#1250).
* Migrate the data store from SQLite to PostgreSQL — drop Atlas, switch to Alembic, add a one-shot migration script (#1102).
* Mass dependency bump (#1177).
* Rename `popsink` → `data-plane` across the repo, namespace, image and chart (#1167).

### Fixes

* Validate inputs everywhere and improve error messages (#1160).
* Reject duplicate names (#1158).
* Fix control-plane e2e tests (#1224).
* Fix CVE #1268 (#1279).
* CVE batch (#1322).
* Force `h3@1.15.5` and `devalue@5.6.2` via Yarn resolutions (#1263).
* Fix CVE 27/01/2025 (#1338).
* Use `popsink.com` as the email domain in e2e tests (#1326).
* Fix exposed credentials in code (#1291).
* Switch to the `onprem-internal/control-plane` image registry (#1349).
* Add a generic page-code component (#1255).


# v2026.02.03 03/02/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-02-03



***

**Description**: User management and Hyperline as the billing data provider.

### Changes

* User management UI and APIs (#1315).
* Use Hyperline as the billing data provider on control-plane (#1290).
* Bump `tar` (#1364).
* Deploy bump in preprod (#1377).


# v2026.02.03.1 03/02/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-02-03-1



***

**Description**: Config sync between control-plane and data-plane.

### Changes

* Add config sync from control-plane to data-plane (#1314).
* Deploy bump in preprod (#1379).


# v2026.02.06 06/02/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-02-06



***

**Description**: Self-hosted auth fix; auto team creation; region constants refactor.

### Changes

* Fix auto team creation (#1403).
* Fix self-hosted auth (#1401).
* Refactor region constants and improve type consistency in deployments (#1392).
* Add `@isaacs/brace-expansion` dependency across control-plane and data-plane (#1387).


# v2026.02.09 09/02/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-02-09



***

**Description**: WebSocket deployment status; production kustomize overlay; heartbeat refactor.

### Features

* Implement WebSocket support for deployment-status updates (#1423).
* Self-hosted deployments are now supported in the organization-creation flow (#1424).
* Add a production kustomization overlay (#1427).
* Refactor `ReceiveDeploymentHeartbeatCommand` to include the environment repository (#1429).
* kafka-connect healthcheck (#1422).

### Tests

* Set up the new modular e2e architecture for control-plane tests (#1407).


# v2026.02.10 10/02/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-02-10



***

**Description**: Auth flow fixes; internal ingress routes; WebSocket URL hardening.

### Changes

* Fix authentication flow (#1450).
* E2E auth module test cases (#1451).
* Refactor `useDeploymentWebSocket` URL construction (#1448, #1441).
* Work on internal ingress routes (#1446).
* Add middleware to the control-plane `IngressRoute` (#1440).
* Misc fixes (#1443).


# v2026.02.25 25/02/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-02-25



***

**Description**: Large release: membership-scoped queries, environment management, e2e test cases, Postgres source, and Nuxt 4.3 bump.

### Features

* Filter deployment and environment queries by user membership (#1565).
* Enhance organization member removal (#1559).
* Block environment listing based on permissions; align frontend interactions (#1535).
* Self-hosted wizard now handles JWT tokens via `DeploymentDTO` (#1425).
* Refactor `useSelfHostedWizard` and related components (#1470).
* Tighter control-plane threshold (#1548).
* Refactor the auth flow and routing logic in `app.vue` (#1483).
* Add a Postgres source connector (#1466).
* Bump Nuxt to 4.3 (#1497).

### Fixes

* Fix bug-batch on control-plane (#1563).
* Fix CP deployment bug #1408 (#1517).
* Front-end fixes (#1511).
* Fix user deletion and config drift (#1473).
* Refactor region type imports and improve formatting in deployment-creation pages (#1455).

### Tests / CI

* Add integration tests in CI (#1555).
* E2E test cases for navigation and security (#1502).
* E2E test cases for member management (#1494).
* E2E test cases for environment management (#1488).
* E2E test cases for deployment management (#1474).
* E2E test cases for organization management (#1460).
* Auto-cleanup for Clerk users; fix auth CI failures (#1485).

### Maintenance

* Helm chart CI experience improvements (#1516).
* Unify local prod ingress routes (#1454).
* First batch of changes (#1515).
* Bump `tar` (#1525); bump `@isaacs/brace-expansion` for control-plane e2e (#1522); generic bump (#1482).


# v2026.02.27 27/02/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-02-27



***

**Description**: Design system kicks off; billing dashboard, audit log and usages API land.

### Changes

* Introduce the new design system (#1550).
* Add a billing section to the dashboard and enhance the billing-details card (#1583).
* Add audit-log functionality (#1584).
* Implement the usages API and related functionality (#1582).


# v2026.03.26 26/03/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-03-26



***

**Description**: Design-system rollout across CP, plus the metrics worker.

### Design system

* Migrate CP onboarding / organization creation (#1727).
* Migrate CP billing (#1728).
* Migrate CP members (#1729).
* Migrate CP usages (#1741).
* Migrate CP deployment (#1736).
* Migrate CP monitoring (#1734).
* New design-system menu (#1730).
* Add the chip component (#1799).
* Documentation component (#1796).
* Design-system component headers (#1753).
* Include resources section in the shared menu (#1758).
* Data-plane breadcrumb routing (#1777).

### Features

* Add the metrics worker (#1664).
* Finalize metrics (#1733).
* Block billing / usages / monitoring pages for non-admin users (#1608).
* Link hook (#1683).
* Script to fill the data-plane with data (#1704).

### Fixes / CI

* Fix deployment listing (#1606).
* Fix the git hook (#1805).
* CVE 19/02/2026 (#1653).
* Rewrite control-plane e2e test cases (#1615).
* Playwright config (#1666); Playwright MCP bug-tracking reports (#1707); manual test plan v2 (#1657).
* Update dependencies for improved functionality (#1687).


# v2026.04.01 01/04/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-04-01



***

**Description**: Design-system: dialog, alert, dedicated button design; breadcrumb fix.

### Changes

* DS — implement & encapsulate `Dialog` (#1853).
* DS — implement & encapsulate `Alert` (#1816).
* Button design (#1815).
* Fix breadcrumb (#1834).


# v2026.04.15 15/04/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-04-15



***

**Description**: Design-system batch (Datatable / Tabs / Tooltips / Date picker), refresh-loop fix, default-env handling, CVE batch.

### Design system

* DS — implement & encapsulate `Datatable` (#1872).
* DS — implement & encapsulate `Tabs` (#1871).
* DS — implement & encapsulate `Tooltips` (#1864).
* DS — implement & encapsulate `Date picker` (#1854).

### Features

* Default environment handling in DP/CP (#1982).
* Update admin URL (#2043).
* Update Nuxt configuration; add `NUXT_PUBLIC_APP_URL` to CI/CI-E2E/Dev/Dev-E2E overlays (#1889).
* Update page titles for control-plane and data-plane in `nuxt.config.ts` (#1893).
* Topic-prefix enhancements (#1901).
* `prod gogo` — production wiring (#1866).

### Fixes

* Fix deployment-name collision error not displayed (#2023).
* Fix control-plane setup in dev (#1955).
* CP multiple fixes (#1943).
* Fix connector requirement for at least one table (#1915).
* Make the data-plane Docker frontend build reproducible (#1935).
* Notification bell does not open any panel (#1887).
* Refresh-loop bug on control-plane (#1880).
* CVE batch (#1883).

### Tests / Chart

* Replace Playwright MCP with the Playwright CLI for e2e (#1966).
* Generate chart secrets from direct values in the data-plane chart (#1961).


# v2026.04.22 22/04/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-04-22



***

**Description**: Snowflake app on CP/DP and a critical Clerk CVE patch.

### Changes

* Snowflake app, control-plane and data-plane integration (#2100).
* Critical Clerk CVE fix (#2073).


# v2026.06.06 07/06/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-06-06



***

**Description**: Replace Karapace with Kora schema registry (dev).

### Changes

* Replace Karapace with Kora schema registry (dev) (#2160).
* Fix usages page: typo, layout, and evolution chart (#2184).
* Profile page: fix name spacing and add edit capability (#2185).
* Control plane CI back to 80% (#2212).
* Implement new support page (#2266).
* Update dependancies due to cve (#2294).
* Language and copy cleanup across CP and DP (#2290).
* Dead navigation affordances: href="#" links and non-clickable DP logo (#2296).
* Upgrade dependancies (#2312).
* CP & DP - Front fixes (#2307).
* Provisioner v0.1 (#1918).
* When inviting a already invited user in an org, server errors on member-invite form are never surfaced to the user (#2332).
* CP deployment creation — Zendesk chat opens on first click + CREATE CHART feedback (#2346).
* Member invitation: no success toast and no Pending badge on the new row (#2343).
* Superuser impersonation (login as another user) (#2330).
* Dedupe singleton background tasks via PG advisory lock (#2323).
* Audit log for CP (#2358).
* Deployment row 3-dot menu: only Delete + Snowflake token, missing generic actions (#2404).
* Romderful to popsink (#2444).
* Include data plane version in heartbeat (#2415).
* CP/DP Login with json web token (#2438).
* CVE 01 06 2026 (#2460).
* Migrate from kafka UI to kotatsu (#2468).
* DP - Multiple fixes (#2503).
* Billing usage alert: email when monthly contract usage crosses configurable % thresholds (#2464).
* Wire SaaS provisioner end-to-end in dev kind (#2410).
* Update with fix scheme (#2516).


# v2026.06.08 08/06/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-06-08



***

**Description**: Canonicalize config hash so drift detection converges.

### Changes

* Canonicalize config hash so drift detection converges (#2523).
* Stop logging full config (member emails / PII) every cycle (#2525).
* Optional dbo (#2545).


# v2026.06.12 12/06/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-06-12



***

**Description**: Remove metrics exporter.

### Changes

* Remove metrics exporter (#2598).
* Tidy up provisioner (#2637).
* Usage alert: email when a connector's replication volume stays below a threshold (#2484).
* Google Ads OAuth — generic, plug-and-play OAuth plumbing (#2511).
* Replace devspace port-forward by a Kind extraPortMapping (#2660).
* Bump control-plane to 2026.06.12 and deploy in preprod (#2670).


# v2026.07.12 12/07/2026
Source: https://docs.popsink.com/changelog/control-plane/cp-2026-07-12



***

**Description**: Front: URL per step in connector creation flow.

### Changes

* Front: URL per step in connector creation flow (#2651).
* CP - Fix UI — Multiple fixes (#2675).
* Removing k8s registry from extraenvvars (#2740).
* Resolve npm Dependabot CVEs — nuxt 4.4.7, vite 7.3.5, ws 8.21.0 (#2780).
* Fixkube rust build context smt rs dropped by 2785 inv upsetupkubeingest fails to build kafka to unity catalog (#2800).
* Tune provisioner some more (#2862).
* Start new deployments in BUILDING until first heartbeat (#2865).
* Fix deprecated annotations on ingressroutes (#2972).
* Double memory for control-plane (#3052).
* CP - Api Page (#2924).
* Make self-hosted connection URL reloadable via deployment id (#3033) (#3041).
* Patch CVEs in control-plane and data-plane dependencies (#3059).
* Introduce observability in dev (#3096).
* \#3098 aggregate worker-metrics in SQL + composite index (#3099).
* \[CP] #3085: accept and edit self-hosted deployment URL (#3097).
* Surface each deployment's running data-plane version in the admin (#3114).
* Bump urllib3 to 2.7.0 in control-plane (#3117).
* Bump control plane 12 07 2026 (#3128).


# v0.1.0-alpha.10 09/02/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-10



***

**Description**: Implement WebSocket support for deployment status updates.

### Changes

* Implement WebSocket support for deployment status updates (#1423).
* Refactor ReceiveDeploymentHeartbeatCommand to include environment rep… (#1429).
* Kconnect heathcheck (#1422).
* Kafka connect fix (#1431).
* Update image tags to version 2026.02.09 in data-plane and control-plane configurations (#1435).
* Update Chart.yaml version to 0.1.0-alpha.10 (#1437).


# v0.1.0-alpha.100 24/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-100



***

**Description**: Show metric tooltips on hover in datamodel detail page (#2879).

### Changes

* Show metric tooltips on hover in datamodel detail page (#2879) (#2885).
* Sync Cargo.lock with popsink-smt 0.2.0 (#2888).


# v0.1.0-alpha.101 26/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-101



***

**Description**: Implement in/nin filter operators (#2856).

### Changes

* Implement in/nin filter operators (#2856) (#2891).
* Make resource search case-insensitive (#2830) (#2887).
* Upgrade ibmi dependency with on prepared statement cache (#2883).
* Bump tansu/kotatsu (#2897).
* Verify role OWNERSHIP on target schema at validation (#2828) (#2898).
* \[DP] Allow pod tolerations on connectors (schedule onto tainted nodes) (#2895) (#2899).
* \[#2876]\[DP] Snapshot v1: orchestrated blocking sync (row counts, syncing state, orchestrator+gate, queue API, UI) (#2919).


# v0.1.0-alpha.102 26/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-102



***

**Description**: \[#2895]\[DP] Front: tolerations editor on connectors (mirror node\_selector).

### Changes

* \[#2895]\[DP] Front: tolerations editor on connectors (mirror node\_selector) (#2921).


# v0.1.0-alpha.103 30/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-103



***

**Description**: Install data-plane via Helm with save/replay across `inv stop` (#2904).

### Changes

* Install data-plane via Helm with save/replay across `inv stop` (#2904) (#2905).
* Add ruff, clippy/rustfmt and gitleaks quality gates (#2890) (#2896).
* Batch metrics (#2941).
* \[#2874]\[DP] Snowflake target: three explicit metadata timestamp columns (#2940).
* On ibmi\_source, add eur dateformat at dds level (non-sql) (#2937).


# v0.1.0-alpha.104 01/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-104



***

**Description**: Salesforce source connector.

### Changes

* Salesforce source connector (#2801).
* \[#2931]\[#2932]\[DP] Unity Catalog target: push typed schema to UC + full Avro type fidelity (#2939).
* Bump msgpack 1.1.2 -> 1.2.1 (Dependabot SEGV/DoS fix) (#2948).
* DP - Subscription fixes (#2882).
* DP - New admin & monitoring page (#2916).


# v0.1.0-alpha.105 01/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-105



***

**Description**: Remove ownership test.

### Changes

* Remove ownership test (#2960).


# v0.1.0-alpha.106 01/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-106



***

**Description**: First version of MS SQL raw transaction log connector.

### Changes

* First version of MS SQL raw transaction log connector (#2951).


# v0.1.0-alpha.107 01/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-107



***

**Description**: No ownership.

### Changes

* No ownership (#2961).


# v0.1.0-alpha.108 01/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-108



***

**Description**: Recover the CI smoke — fixes, IBM i/Snowflake modules, reorg, rich Slack digest.

### Changes

* Recover the CI smoke — fixes, IBM i/Snowflake modules, reorg, rich Slack digest (#2950).
* \[#2932] DP - Unity Catalog: resolve Avro named-record refs for typed schema (#2962).
* \[#2966]\[DP] Snowflake target: heal pre-#2874 tables (ambiguous \_\_snowflake\_ts\_ms) (#2967).


# v0.1.0-alpha.109 01/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-109



***

**Description**: Force bump.

### Changes

* Force bump (#2968).


# v0.1.0-alpha.11 17/02/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-11



***

**Description**: SSL support for PostgreSQL connections; chart maintenance.

### Changes

* Add SSL support for PostgreSQL connections in data-plane (#1507).
* Maintenance bump folded in from `0.1.0-alpha.10` (#1437).


# v0.1.0-alpha.110 01/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-110



***

**Description**: \[#2966]\[DP] Snowflake target: actually heal ambiguous \_\_snowflake\_ts\_ms (SHOW COLUMNS + reconcile backstop).

### Changes

* \[#2966]\[DP] Snowflake target: actually heal ambiguous \_\_snowflake\_ts\_ms (SHOW COLUMNS + reconcile backstop) (#2969).


# v0.1.0-alpha.111 01/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-111



***

**Description**: Upgrade kora to v040 in data plane chart.

### Changes

* Upgrade kora to v040 in data plane chart (#2975).


# v0.1.0-alpha.112 02/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-112



***

**Description**: Pin dev install image to loaded `latest` to avoid ImagePullBackOff (#2936).

### Changes

* Pin dev install image to loaded `latest` to avoid ImagePullBackOff (#2936) (#2949).
* \[#2963]\[DP] Worker CPU/memory tier configurable via advanced.resources (#2971).
* \[#2987]\[DP] Pin ruff to CI version (0.14.14) and clear repo-wide format drift (#2988).
* \[#2963]\[DP] Front: worker resource-tier selector on CDC source connectors (#2996).


# v0.1.0-alpha.113 06/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-113



***

**Description**: Authenticate notification & kafka-stream WebSockets.

### Changes

* Authenticate notification & kafka-stream WebSockets (#2946).
* Extract user profile into service + composable (#2930).
* \[#2979]\[DP] Batch tables under 10k rows into a single blocking snapshot (#3001).
* \[#2981]\[DP] Gate Backfill banner on DB sync state, not worker WS status (#3002).
* Oracle support for the metadata DB (#2826) (#2928).
* Design system - New bulk action data table (#2953).
* \[FRONT] Part 1 of #3005: extract team creation into service + composable (#3008).
* \[#2947]\[DP] Connectors: scope reads to caller's team + redact inline credentials (#2984).
* Move connector check DTOs out of presentation into application (#2929).
* Cover ConnectorAuthorizationDomainService permission branches (#2958).
* Cover GET /error-table/error-tables listing endpoint (#3011).
* \[SEC] Part 1 of #3006: require authentication on connector pre-flight endpoints (#3012).
* Subscription form: disable Apply + show loader during POST /subscriptions/sync (#2998).
* Connector detail: subscriptions list shows the target logo in the Source column (#2994).
* Normalise error\_table target\_id UUID across DB dialects (#3017).
* Move PasswordHasherPort to domains/ports (#2982) (#2983).
* \[#2985]\[DP] Enforce cross-team object-level authorization on connector/datamodel/subscription/error-table writes (#2986).
* \[ARCH] Part 1 of #3007: pass user\_id to domain stream services (drop infra User import) (#3009).
* Sync queue: drag-drop reorder + move to top/bottom for queued tables (#2999).
* Wipe shared Oracle schema on a fresh `inv dp.install --db oracle` (#3020).
* \[FRONT] Part 2 of #3005: extract env join-requests into service + composable (#3014).


# v0.1.0-alpha.114 06/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-114



***

**Description**: Include current day in per-datamodel usage chart.

### Changes

* Include current day in per-datamodel usage chart (#3035).
* \[#3015]\[DP] Build Oracle metadata DB URL from components (#3016).


# v0.1.0-alpha.115 07/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-115



***

**Description**: Surface a failed snapshot as FAILED instead of a silent COMPLETED/0-rows (#3042).

### Changes

* Surface a failed snapshot as FAILED instead of a silent COMPLETED/0-rows (#3042) (#3045).
* Zip nested/duplicate/sibling wildcards instead of fanning out (#3028) (#3029).
* Bump 07 07 2026 (#3057).


# v0.1.0-alpha.116 09/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-116



***

**Description**: \[FRONT] Part 3 of #3005: extract team members-list into service + composable.

### Changes

* \[FRONT] Part 3 of #3005: extract team members-list into service + composable (#3049).
* OpenTelemetry OTLP phase 1 — backend traces/metrics/logs (#3039) (#3046).
* \[TESTS] Part 2 of #3010: cover ErrorTableValidationService Oracle-only target rule (#3025).
* \[SEC] Part 3 of #3006: object-level authz on /ws/kafka-stream (#3024).
* Patch CVEs in control-plane and data-plane dependencies (#3059).
* Stop editing a connector from destroying inline credentials (#3047) (#3048).
* OpenTelemetry OTLP phase 2 — Python workers (#3039) (#3062).
* \[SEC] Redact inline credentials in PipelineRead source/target config (#3069).
* \[TESTS] Part 1 of #3050: cover worker-lifecycle domain services (#3071).
* \[TESTCLEAN] Part 1 of #3022: drop duplicated validation coverage in test\_pipeline\_name (#3023).
* \[ARCH] Part 2 of #3007: move advanced-config validators to application layer (#3026).
* OpenTelemetry OTLP phase 3 — Java/Quarkus workers (#3039) (#3073).
* \[FRONT] Part 1 of #3068: extract add-member-dialog into service + composable (#3070).
* \#3077 aggregate metric monitoring in SQL + composite indexes (#3100).
* Introduce observability in dev (#3096).
* Connector "Last sync" should show sync end, not start (#3092).
* Decode avro\_unicode \_uXXXX escapes to their character in the UI (display-only) (#3106).
* In snow\_target, add \_\_hash key column computed on non-LOB columns (#3109).


# v0.1.0-alpha.117 16/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-117



***

**Description**: \[FRONT] Part 2 of #3068: extract connector list into service + composable.

### Changes

* \[FRONT] Part 2 of #3068: extract connector list into service + composable (#3105).
* Surface each deployment's running data-plane version in the admin (#3114).
* Dashboard shows "Pipelines" stat card when pipeline mode is off (#3112).
* \[FRONT] Part 3 of #3068: extract error-table-config-drawer into services + composable (#3116).
* OTel phase 4 — browser trace-context propagation (data-plane #3090 + control-plane) (#3110).
* \[TESTS] Part 2 of #3050: cover member-deletion authorization branches (#3103).
* OTel phase 4 — Tansu broker OTLP metrics export (#3089) (#3094).
* \[TESTCLEAN] De-flake worker reconciliation error-recovery tests (#3104).
* OpenTelemetry OTLP phase 4 — native Rust worker (#3039) (#3093).
* IBM i auto-recovers from a pruned journal (earliest) + bump popsink\_connect images (#3126).
* Sync orchestration floods ERROR tracebacks when a worker is transiently unavailable (no running pod / 5xx) (#3118).
* Allow W3C trace-context headers through Traefik CORS middleware (#3132).
* \[ARCH] Part 3 of #3007: move get\_user\_logs to a UserLog read port (#3102).
* Add SALESFORCE\_SOURCE to connector credential inventory (#3101).
* Sync history: persist & expose past sync runs (back + front) (#3040).
* Allow editing a subscription's name (backend already supports it) (#3133).
* Cover is\_config\_ready completion gate branches (#3148).
* \[ARCH] Part 1 of #3144: relocate PipelineState enum into the domain layer (#3146).
* \[FRONT] Part 1 of #3143: extract notifications drawer into service + composable (#3145).
* \[SEC] Part 3 of #3067: Redact sensitive headers in /healthchecks/debug reflection (#3142).
* Harden over-mocked sync tests to assert observable behavior (#3147).
* Upgrade a SaaS deployment to the latest version from the CP admin (#3115).
* DP: cache license verdict from heartbeat (grace window) (#3157).
* DP: soft-gate get\_current\_active\_user on license (+ kill-switch) (#3159).
* Add pop-access-denied (on pop-empty-state) and unify the CP + DP gate (#3156).
* \[SEC] Part 1 of #3153: Close cross-tenant IDOR on worker state & heartbeat endpoints (#3174).
* \[ARCH] Part 2 of #3144: raise domain exceptions from env\_request domain service (#3176).
* \[TESTCLEAN] Harden can't-fail assertions in connector-types credential/fetch tests (#3172).
* Sql\_database relational sources (Postgres, MySQL, SQL Server, Oracle) (#3158).
* Bump 15 07 2026 (#3170).


# v0.1.0-alpha.118 16/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-118



***

**Description**: DP/front: license-inactive screen when access is blocked.

### Changes

* DP/front: license-inactive screen when access is blocked (#3162).
* Disable Quarkus OTel SDK on popsink\_connect sources (#3179).


# v0.1.0-alpha.119 16/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-119



***

**Description**: TBD.

### Changes

* *No tracked changes detected for this release.*


# v0.1.0-alpha.12 18/02/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-12



***

**Description**: Helm chart cleanup and CI experience improvements.

### Changes

* Clean up chart values and add CI helpers to manage chart versioning (#1516).


# v0.1.0-alpha.120 17/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-120



***

**Description**: Better naming snapshot.

### Changes

* Better naming snapshot (#3190).


# v0.1.0-alpha.121 18/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-121



***

**Description**: \[TESTCLEAN] Part 2 of #3154: harden connector-types listing / config-schema tests.

### Changes

* \[TESTCLEAN] Part 2 of #3154: harden connector-types listing / config-schema tests (#3191).
* \[ARCH] Part 3 of #3144: raise domain exceptions from user domain service (#3192).
* \[FRONT] Part 2 of #3143: extract shared members-list into service + composable (#3173).
* \[SEC] Part 2 of #3153: Authorize worker metric ingest endpoints (#3193).
* Backfill / Sync queue: copy & wording fidelity vs Figma (#3168).
* Relax kafka polling, upgrade tansu in IT, bump versions (#3206).


# v0.1.0-alpha.122 21/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-122



***

**Description**: \[ARCH] Part 1 of #3208: relocate ControlPlanePort contract DTOs into the domain layer.

### Changes

* \[ARCH] Part 1 of #3208: relocate ControlPlanePort contract DTOs into the domain layer (#3209).
* \[SEC] Part 3 of #3153: authenticate POST /smt/process\_mapper (#3212).
* \[TESTCLEAN] Part 3 of #3154: remove empty test\_kafka\_list\_topics.py stub (#3213).
* Ask for warehouse to enable dynamic tables (#3220) (#3224).
* \[FRONT] Part 3 of #3143: extract subscribe-dialog into service + composable (#3210).
* \[TESTCLEAN] Part 1 of #3233: harden over-mocked delete\_datamodel command test (#3234).
* \[SEC] Part 1 of #3219: require auth on POST /envs/check-byok-credentials (#3231).
* Connector registration, PK warning, OAuth backbone (Part 1 of #3160) (#3204).
* Bump data-plane + kafka-to-snowflake worker to 2026.07.21+ enhancements to kafka-to-snowflake (#3241).


# v0.1.0-alpha.123 23/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-123



***

**Description**: \[TESTS] Part 1 of #3155: integrity-error → 409/500 mapping coverage.

### Changes

* \[TESTS] Part 1 of #3155: integrity-error → 409/500 mapping coverage (#3175).
* Live property endpoints + mapper gestion UX (Part 2 of #3160) (#3238).
* \[FRONT] Part 1 of #3218: extract connector lineage metrics into service + composable (#3250).
* \[TESTCLEAN] Part 2 of #3233: drop implementation-detail call assertions in get\_worker\_status query test (#3252).
* \[ARCH] Part 2 of #3208: domain AuthRepository references domain entities, not infra model / app DTO (#3249).
* \[SEC] Part 2 of #3219: regression guard for unauthenticated routes (#3251).
* Qa: E2E exploration session 2026-07-20 — findings + mapper test harness (#3243).
* Sync queue: add Cancelled and Errors table sections (#3165).
* Remove two redundant edge-case tests duplicating richer coverage (#3271).
* Warehouse picker + dynamic-tables toggle in the form (#3263) (#3264).
* Bump versions data-plane & snow\_target (#3279).


# v0.1.0-alpha.124 23/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-124



***

**Description**: \[TESTS] Part 2 of #3155: subscription sync reconciliation branches.

### Changes

* \[TESTS] Part 2 of #3155: subscription sync reconciliation branches (#3270).
* Kafka\_to\_jdbc emits per-subscription consumption metrics (#3127).
* In back-end and front-end, wire MS Sql Server target (#3164).
* Use stored credentials when testing in edit mode + Snowflake warehouse edit (#3280) (#3284).


# v0.1.0-alpha.125 27/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-125



***

**Description**: \[SEC] Part 1 of #3268: SSRF guard for connector URL credential checks.

### Changes

* \[SEC] Part 1 of #3268: SSRF guard for connector URL credential checks (#3269).
* Consolidate all connector Kubernetes config under a single Advanced configuration section (#3237).
* \[FRONT] Part 2 of #3218: extract datamodel lineage metrics into composable (#3267).
* \[TESTCLEAN] Part 1 of #3291: remove can't-fail / zombie / duplicate pipeline-entity tests (#3292).
* DP - Auto initial load (#3207).
* BigQuery target: support key-less (append) subscriptions — make the primary key optional (#3260).
* \[SEC] Part 2 of #3268: SSRF guard for DLT schema-fetch background task (#3289).
* \[TESTS] Part 3 of #3155: env/team request-creation rejection branches (#3290).
* \[ARCH] Part 3 of #3208: relocate repository-interface contract DTOs into the domain layer (#3266).
* \[SEC] Part 3 of #3268: reject non-DNS-label subdomain in Pipedrive/Zendesk checks (#3321).
* \[TESTS] Part 1 of #3322: connector-OAuth proxy route auth + CP-error mapping (#3323).
* \[TESTCLEAN] Part 2 of #3291: harden can't-fail env id assertions (#3324).
* Kafka to jdbc dependency upgrades better jvm docker file (#3326).


# v0.1.0-alpha.126 27/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-126



***

**Description**: New deps need to be added to pass es\_target native build.

### Changes

* New deps need to be added to pass es\_target native build (#3331).


# v0.1.0-alpha.127 30/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-127



***

**Description**: Filesystem source (S3, GCS, Azure ADLS, SFTP, local).

### Changes

* Filesystem source (S3, GCS, Azure ADLS, SFTP, local) (#3214).
* Add source offset & partition as record headers on kafka-to-kafka (#3265).
* BigQuery target: remove the streaming-write throughput ceiling (Storage Write API) (#3315).
* Security - Fix all open Dependabot CVEs across lockfiles (#3340).
* \[SEC] Part 1 of #3345: redact Unity Catalog target credentials on read (#3347).
* \[TESTS] Part 2 of #3322: WorkerHeartbeatRequest legacy packed-state parsing (#3348).
* \[FRONT] Part 3 of #3218: extract broker BYOK check into service + composable (#3320).
* \[TESTCLEAN] Part 3 of #3291: remove redundant kafka-source accessibility probe (#3349).
* BigQuery target: BigQuery-specific target-table-name sanitization (override the Snowflake default) (#3254).
* Serve data-plane front from a dedicated dev container via Traefik (#3335).
* Serve the control-plane front from a dedicated dev container (#3339).
* Kafka->HubSpot delivery worker (Part 3 of #3160) (#3255).
* Record refresh\_token in the credential inventory + OAuth-target pattern (#3160) (#3354).
* Remove log filtering for java workers => logs were stopping at the first stacktrace (#3226).
* Expose the connector's char-field trim mode as an advanced option (#3332) (#3362).
* \[ARCH] Part 1 of #3288: stop schema domain service importing SchemaRead DTO (#3363).
* \[SEC] Add SSRF guard to Unity Catalog target credential check (#3366).
* Cover POST /brokers/ normalization + error swallowing (#3367).
* \[TESTCLEAN] Part 1 of #3368: remove zero-assertion kube\_api tests (#3369).
* \[FRONT] Part 1 of #3364: extract team-card into service + composable (#3365).
* DP - CDC events experience (#3319).
* DataModel detail: Consumers list shows the source logo instead of the target (#3353).
* Introduce networkpolicies in data plane chart (#3376).


# v0.1.0-alpha.128 30/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-128



***

**Description**: Drop Oracle as the data-plane application/metadata database (Postgres only).

### Changes

* Drop Oracle as the data-plane application/metadata database (Postgres only) (#3352).
* \[ARCH] Part 2 of #3288: relocate UserLazyLoadService to application layer (#3378).
* Bump versions data-plane + snow\_target (#3385).


# v0.1.0-alpha.129 30/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-129



***

**Description**: \[TESTS] Part 1 of #3381: lock ConnectorType classification rules.

### Changes

* \[TESTS] Part 1 of #3381: lock ConnectorType classification rules (#3382).
* \[TESTCLEAN] Part 2 of #3368: consolidate ResourceName accept-case tests (#3383).
* In snow\_target, unit test update was missing (#3388).


# v0.1.0-alpha.13 25/02/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-13



***

**Description**: Configurable Tansu replicas via Helm value.

### Changes

* Tansu replica count is now configurable through a Helm value; default replica count increased and resources reduced (#1567).
* Image-tag refresh to `2026.02.25` folded in from `0.1.0-alpha.14` (#1568).


# v0.1.0-alpha.130 30/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-130



***

**Description**: \[FRONT] Part 2 of #3364: extract error-table into services + composable.

### Changes

* \[FRONT] Part 2 of #3364: extract error-table into services + composable (#3379).
* Connectors - Fix SSH tunnels broken by paramiko >= 4 (#3390).
* Fold JAVA\_OPTS\_APPEND into JAVA\_OPTIONS for the fabric8 launcher (#3395).


# v0.1.0-alpha.131 30/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-131



***

**Description**: Derive DataModel primary keys from the Debezium message-key schema.

### Changes

* Derive DataModel primary keys from the Debezium message-key schema (#3358).


# v0.1.0-alpha.132 30/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-132



***

**Description**: Upgrade tansu to 0.7.0-beta.27, fix its env, expose its config.

### Changes

* Upgrade tansu to 0.7.0-beta.27, fix its env, expose its config (#3377).


# v0.1.0-alpha.133 31/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-133



***

**Description**: \[SEC] Part 3 of #3345: remove hardcoded live credentials from sandbox.py.

### Changes

* \[SEC] Part 3 of #3345: remove hardcoded live credentials from sandbox.py (#3380).
* DP - Bulk actions (#3283).
* For release, bump ibmi\_source + data-plane (#3412).


# v0.1.0-alpha.134 31/07/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-134



***

**Description**: \[SEC] Part 1 of #3413: remove admin/admin default-credential fallback in AdminAuth.

### Changes

* \[SEC] Part 1 of #3413: remove admin/admin default-credential fallback in AdminAuth (#3414).
* \[TESTCLEAN] Part 3 of #3368: harden mock/behavior mismatch, drop duplicated delegation test (#3417).
* \[ARCH] Part 3 of #3288: remove ExportData DTO import from domain admin layer (#3409).
* \[TESTS] Part 2 of #3381: lock Subscription entity PK-resolution & invariants (#3416).
* \[FRONT] Part 3 of #3364: extract debezium-source list-tables into service + composable (#3411).
* Front - gate mapper dynamic-tables option by target type (#3356).
* In snow\_target, Force jvm mem settings (#3428).


# v0.1.0-alpha.135 01/08/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-135



***

**Description**: Workers - replace status WebSocket with HTTP polling.

### Changes

* Workers - replace status WebSocket with HTTP polling (#3393).
* Workers - replace logs WebSockets with HTTP polling (#3396).
* Kafka live stream - HTTP polling instead of WebSocket (#3397).
* Notifications - poll unread count over HTTP instead of WebSocket (#3407).
* Tear down shared WebSocket infrastructure (#3278) (#3408).
* Make organization API keys authenticate (#3327) (#3373).
* Allow finer control over OTEL (#3403).
* Quarantine an unprovisionable table instead of starving the connector (#3433).


# v0.1.0-alpha.14 25/02/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-14



***

**Description**: Implement test components.

### Changes

* Implement test components (#1558).
* Bump version numbers for data-plane and control-plane components (#1568).


# v0.1.0-alpha.15 27/02/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-15



***

**Description**: Default `Service` type for data-plane changed.

### Changes

* Change the default Kubernetes `Service` type for the data-plane Service (#1595).


# v0.1.0-alpha.16 03/03/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-16



***

**Description**: New design system: shared `@popsink/ui` package, Tailwind v4, AppBar / UserDropdown.

### Changes

* Major UI overhaul (#1550):
  * Extract a shared `@popsink/ui` package with design tokens, Vuetify config and `Geist` typography.
  * Migrate from `@nuxtjs/tailwindcss` (v3) to `@tailwindcss/vite` (v4); centralise shared modules in `packages/ui`.
  * Apply `pop-title` / `pop-content` typography to data-plane and control-plane.
  * Introduce shared `AppBar` and `UserDropdown` components; remove the legacy `AppNavbar`.
  * Configure CSS cascade layers so Vuetify and Tailwind v4 coexist cleanly.
  * Standalone build pipelines updated to integrate `@popsink/ui` for both data-plane and control-plane.


# v0.1.0-alpha.17 05/03/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-17



***

**Description**: Pull `snowflake-kafka-connector` from Artifact Registry.

### Changes

* Update the `kafka-connect` image to fetch the `snowflake-kafka-connector` from Artifact Registry; chart image bumped to `2026.03.05` (#1620).


# v0.1.0-alpha.18 09/03/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-18



***

**Description**: Azure dependency bump and miscellaneous fixes.

### Changes

* Bump `cryptography` to 46.0.5 and refresh component versions to `2026.03.09` (#1630).
* Disable SSL warnings for `KubeApi` insecure requests.
* Standardise logging timestamp format across modules.


# v0.1.0-alpha.19 27/03/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-19



***

**Description**: Fix GET /connectors/ returning 500 instead of 404 for deleted.

### Changes

* Fix GET /connectors/ returning 500 instead of 404 for deleted (#1629).
* Fix settings.json with select-deployment (#1621).
* Refactor error table component to improve data handling and UI consistency (#1649).
* Maxi lint (#1650).
* Delete connector fix + maxi lint (#1651).
* Implement connector configuration encryption and decryption functiona… (#1604).
* Fix (#1655).
* CVE 19 02 2026 (#1653).
* Harden curl commands in kafka connect dockerfile (#1652).
* Refac (#1654).
* Metric send usage (#1619).
* E2e/manual test plan v2 (#1657).
* Schema Primary Key (#1617).
* Unwrap Debezium envelope in primary key field selection (#1659).
* Streamline code formatting and enhance readability across multiple files (#1661).
* E2e/playwright config (#1666).
* Add the metrics worker (#1664).
* Add source topics listing functionality for connectors (#1667).
* Fix pipeline connector name (#1668).
* Link hook (#1683).
* Integrate metrics-exporter worker functionality (#1684).
* Update dependencies for improved functionality (#1687).
* Create a script to fill the data-plane with data (#1704).
* Add playwright MCP bug tracking reports and migration (#1707).
* Update user management form and logic for password handling (#1706).
* Remove outdated test reports and planning files (#1720).
* Introduce TopicPrefixService for topic prefix generation in… (#1699).
* Enhance data model with source connector type and update UI logic (#1705).
* Dlt source (#1726).
* Update pyasn1 dependency to version 0.6.3 in pyproject.toml and uv.lock (#1745).
* New design system - Menu (#1730).
* Audit, bug fixes and TypeScript migration plan (#1743).
* Enhance mapper tests for handling wildcard paths and nullable fields (#1737).
* DP - Data Models (#1750).
* DP - Connectors (#1731).
* Add transaction\_buffered configuration for IBM i source connector (#1742).
* Finalize metrics (#1733).
* Include resources section in shared menu (#1758).
* Scaffold of the typeScript migration (#1759).
* Enhance pipeline status counting with search and team filtering (#1749).
* Add Iceberg Target connector documentation and update component to display it (#1748).
* TypeScript migration — Phase 2 auth + navigation (#1765).
* TypeScript migration — catalog and pipelines modules (#1769).
* Update data plane helm (#1768).
* M tls kafka (#1762).
* Add TypeScript specs for sources and targets modules (#1771).
* Bug zindex sync (#1782).
* Migrate domains and security modules to TypeScript (#1773).
* Migrate env/notif/user/info modules and remove legacy JS (#1797).
* DP - Domain action (#1770).
* DP - DLT integration (#1778).
* Rustfs instead of minio (#1802).
* Fix git hook (#1805).
* Components design system headers (#1753).
* Enhance connector detail navigation by introducing a dedicated route function. Updated row click and detail view links to use the new `connectorDetailPath` method for improved clarity and maintainability (#1813).
* Create documentation component (#1796).
* Cleanup and remove legacy JS files and migrate to TypeScript (#1806).
* Data plane breadcrumb routing (#1777).
* Add chip component (#1799).
* Enhance transform step configuration UI with informative messages for missing datamodel and transform steps. Added conditional rendering for user guidance in the pipeline setup process (#1810).
* Update version numbers for data-plane and control-plane components to 2026.03.26 and bump chart version to 0.1.0-alpha.19 (#1820).


# v0.1.0-alpha.2 27/01/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-2



***

**Description**: Mass bump dependencies.

### Changes

* Mass bump dependencies (#1177).
* Upgrade node-forge to 1.3.2 and qs to 6.14.1 to fix CVE vulnerabilities (#1194).
* Add filtered existing connectors by selected type in selection component (#1193).
* Add margin to loading indicator in status component (#1196).
* Update versions for data-plane and kafka-to-iceberg components to 2026.01.08.1; remove debug logging in IcebergTargetAdapter class; set logging levels for pyiceberg and azure to WARNING in main.py (#1198).
* Move cleanup e2e tests from popsink to data-plane directory (#1200).
* Error table behaviour (#1195).
* Az bugs (#1208).
* \[Sqlite -> PG migration] Remove everything related to Atlas and use Alembic instead (#1102).
* Feature/test refactoring plan (#1230).
* Add PIPELINE\_MODE feature flag (#1199).
* Fix typing (#1234).
* Update primary key toggle logic and clean up column configuration checks in mapper.vue (#1239).
* Align breadcrumb and burger menu (#1225).
* Close #1212 (#1226).
* Pipeline name ellipsis (#1227).
* Subscription front fixes (#1231).
* It should not be possible to edit the source or target connector of a pipeline (#1235).
* Update azure-core dependency to version 1.38.0 and add it to project dependencies (#1241).
* Change pipeline start/pause to manage subscription state instead of worker pods (#1192).
* Datamodel front fix (#1244).
* Env creation fix (#1245).
* Source target fix (#1246).
* Add zendesk resources (#1250).
* Fix tests (#1251).
* Connector types enum (#1258).
* Update entrypoint.sh to increase Uvicorn worker count from 1 to 3 for improved performance (#1265).
* Force h3\@1.15.5 and devalue\@5.6.2 via yarn resolutions (#1263).
* Fixed TC10 and TC11 due to front changes (#1272).
* Pg pools (#1278).
* Fix CVE issue #1268 (#1279).
* Implement connector lock state for edit mode (#1275).
* Update image tags for data-plane and connectors to version 2026.01.19 (#1284).
* Feature/test refactoring (#1253).
* Remove pvc for Allianz Chart (#1288).
* Enhance Kafka and message processing functionality (#1289).
* Data-Plane - Control-Plane communication (#1280).
* Add karapace to data plane chart (#1281).
* Update data-plane image tag to version 2026.01.20 and modify service account annotations for EKS role ARN. Change v-data-table to v-data-table-virtual in configure-many.vue for improved performance (#1299).
* Refactor WebSocket routes to use session context manager (#1294).
* Merge pipeline modules and fix test selectors (#1301).
* Update Kafka source form to include metadata and adjust layout in configure-many.vue for improved styling. Modify subscriptions-builder store to return selected subscriptions directly instead of readonly (#1307).
* User logs (#1302).
* Refactor DEPLOYMENT\_MODE setting for consistency and update app lifespan logic. Changed DEPLOYMENT\_MODE from "SELF HOSTED" to "SELF\_HOSTED" for improved type safety and consistency across the codebase (#1300).
* Update SQLAlchemyConnectorReadPort to reintroduce distinct clause for PostgreSQL compatibility. This change ensures proper handling of JSON columns while maintaining the intended functionality in subscription queries (#1309).
* Update image tags and buffer sizes for Kafka connectors and increase memory limits in Kubernetes configuration. Updated data-plane and connector versions to 2026.01.21, reflecting the latest changes in image tags and buffer size adjustments for Iceberg and Oracle target adapters. (#1310) (#1312).
* Refactor authentication and user management components (#1313).
* Revert "Refactor authentication and user management components (#1313)" (#1316).
* Fix admin auth (#1319).
* Cves (#1322).
* Control plane preprod environment (#1292).
* E2E Tests: Catalog module activation & Slack reporter fix (#1323).
* Fixing credentials in the code (#1291).
* Fix failing E2E tests and team deletion with requests (#1324).
* Use popsinkcom as email domain in e2e tests (#1326).
* E2e/fix pp tests version (#1335).
* Host data plane helm chart (#1336).


# v0.1.0-alpha.20 30/03/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-20



***

**Description**: Breaking: rename chart secret key `CA_KEY` to `KEY`.

### Changes

* Rename `CA_KEY` to `KEY` in the data-plane chart — existing values referencing `CA_KEY` must be updated (#1833).
* Image-tag refresh to `2026.03.26` folded in from `0.1.0-alpha.19` (#1820).


# v0.1.0-alpha.21 30/03/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-21



***

**Description**: Kafka integration improvements: Avro deserialization and DLT passthrough.

### Changes

* Refactor Kafka message fetching to better handle internal broker credentials (#1836).
* Introduce Avro deserialization support in the Kafka API.
* Add a passthrough mode for the DLT-to-Kafka connector.
* Refresh component image tags to `2026.03.30`.


# v0.1.0-alpha.22 01/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-22



***

**Description**: More Kafka integration polish.

### Changes

* Handle decimal serialization in the Kafka target adapter (#1863).
* Adjust schema-evolution handling in `StatelessMessageRunner` for non-Kafka targets.
* Improve Kafka source adapter logging for better configuration visibility.
* Refresh component image tags to `2026.04.01`.


# v0.1.0-alpha.23 01/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-23



***

**Description**: Refactor admin credentials in chart and backend.

### Changes

* Refactor backend admin credentials and chart deployment template; improved `adminCredentials` documentation (#1857).


# v0.1.0-alpha.24 07/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-24



***

**Description**: Debezium logical date fields normalised to ISO format.

### Changes

* Add normalisation for Debezium logical date fields after Avro deserialization, including nullable and nested-record handling (#1891).
* Update `SchemaRegistryMessageDecoder` and `KafkaSourceAdapter` to use the new normalisation utility.
* Image tag refresh to `2026.04.07`.


# v0.1.0-alpha.25 09/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-25



***

**Description**: Single JWT secret per data-plane; stricter token-type checks.

### Changes

* Unify `JWT_SERVICE_ACCOUNT_SECRET` and `METRICS_EXPORTER_TOKEN` into a single `JWT_SECRET` per data-plane; remove the `serviceAccountSecret` from the chart (#1922).
* Reject service-account tokens in `get_current_user` so the existence check still runs through `get_current_service_account_user`.
* Add a `token_type` claim to user and service-account tokens.


# v0.1.0-alpha.26 10/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-26



***

**Description**: metrics-exporter is now part of the data-plane chart.

### Changes

* Bootstrap `metrics-exporter` templates inside the data-plane chart; build and push images alongside data-plane (#1902).
* Refactor CI tasks to push to each registry independently.
* `metrics-exporter` version is now pinned in chart values.
* Dev: expose port 5005 for `metrics-exporter` debugging.


# v0.1.0-alpha.27 10/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-27



***

**Description**: mTLS support for `metrics-exporter`.

### Changes

* Add mTLS support to `metrics-exporter`, including a workaround for a Quarkus config bug and a Quarkus upgrade (#1959).


# v0.1.0-alpha.28 10/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-28



***

**Description**: Generate chart secrets directly from values.

### Changes

* Generate Kubernetes Secrets from direct chart values for `deploymentJwtToken`, `connectorConfigEncryptionKey`, `adminCredentials`, JWT secret, `schemaRegistry`, `tansuStorageAws`, `defaultKafka` and `externalDatabase` (#1961).
* Drop `-ref` suffix from data-plane deployments and fix checksum annotation names.
* Add the missing `secrets.yaml` template for `metrics-exporter`.
* `metricsExporter` is now enabled by default.


# v0.1.0-alpha.3 30/01/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-3



***

**Description**: Fix CVE 27 01 2025.

### Changes

* Fix CVE 27 01 2025 (#1338).
* Add available targets and sources to deployment configuration (#1329).
* E2e/pipelines (#1346).
* Implement toaster notifications on data-plane (#1283).
* Ibmi connector source (#1348).
* E2e/flakky tests (#1353).
* User management (#1315).
* Integrate Schema Registry for Avro schema retrieval (#1356).
* Bump tar (#1364).
* Snowflake target part 1 (#1339).
* Fix cmd inv dp.sd (#1365).
* Add support for imagepullsecrets in data plane chart (#1358).


# v0.1.0-alpha.30 10/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-30



***

**Description**: Fix `metrics-exporter` entrypoint and missing secret.

### Changes

* Fix `metrics-exporter` entrypoint, add the missing `METRICS_EXPORTER_TOKEN` secret, and add a PKCS8 key for `metrics-exporter` (#1964).


# v0.1.0-alpha.31 13/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-31



***

**Description**: SHA1 prefix for `deploymentId`; clearer chart error messages.

### Changes

* Use a SHA1 prefix for `deploymentId` in `metrics-exporter` and fail fast with an explicit error message when it is empty (#1993).
* Add validation error messages for missing values and missing `existingSecrets` in the data-plane chart.
* `schemaRegistry` secret is now optional when using Karapace.


# v0.1.0-alpha.32 14/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-32



***

**Description**: Two replicas by default; explicit error when Tansu lacks S3 retention.

### Changes

* Default to 2 data-plane replicas; bump `metrics-exporter` and chart (#2006).
* Surface an explicit error if Tansu is enabled but S3 retention is not provided.
* Remove the dedicated preprod / prod overlays for `metrics-exporter` (now bundled in the chart).


# v0.1.0-alpha.33 14/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-33



***

**Description**: Fix `metrics-exporter` readiness probe.

### Changes

* Fix the HTTP call in the `metrics-exporter` readiness probe (#2013).


# v0.1.0-alpha.34 14/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-34



***

**Description**: Add missing `API_URL` variable for `metrics-exporter`.

### Changes

* Add the missing `API_URL` variable for `metrics-exporter`, including the dev configmap; simplify URLs in `_helpers` (#2026).


# v0.1.0-alpha.35 14/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-35



***

**Description**: Image-sync workflow rewritten in Python; bump Tansu to v0.6.0.

### Changes

* Rewrite the `sync-image-reusable` workflow in Python (with unit tests) to fix flaky SemVer comparisons in Bash; bump the Tansu image to `v0.6.0` (#2027).


# v0.1.0-alpha.36 14/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-36



***

**Description**: Breaking: drop `retention.s3` block from chart values.

### Changes

* Remove the legacy S3-retention configuration block from the data-plane chart (#2031).


# v0.1.0-alpha.37 15/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-37



***

**Description**: SMT: Rust port + PBKDF2 encrypt fix (2-24 000x speedup).

### Changes

* SMT: Rust port + PBKDF2 encrypt fix (2-24 000x speedup) (#1972).
* E2E: extend smoke test to 23 steps + remove kafka-oracle-manager (#2033).
* Additional delay (#2044).


# v0.1.0-alpha.38 15/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-38



***

**Description**: Simpler websocket handling.

### Changes

* Simplify websocket handling in the data-plane backend (#2049).
* Minor delay tweak folded in from `0.1.0-alpha.37` (#2044).


# v0.1.0-alpha.39 15/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-39



***

**Description**: Drop IRSA / credentials enforcement for Tansu.

### Changes

* Stop enforcing the presence of IRSA or static credentials for Tansu — supply whichever the deployment needs (#2055).


# v0.1.0-alpha.4 30/01/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-4



***

**Description**: Fix data plane entrypoint.

### Changes

* Fix data plane entrypoint (#1369).


# v0.1.0-alpha.40 16/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-40



***

**Description**: Strategy.

### Changes

* Strategy (#2057).
* New audit logs (#2056).


# v0.1.0-alpha.41 16/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-41



***

**Description**: Deployment strategy adjusted away from `Replace`.

### Changes

* Use a non-`Replace` deployment strategy for the data-plane Deployment (#2062).
* Image bump folded in from `0.1.0-alpha.40` (#2060).


# v0.1.0-alpha.42 17/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-42



***

**Description**: Settings: env vars now required, no in-code fallbacks.

### Changes

* Refactor `settings.py` to remove all hard-coded fallback defaults — every setting must come from an environment variable (ConfigMap in Kubernetes or GitHub Actions env in CI). `.settings.json` remains a local-dev override mechanism (#2067).
* CI: use valid UUIDs for `DEPLOYMENT_ID` to prevent UUID parse errors.
* Migration: widen `user_log.action` from VARCHAR(19) to VARCHAR(32) to fit `subscription_*` actions.
* Tests: isolate flaky tests from xdist parallel pollution (unique subject / topic names, real `UserRepository` in `ConfigSyncService` tests).


# v0.1.0-alpha.43 17/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-43



***

**Description**: Tests now hit the test database, not dev; restore safe defaults.

### Changes

* Parse the `CI` environment variable as a boolean — `CI="false"` was previously truthy and made tests run against the `data_plane` dev database (#2079).
* Restore safe defaults for `DEBUG`, `CI` and `ALLOW_DESIGN_LOGIN` to prevent `KeyError` when those env vars are absent.
* Drop `DATA_PLANE_FRONTEND_URL` in favour of `INGRESS_URL`.


# v0.1.0-alpha.44 17/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-44



***

**Description**: Restore default value for `DB_DRIVER`.

### Changes

* Restore the default value for `DB_DRIVER` so the data-plane boots without explicit configuration (#2083).


# v0.1.0-alpha.45 22/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-45



***

**Description**: Use env retention\_configuration instead of DEFAULT\_KAFKA\_\* settings.

### Changes

* Use env retention\_configuration instead of DEFAULT\_KAFKA\_\* settings (#2075).
* Webhook Target (#2066).
* Datamodel - fetch schema from source DB (#2077).
* Replace completed manual plan with three focused QA plans (#2081).
* Less verbose logs (#2091).
* Behavior drift execution — rewrite 101 test-cases +       close plan (#2097).
* Snowflake app cp/dp (#2100).
* Pipeline mode off (#2115).
* Add popsink connect to devspace + switching as400 from kafka connect to popsink connect (#2116).


# v0.1.0-alpha.46 23/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-46



***

**Description**: Incremental-load trigger endpoint.

### Changes

* Add an incremental-load trigger endpoint, including the missing service for external access (#2128).
* Image bump folded in from `0.1.0-alpha.45` (#2124).


# v0.1.0-alpha.47 23/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-47



***

**Description**: Fix keystore generation in `jdbc-target` and `popsink-connect`.

### Changes

* Fix keystore generation for `jdbc-target` and `popsink-connect` (#2139).


# v0.1.0-alpha.48 23/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-48



***

**Description**: Unique consumer group id per cluster in `metrics-exporter`.

### Changes

* Ensure a unique Kafka consumer group id per cluster in `metrics-exporter`, with minor tweaks and additional logging (#2143).


# v0.1.0-alpha.49 23/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-49



***

**Description**: Dev: full mTLS for Tansu via Vault + Istio Gateway.

### Changes

* Bring the dev environment in line with production by adding mTLS for Tansu (#2145):
  * Deploy Vault (bank-vaults operator) with cfssl-generated PKI: root CA + intermediate CA imported via devspace hooks.
  * Add Istio (base, `istiod`, gateway) with an mTLS listener on port 9092.
  * Add `cert-manager` + `vault-issuer` `ClusterIssuer` to auto-issue the gateway server cert.
  * Add an hourly CRL fetcher CronJob plus an `EnvoyFilter` with `only_verify_leaf_cert_crl` to enforce revocation.
  * Wire data-plane, `metrics-exporter`, `kafka-to-jdbc` and `popsink-connect` dev entrypoints to consume the new SSL certs.
  * Run a second `kafka-plain` Tansu deployment so Karapace and `kafka-ui` can keep using plain TCP.


# v0.1.0-alpha.5 02/02/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-5



***

**Description**: Snowflake target part 2.

### Changes

* Snowflake target part 2 (#1368).
* Add missing variables (#1373).


# v0.1.0-alpha.50 24/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-50



***

**Description**: popsink-connect: configuration cleanup, better shutdown, native error reporting.

### Changes

* Fix misconfiguration in `application.properties`, improve shutdown behaviour, and detect errors more reliably in `HeartbeatService`; better stacktrace info in native mode (#2154).


# v0.1.0-alpha.51 24/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-51



***

**Description**: popsink-connect: AS/400 incremental-load configuration.

### Changes

* Configure the unusual AS/400 incremental-load path correctly in `popsink-connect` (#2157).


# v0.1.0-alpha.52 27/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-52



***

**Description**: Correct config.

### Changes

* Correct config (#2158).


# v0.1.0-alpha.53 27/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-53



***

**Description**: Raw-message support.

### Changes

* Add raw-message support to the data-plane (#2179).
* Configuration correction folded in from `0.1.0-alpha.52` (#2158).


# v0.1.0-alpha.54 28/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-54



***

**Description**: Service-Principal authentication.

### Changes

* Add Service-Principal authentication, including a missing import fix (#2183).


# v0.1.0-alpha.55 28/04/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-55



***

**Description**: popsink-connect: resume incremental load on restart; pause / resume / stop-snapshot endpoints.

### Changes

* Upgrade Debezium (IBMi) and add pause / resume / stop-snapshot endpoints so popsink-connect can resume an incremental load on restart (#2198).


# v0.1.0-alpha.56 04/05/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-56



***

**Description**: Fix secret reference in data plane chart template.

### Changes

* Fix secret reference in data plane chart template (#2199).
* User deletion FK constraint on service\_account\_user (#2189) (#2201).
* Qa(ux-plan): Execute UX Test plan (#2222).
* API-source connector types with front forms (#2096) (#2238).
* Stop exposing broker credentials via /api/envs/ (#2232).
* Worker decomposition, typed config, persistent state, observability (#2239).
* Real credential check, API-driven tables, incremental on hs\_lastmodifieddate (#2240).
* Real credential check, API-driven tables, incremental on update\_time (#2241).
* Real credential check, API-driven tables, incremental on updated\_at, inv src.zd cheat-sheet (#2242).
* Hide unmanaged providers from selection (#2245).


# v0.1.0-alpha.57 04/05/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-57



***

**Description**: Standalone connector: healthcheck and websocket fix.

### Changes

* Fix the healthcheck and websocket on the standalone connector (#2257).
* Image bump folded in from `0.1.0-alpha.56` (#2253).


# v0.1.0-alpha.58 05/05/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-58



***

**Description**: Bump Kafka for Oracle.

### Changes

* Bump the Kafka image for the Oracle path (#2265).


# v0.1.0-alpha.59 06/05/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-59



***

**Description**: Support `existingSecret` for ingress TLS.

### Changes

* Allow specifying an `existingSecret` for ingress TLS; truncate long names to avoid Kubernetes errors (#2280).


# v0.1.0-alpha.6 03/02/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-6



***

**Description**: Add missing envvars in chart (again).

### Changes

* Add missing envvars in chart (again) (#1374).


# v0.1.0-alpha.60 12/05/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-60



***

**Description**: Use Rust NativeMapper in backend SMT route (#2197).

### Changes

* Use Rust NativeMapper in backend SMT route (#2197) (#2200).
* Snowflake target fix (#2287).
* Require Schema Registry credentials on adapter init (#2262).
* Better logging and error handling (#2293).
* UI: Test credentials button hidden below the fold + error table list does not auto-refresh (#2305).
* DP - New DM page details (#2278).
* Language and copy cleanup across CP and DP (#2290).
* Domain creator as owner and not member (#2292).
* Dead navigation affordances: href="#" links and non-clickable DP logo (#2296).
* Fix replay, fix test mapper (#2310).
* Type handle iceberg (#2309).


# v0.1.0-alpha.61 18/05/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-61



***

**Description**: Upgrade dependancies.

### Changes

* Upgrade dependancies (#2312).
* Filter inaccessible envs from navigation breadcrumb (#2297).
* Connector wizards (sources & targets): required fields and empty-input button gating (#2313).
* CP & DP - Front fixes (#2307).
* ValueFrom references to existing K8s secrets for connector fields (#2308).
* K8s requests/limits per worker resource tier (#2316).
* Few fixs for az 18 (#2328).


# v0.1.0-alpha.62 19/05/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-62



***

**Description**: Connector check-credentials routes now declare typed bodies.

### Changes

* Connector check-credentials routes now declare typed bodies (#2335).
* Bump urllib3 to 2.7.0 and devalue to 5.8.1 for CVE fixes (#2337).
* SMT errors mention the full path, not just the leaf segment (#2333).
* Accept empty hostname in Ingress (#2349).


# v0.1.0-alpha.63 28/05/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-63



***

**Description**: Member invitation: no success toast and no Pending badge on the new row.

### Changes

* Member invitation: no success toast and no Pending badge on the new row (#2343).
* Profile on new tab for dp only (#2357).
* Add oracle source (#2356).
* DP — multiple UI/UX fixes (v2) (#2282).
* Use Rust NativeMapper in workers (#2317).
* Superuser impersonation (login as another user) (#2330).
* Mapper redesign Front issue (#2322).
* Incremental load single table (#2355).
* Dedupe singleton background tasks via PG advisory lock (#2323).
* QA: connector refacto audit — 9 test-cases updated to match current UI (#2369).
* Arch amd64 (#2406).
* Silent success-like redirect after 403 on connector delete and team leave (#2363).
* Hide destructive connector actions for non-admin members (#2365).
* Promoting env-admin in CP does not sync team\_member.admin in DP (#2370).
* Last owner can leave a domain, creating an orphan with no UI recovery path (#2376).


# v0.1.0-alpha.64 29/05/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-64



***

**Description**: Dp make schema registry credentials optional.

### Changes

* Dp make schema registry credentials optional (#2422).


# v0.1.0-alpha.65 29/05/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-65



***

**Description**: Make schema registry creds trully optionals.

### Changes

* Make schema registry creds trully optionals (#2429).


# v0.1.0-alpha.66 01/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-66



***

**Description**: Worker pods stuck Pending on Apple Silicon.

### Changes

* Worker pods stuck Pending on Apple Silicon (#2430).
* To test (#2434).


# v0.1.0-alpha.67 02/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-67



***

**Description**: Front logs display fixs.

### Changes

* Front logs display fixs (#2435).
* Restructure smoke into per-module test blocks + Daily Slack (#2412).
* Include data plane version in heartbeat (#2415).
* Event detail infos (#2426).
* Delta/Databricks connector (#2364).
* CP/DP Login with json web token (#2438).
* CVE 01 06 2026 (#2460).
* Add credentials proxy (#2463).


# v0.1.0-alpha.68 02/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-68



***

**Description**: Flatten Debezium envelope before mapper transform.

### Changes

* Flatten Debezium envelope before mapper transform (#2449).
* Proxy only on credentials (#2466).


# v0.1.0-alpha.69 02/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-69



***

**Description**: Sql server source.

### Changes

* Sql server source (#2472).


# v0.1.0-alpha.7 03/02/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-7



***

**Description**: Add missing envvars in chart again.

### Changes

* Add missing envvars in chart again (#1378).


# v0.1.0-alpha.71 04/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-71



***

**Description**: Migrate from kafka UI to kotatsu.

### Changes

* Migrate from kafka UI to kotatsu (#2468).
* Enable thick mode on demand for source/target (#2488).
* Worker images missing popsink\_smt in production (#2479).
* Tune client timeouts for high-latency (S3-backed) broker (#2486).


# v0.1.0-alpha.72 08/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-72



***

**Description**: Flatten Debezium envelope in mapper tester autocomplete.

### Changes

* Flatten Debezium envelope in mapper tester autocomplete (#2487).
* Per-connector NodeSelector injected into the worker manifest (#2502).
* Flatten Debezium envelope in datamodel schema tab (#2508).
* DP - Multiple fixes (#2503).
* Wire SaaS provisioner end-to-end in dev kind (#2410).
* Heartbeat loop interval 20s -> 600s (10 minutes) (#2522).
* Qa Subscription Analysis (#2446).
* Canonicalize config hash so drift detection converges (#2523).
* Stop WebSocket reconnect storm with expired JWT (#2524).
* Stop logging full config (member emails / PII) every cycle (#2525).
* Fix schema (#2548).
* In popsink connect, BigDecimal and BigInteger need to be added to serialization-config.json for incremental load (#2549).
* Optional dbo (#2545).


# v0.1.0-alpha.73 08/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-73



***

**Description**: Point dead Subscriptions breadcrumb to parent target tab.

### Changes

* Point dead Subscriptions breadcrumb to parent target tab (#2513).
* Add a list of typical column types in serialization-config.json, bump versions (#2553).


# v0.1.0-alpha.74 08/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-74



***

**Description**: JVM serialization into quarkus requires handling arrays of typical types too.

### Changes

* JVM serialization into quarkus requires handling arrays of typical types too (#2555).


# v0.1.0-alpha.75 09/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-75



***

**Description**: Per-card skeleton loading state for dashboard stats.

### Changes

* Per-card skeleton loading state for dashboard stats (#2557).
* Revert pop-alert design, apply to snackbar instead (#2551).
* Remove metrics exporter (#2598).
* Snowflake target mapper config (#2591).


# v0.1.0-alpha.76 11/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-76



***

**Description**: Migrate workers to env-only credentials.

### Changes

* Migrate workers to env-only credentials (#2556).
* \[IBM i] Incremental load: status, cancel & concurrency guard (#2592) (#2603).
* Trigger a blocking snapshot for source connectors (connector + data model scope) (#2610).
* Worker heartbeat 422: state field carries reason, backend expects bare enum (#2507).
* Do not auto-trigger the credentials test when opening the connector edit modal (#2607).
* Connector edit must not be able to remove whitelist topics (additive only; removal solely via data-model deletion) (#2609).
* Allow changing a pipeline's target connector (recreates subscriptions) (#2632).
* Source page: backfill on add, fix table delete, restore Subscribe all (#2638).
* Tidy up provisioner (#2637).


# v0.1.0-alpha.77 11/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-77



***

**Description**: Fix pipeline subscription using wrong source datamodel.

### Changes

* Fix pipeline subscription using wrong source datamodel (#2634).
* Backend: restart worker + reliable deferred backfill on table add (#2640).
* Migrate kafka\_to\_kafka\_target / kafka\_to\_oracle / kafka\_to\_webhook / snowflake\_to\_kafka to env-only credentials (#2611).
* Remove kafka\_connect worker; migrate Postgres/MySQL to popsink-connect (#2622).
* Google Ads OAuth — generic, plug-and-play OAuth plumbing (#2511).
* Subscription logs tab: replace infinite "Loading logs" with a clear empty state (#2645).
* Switch to popsinktansu (#2657).


# v0.1.0-alpha.78 11/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-78



***

**Description**: Version 11 06 2026.

### Changes

* Version 11 06 2026 (#2655).


# v0.1.0-alpha.79 11/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-79



***

**Description**: Impl.

### Changes

* Impl (#2662).


# v0.1.0-alpha.8 03/02/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-8



***

**Description**: Add config sync.

### Changes

* Add config sync (#1314).
* Add missing envvars in chart 3 (#1381).


# v0.1.0-alpha.80 12/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-80



***

**Description**: Snow target fix 2.

### Changes

* Snow target fix 2 (#2665).


# v0.1.0-alpha.81 15/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-81



***

**Description**: TBD.

### Changes

* *No tracked changes detected for this release.*


# v0.1.0-alpha.82 15/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-82



***

**Description**: Default config mapper.

### Changes

* Default config mapper (#2694).
* Fix and bump (#2698).


# v0.1.0-alpha.83 15/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-83



***

**Description**: URL per step in pipeline creation flow.

### Changes

* URL per step in pipeline creation flow (#2678).
* Server-side pagination on detail pages (#2704).
* Per-subscription consumption metrics + ingest auth fix (#2703).
* Honor adls\_endpoint in Unity Catalog pre-flight (#2711).


# v0.1.0-alpha.84 15/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-84



***

**Description**: Update dbz in ibmi\_source, activate IT.

### Changes

* Update dbz in ibmi\_source, activate IT (#2718).


# v0.1.0-alpha.85 16/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-85



***

**Description**: Remove backend add-table backfill machinery.

### Changes

* Remove backend add-table backfill machinery (#2720).
* Inject env-level hostAliases into worker pods (#2721).
* Emit per-topic production metrics (#2714).
* Vendor kotatsu reader wheel into prod image (#2728).
* Current image (#2723).
* Async kotatsu S3-direct reader adapter (#2729).
* Collapse SQL Server source scan N+1 into one query (#2731).
* Bump popsink connect (#2732).


# v0.1.0-alpha.86 16/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-86



***

**Description**: Version mismatch.

### Changes

* Version mismatch (#2733).


# v0.1.0-alpha.87 16/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-87



***

**Description**: Bound lock\_timeout so a stuck lock can't gridlock a deploy.

### Changes

* Bound lock\_timeout so a stuck lock can't gridlock a deploy (#2712).
* No escape (#2738).


# v0.1.0-alpha.88 16/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-88



***

**Description**: Fix oracle source.

### Changes

* Fix oracle source (#2743).


# v0.1.0-alpha.89 16/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-89



***

**Description**: Removing k8s registry from extraenvvars.

### Changes

* Removing k8s registry from extraenvvars (#2740).


# v0.1.0-alpha.9 06/02/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-9



***

**Description**: First tracked chart release. Auto team creation fix and ConfigSync improvements.

### Changes

* Fix auto team creation: enhanced environment synchronization and user management in `SyncEnvsService` (#1403).
* Refactor: assert-based user repository check in environment lazy loading.
* Bump component image tags to `2026.02.06`.


# v0.1.0-alpha.90 16/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-90



***

**Description**: MTLS broker SSL + real SMT transform (Avro, shared mapper, typed/dynamic Delta schema).

### Changes

* MTLS broker SSL + real SMT transform (Avro, shared mapper, typed/dynamic Delta schema) (#2749).
* IBMi whitelist "\$" preservation + Unity Catalog table-exists idempotency (#2761).


# v0.1.0-alpha.91 17/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-91



***

**Description**: Use timezone-aware datetimes in heartbeat staleness/reconciliation.

### Changes

* Use timezone-aware datetimes in heartbeat staleness/reconciliation (#2717).
* Replace Karapace with Kora schema registry (prod) (#2176).


# v0.1.0-alpha.92 18/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-92



***

**Description**: Navigation exploratory drift report + macOS playwright-cli config.

### Changes

* Navigation exploratory drift report + macOS playwright-cli config (#2782).
* \[2391/2395/2397/2398/2399] feat(targets): emit per-subscription consumption metrics (#2747).
* Unity Catalog target - pre-flight checks for External Data Access + external-table grants (#2775).
* \[2392/2393/2396/2400] feat(connectors): emit per-topic metrics from python workers (#2741).
* Publish a versioned OpenAPI spec per release (#2750).
* Kafka to jdbc refactor into module (#2785).
* Refac(metrics): rename Metric to ProductionMetric (#2787).
* Bump UC (#2764).
* Resolve Dependabot Python CVEs in data-plane, provisioner & connectors (#2777).
* Resolve npm Dependabot CVEs — nuxt 4.4.7, vite 7.3.5, ws 8.21.0 (#2780).
* Expose datagen connector configuration knobs (#2700).
* Bump versions of kafka\_to\_jdbc (#2794).


# v0.1.0-alpha.93 19/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-93



***

**Description**: An old workaround was causing an issue in yarn.

### Changes

* An old workaround was causing an issue in yarn (#2799).
* Fixkube rust build context smt rs dropped by 2785 inv upsetupkubeingest fails to build kafka to unity catalog (#2800).
* Full bump (#2804).


# v0.1.0-alpha.94 21/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-94



***

**Description**: IBMi: capture tables across multiple libraries sharing one journal (data-plane).

### Changes

* IBMi: capture tables across multiple libraries sharing one journal (data-plane) (#2814).


# v0.1.0-alpha.95 22/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-95



***

**Description**: Find a gradle workaround to let multiple pod run.

### Changes

* Find a gradle workaround to let multiple pod run (#2809).
* DP - Clickable labels — Redirect to resource (#2705).
* IBMi: scale Debezium task-start timeout with captured table count (#2817) (#2819).
* Fix subscriptions/sync 500 on valid payloads: coerce None optionals (#2811) (#2821).
* Fix GET /subscriptions/?source\_connector\_id 500: drop duplicate datamodel join (#2813) (#2823).
* Fix POST /subscriptions/ ignoring the backfill field on creation (#2812) (#2822).
* Oracle: bound buffered LogMiner memory to survive streaming catch-up (#2818) (#2820).
* Allow scoping connector snapshot triggers to a list of tables (#2816) (#2824).


# v0.1.0-alpha.96 22/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-96



***

**Description**: Subscription detail view does not display the mapper config.

### Changes

* Subscription detail view does not display the mapper config (#2783).
* Emit task-start timeout under the internal. key the engine reads (#2817) (#2834).


# v0.1.0-alpha.97 23/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-97



***

**Description**: Format quarkus/log4js logs correctly and add colors.

### Changes

* Format quarkus/log4js logs correctly and add colors (#2836).
* Update tests with ibmi multi-lib and blocking snapshot (#2833).
* Make the automatic initial snapshot trackable (#2827) (#2855).
* Expose canonical config\_hash for drift detection (#2795).
* In popsink connect, fix ibm i decimal numberformatexception and add DLQ (#2858).


# v0.1.0-alpha.98 23/06/2026
Source: https://docs.popsink.com/changelog/data-plane/alpha-98



***

**Description**: Tune provisioner some more.

### Changes

* Tune provisioner some more (#2862).


# v1.78 08/02/2024
Source: https://docs.popsink.com/changelog/legacy/v1.78



***

**Description**: hubspot, bigquery, kafka

## Features

* Hubspot source connector : Choose your resources
* Hubspot source connector : Initial load
* BigQuery target connector : History table and Upsert materialized view
* Bring your own Kafka

### Fix

* Dashboard data models search


# v1.83 18/03/2024
Source: https://docs.popsink.com/changelog/legacy/v1.83



***

**Description**: avro, ssh, hubspot

### Features

* Messages are now Avro encoded
* Frontend updates
* Mysql source connector with SSH tunnel
* Better HubSpot source initial load


# v1.84 03/04/2024
Source: https://docs.popsink.com/changelog/legacy/v1.84



***

**Description**: clickhouse

### Features

* Introducing ClickHouse target connector
* Frontend fixes
* Fix on data models statistics


# v3.2302 01/10/2024
Source: https://docs.popsink.com/changelog/legacy/v3.2302



***

**Description**: clickhouse, atio, hubspot, mssql, oracle, ibm

## Changelog Summary

### New Features & Enhancements:

* Added parent record support to Attio (#4183)
* HubSpot association initial load (#4199)
* Salesforce push topic support (#4290)
* Added migration process for new database (#4310)
* Added retry and error handling improvements for HubSpot (#4477, #4483)
* Implemented Clickhouse target (#4573)

### Onboarding Improvements:

* Fixed onboarding issues, including OAuth connector handling (#4191)
* Trigger backfill during onboarding (#4225)

### Bug Fixes:

* Fixed MSSQL issues and configurations (#4219, #4251)
* Resolved HubSpot token and target build problems (#4441, #4509)
* Fixed Airtable source integration (#4317, #4478)
* Addressed multiple Salesforce-related issues (#4412, #4498)
* Fixed soft deletion renaming (#4582)

### Performance & Optimization:

* Improved connectors list and subscription detail requests (#4301, #4292)
* Enhanced e2e tests (#4155, #4316)
* Optimized Kafka-connect images and avoided IBM-related issues (#4556)

### Refinements:

* Unified manifests labels (#4198)
* Improved error messaging and user experience (#4376)
* Standardized source/target data tables (#4544)

### Version Updates:

* Multiple version bumps for stability and feature releases (#4261, #4341, #4527)
* Released versions: `0.3.229`, `0.3.2298`, `0.3.2299`, `0.3.2300`, `0.3.2301`

### Miscellaneous:

* Improved logging and monitoring systems (#4169, #4418)
* Fixed email UTF-8 encoding issues (#4431)
* Preload schema webhook slack (#4517)

This summary includes major feature releases, critical fixes, and version updates based on the commits.


# v3.24 14/10/2024
Source: https://docs.popsink.com/changelog/legacy/v3.24



***

**Description**: connectors

## Changelog Summary

### Features:

* **Connectors:**
  * Added search filter functionality (#4568)
  * Added a category feature to target/source connectors (#4598)
  * New target support added (#4506)
* **Datamodel:**
  * Added a detailed usage tab in the data model (#4269)
  * Refresh logic implemented for data model list in connector detail (#4554)
* **Onboarding:**
  * Added notification for onboarding (#4554)
  * Removed "About" link during onboarding (#4577)
* **Region Management:**
  * Request region functionality added (#4551)
* **Whitelisting:**
  * Added/removed table whitelisting (#4547)
  * Fixed whitelisting issue (#4594)

### Bug Fixes:

* Fixed issue with visible existing popsink tags on update (#4551)
* Replay initialization load issue fixed for subscriptions (#4612)

### Updates:

* General config updates (#4619)

### Release:

* Version 0.3.24 released for production (#4614)


# v3.29 13/11/2024
Source: https://docs.popsink.com/changelog/legacy/v3.29



***

**Description**: hubspot, salesforce, bigquery, snowflake

## Changelog Summary

### Features:

* **Connectors:**
  * Added group ID support (#4624).
  * Introduced lifecycle management for connectors, datamodels, subscriptions, and schemas (#4841).
  * Added support for Snowpipe streaming and schema evolution (#4859).
  * Enabled popsink source domain (#4953).
  * Implemented single subscription pause/play (#4989).
  * Added admin commands for managing connectors, datamodels, and schemas (#4992, #5035).
  * Configured one consumer per source (#5032).
* **Integrations:**
  * Enhanced HubSpot functionality with retry on 429 errors and support for unique property creation (#4890, #4820).
  * Improved BigQuery target logging (#4807).
  * Service Account BQ experience updated (#4973).
  * Bulk API introduced for Salesforce (#4727).
* **UI/UX Improvements:**
  * Redesigned usage page (#4550).
  * Updated initial load button design (#4692).
  * New loading experience introduced (#4952).
  * Updated environment creation flow (#4974).
* **Admin & Worker Enhancements:**
  * Added worker configuration lifecycle (#4878).
  * Improved logs and added worker logging (#4750, #4943).
  * Added support for user-configurable workers (#4747).
* **Other Features:**
  * Free Trial extended to 30 days (#4812).
  * Enabled blocking of consumer start and improved front status display (#5027).

### Fixes:

* Fixed multiple HubSpot issues, including object IDs and custom object paths (#4627, #4819, #4824).
* Resolved Salesforce deletion and API issues (#4844).
* Addressed Snowflake source issues (#4834).
* Improved schema pool handling and avoided ID conflicts on new versions (#5011).
* Fixed out-of-memory errors by updating memory limits (#5008).
* Handled 403 loops and signout conditions (#5042, #5000).
* Fixed nullable values in datagen source (#4764).
* Corrected typos and improved naming conventions (#4919, #4633).
* Fixed mapping and divider configurations (#5018, #4869).


# v3.7 29/01/2025
Source: https://docs.popsink.com/changelog/legacy/v3.7



***

**Description**: connectors, ux, ui, BigQuery, Snowflake, MongoDB, HubSpot, Salesforce, Slack, Airtable, Oracle, MSSQL, JDBC, and Attio

## Changelog Summary

#### Connectors & Integrations

* Fixes and improvements for **BigQuery, Snowflake, MongoDB, HubSpot, Salesforce, Slack, Airtable, Oracle, MSSQL, JDBC, and Attio** connectors.
* Improved **Popsink, Webhook UX, and HubSpot logging**.
* Support for **Stripe payments and portals**.

#### **Performance & Optimization**

* **Fixed N+1 query issues** to optimize performance.
* More efficient **subscription management and caching**.
* Async batch handling and recursion fixes.

#### **Frontend & UX Enhancements**

* Improved **mapping UI, subscription status display, and error pages**.
* Fixed **icons, colors, labels, and button behaviors**.
* Auto-reload logs and scrollable panels for connector steps.
* Updated **multi-error alerts and v-autocomplete for invites**.

#### **Bug Fixes & Miscellaneous**

* Fixed **event names, query parameters, initialization failures, and credential component issues**.
* Improved **tablelist updates, retry handling, and external service mocks**.
* Prevented tombstone records in JDBC and fixed nullable Avro types.
* 403 handling refined (no forced signout).


# v3.8 25/03/2025
Source: https://docs.popsink.com/changelog/legacy/v3.8



***

**Description**: OAuth & CVE security fixes, Snowflake, Clickhouse async, MongoDB, Teradata, Databricks improvements, Webhooks & connectors fixes, N+1 optimizations, Nuxt 3.16 migration, better logging & retries, Attio & HubSpot integrations, usage dashboard, production stability updates.

## Changelog Summary

#### **Authentication & Security**

* Improved **OAuth token handling** (code token and extra attributes).
* **Critical CVE fix** for frontend security.
* Updated **SSL handling for Aiven** and **Snowflake precision settings**.
* Removed old environment variables and **Mercure**.

#### **Database & Backend Improvements**

* **Improved Snowflake, MongoDB, Clickhouse, and Redpanda** integration.
* **Clickhouse now fully async** for better performance.
* **Incremental load support** added.
* **Multi-primary key support** for workers.
* **PostgreSQL optimizations** (fewer tombstones, better identifier management).
* Fixes for **JDBC memory, IBM i, and Teradata targets**.
* **Optimized worker status updates and logging**.

#### **Connectors & Integrations**

* **Bug fixes and enhancements** for **BigQuery, HubSpot, Salesforce, Stripe, Webhooks, Airtable, MSSQL, Oracle, and JDBC**.
* **Dedicated deployment for Attio webhooks** and **Attio BigQuery integration**.
* **Databricks and Teradata targets introduced**.
* **Salesforce source fixes**.
* **Better handling of schema changes** for Webhooks and connectors.

#### **Performance & Optimization**

* **N+1 query fixes and optimizations** across backend.
* **Retries and timeout handling improved** for production stability.
* **Default retention time** adjustments for optimized storage.
* **Snowflake updates** to avoid dashes in table names.

#### **Frontend & UX Enhancements**

* **Multiple UI fixes**, including **subscription details, button behaviors, and icon updates**.
* **Nuxt 3.16 migration** for frontend framework.
* **Error messages now more specific and informative**.
* **Improved redirects when backend is down**.

#### **Miscellaneous & Other Fixes**

* **Webhook target and source fixes**.
* **SSH tunnel improvements** for IBM i.
* **Cron usage enhancements**.
* **Better logging and debugging**:
  * **More informative logs** with prefixes.
  * **Removed unnecessary hot reload** in Gunicorn.
* **Leftover invoke task removed**.
* **Fixes for load visibility and resource initialization**.


# Application Source
Source: https://docs.popsink.com/connectors/source/application



The Application Source Connector allows you to directly send data to Popsink using Python, Node, Go, Java or other programming languages. It is the most efficient way to build a direct integration between your application and Popsink. Start by creating a connector then you will be able to register the topics you wish to send data to.

## Compatible versions

|                        |                                                                                              |
| ---------------------- | -------------------------------------------------------------------------------------------- |
| **Protocol**           | Kafka wire protocol — any client library (Python, Node.js, Go, Java, …)                      |
| **Client requirement** | Kafka client supporting `SASL_SSL` and API version negotiation (Kafka 2.1 protocol or later) |
| **Schema registry**    | Confluent-compatible Schema Registry API                                                     |
| **Transport**          | TLS 1.2 or later                                                                             |


# Attio Source
Source: https://docs.popsink.com/connectors/source/attio



The Attio Source Connector integrates with Attio, a modern Customer Relationship Management (CRM) platform designed for data-driven teams. Attio offers a flexible and customizable approach to manage customer data, interactions, and relationships. It allows users to create custom objects, fields, and workflows to match their specific business processes. The connector enables real-time synchronization of data from Attio to your data pipeline, facilitating analytics, reporting, and integration with other business systems. This connector is particularly useful for companies looking to leverage their CRM data for advanced analytics, customer insights, and cross-functional collaboration.

## Compatible versions

|                    |                                                |
| ------------------ | ---------------------------------------------- |
| **Attio API**      | API v2                                         |
| **Authentication** | OAuth 2.0                                      |
| **Objects**        | Standard and custom objects exposed by the API |

## Prerequisites

* Attio Account with Connection Privileges: Ensure you have an active Attio account with the necessary permissions to access and replicate data.

## Connecting Attio

In the source selection menu, select the Attio connector and click the "Connect" button.

![Screenshot 2024-07-31 205255](https://github.com/user-attachments/assets/64d88815-d2a4-4068-9863-905d17577b80)

A new window will open, prompting you to log in to your Attio account.
After logging in, grant the necessary permissions to allow Popsink to access your Attio data.

![Screenshot 2024-07-31 205319](https://github.com/user-attachments/assets/9845f0ee-9718-493b-a330-1eaad30659bd)

Once authenticated, you will be able to give your connector a name and select the Attio objects you wish to track.
In the "Table selection", you will be presented with a list of available objects in your Attio CRM.
Select the objects you wish to replicate to Popsink.
Once selected, click "Create connector"

![Screenshot 2024-07-31 205346](https://github.com/user-attachments/assets/95cbebcb-7092-40dd-9b74-adf9a98c4bce)

Your connector is ready! You can now use the "Subscribe to all" button to sync all objects from the connector with the destination of your choice.

![image](https://github.com/user-attachments/assets/3e72aa1a-d8a9-47e6-a35b-2015d4970f64)

## Key Features

* Real-time Data Synchronization: The Attio CRM Connector ensures real-time data synchronization, capturing changes as they happen in your Attio CRM and reflecting them in your target systems.

* Customizable Object Selection: You can choose specific objects from Attio to replicate, allowing you to tailor the data integration to your business needs.

* Schema Evolution and Compatibility: The connector automatically detects changes in the Attio objects schema and adjusts the data replication process accordingly, ensuring seamless data integration even as your CRM data structure evolves.

* Fault-tolerant and Scalable: Built with fault tolerance and scalability in mind, the Attio CRM Connector can resume data replication from the last known state in case of disruptions, ensuring data consistency and minimal downtime.

* Initial Data Load: The connector performs an initial full data load of the selected objects from Attio, setting up a comprehensive foundation for ongoing synchronization.

## Security

* OAuth 2.0 Authentication: The connector uses OAuth 2.0 for secure and reliable authentication, ensuring your Attio credentials are protected.

## Conclusion

The Attio CRM Connector is a robust solution for integrating your Attio data with Popsink, offering real-time synchronization, scalability, and security. With customizable object selection and advanced filtering options, it adapts to your specific data integration needs, driving enhanced business intelligence and operational efficiency.


# BigQuery Source
Source: https://docs.popsink.com/connectors/source/bigquery



The BigQuery Source Connector streams changes from Google BigQuery tables into Popsink pipelines using the native `CHANGES()` function. It performs an initial full load of each selected table, then polls for incremental changes (inserts, updates, and deletes) every 10 minutes, making your warehouse data available downstream without manual exports.

## Compatible versions

|                      |                                                                                         |
| -------------------- | --------------------------------------------------------------------------------------- |
| **Google BigQuery**  | Managed service — no version to pin                                                     |
| **Required feature** | Table change history (`CHANGES()`) on GoogleSQL standard tables                         |
| **Table types**      | Standard tables only — views, materialized views, and external tables are not supported |
| **API**              | BigQuery API v2                                                                         |

<Note>Change history must be enabled per table (`enable_change_history = TRUE`) and is retained for up to 7 days by default.</Note>

## Key Features

* **Native change history:** uses BigQuery's `CHANGES()` function — no extra infrastructure in your GCP project.
* **Initial Load:** the connector automatically performs an initial full table load, then switches to incremental polling.
* **Schema evolution:** the table schema is re-introspected on every poll cycle, and the stream schema is updated automatically when columns are added or removed.

## Prerequisites

1. A **GCP service account** with BigQuery read permissions on the target dataset (including `bigquery.tables.getData`).
2. **Change history** enabled on the tables you want to replicate:
   ```sql theme={null}
   ALTER TABLE <your_table> SET OPTIONS (enable_change_history = TRUE);
   ```
   BigQuery retains change history for up to 7 days by default.
3. Selected tables should be **standard tables** — views and external tables are not supported.

## Configuration

| Field               | Required | Description                                                                     |
| ------------------- | -------- | ------------------------------------------------------------------------------- |
| **Service Account** | Yes      | GCP service account key in JSON format — paste the full content of the key file |
| **Project**         | Yes      | GCP project ID where the BigQuery dataset lives (e.g. `my-gcp-project-123`)     |
| **Dataset**         | Yes      | BigQuery dataset containing the tables to replicate (e.g. `analytics`)          |

## Table Selection

After entering credentials, select tables from the discovered list. For each selected table the connector:

1. **Initial load:** reads all existing rows.
2. **CDC polling:** every 10 minutes, queries `CHANGES()` for inserts, updates, and deletes.

For details on change history retention and limitations, see the [BigQuery Change Data Capture documentation](https://cloud.google.com/bigquery/docs/change-data-capture).


# Source Database Connectors
Source: https://docs.popsink.com/connectors/source/databases

Browse the Popsink source connectors built for operational databases and analytical warehouses.

The connectors in this family capture change events from structured data platforms. They leverage native CDC capabilities to move data from your systems of record into Popsink pipelines.

> Start with the [source connectors overview](./overview) to understand the shared configuration and security strategies across all integrations.

## Relational databases

* [Postgres](./postgres) – Capture logical replication streams from your Postgres clusters.
* [MySQL](./mysql) – Use MySQL or MariaDB binlogs to feed Popsink continuously.
* [MSSQL](./mssql) – Consume Microsoft SQL Server Change Tracking to follow every update.
* [Oracle](./oracle) – Replicate Oracle redo logs without impacting transactional workloads.
* [IBM i](./ibmi) – Synchronize DB2 for i entries through the platform's native journaling.

## Data warehouses & analytics

* [Snowflake](./snowflake) – Subscribe to Snowflake changes using dedicated tasks and streams.
* [BigQuery](./bigquery) – Capture BigQuery changes to power your real-time analytics pipelines.

## Document databases

* [MongoDB](./mongodb) – Track MongoDB operations through change streams.
* [Firestore](./firestore) – Replicate Firestore mutations from Cloud Firestore collections.

Need a database that is not listed? Contact us at [support@popsink.com](mailto:support@popsink.com) to discuss a custom integration.


# Datagen Source
Source: https://docs.popsink.com/connectors/source/datagen



The Datagen Source Connector provide a practical and efficient way to create synthetic data for testing. It acts as a data generator that creates artificial data based on the specific schema that you define.

## Compatible versions

|                     |                                            |
| ------------------- | ------------------------------------------ |
| **External system** | None — events are generated inside Popsink |
| **Schema**          | Avro schema you define on the connector    |

<Note>There is nothing to version on your side: this connector has no external dependency.</Note>


# Eventhub Source
Source: https://docs.popsink.com/connectors/source/eventhub



The EventHub Source Connector integrates with Azure Event Hubs, a highly scalable, real-time data streaming and event ingestion Platform-as-a-Service (PaaS) offered by Microsoft Azure. It provides a distributed, fault-tolerant, and highly available messaging service that can handle millions of events per second, making it suitable for big data, analytics, and real-time processing scenarios.

## Compatible versions

|                      |                                                        |
| -------------------- | ------------------------------------------------------ |
| **Azure Event Hubs** | Standard, Premium, and Dedicated tiers                 |
| **Protocol**         | Kafka wire protocol endpoint (port `9093`, `SASL_SSL`) |
| **Authentication**   | SAS connection string or Microsoft Entra ID            |

<Note>The Basic tier does not expose the Kafka endpoint and cannot be used.</Note>


# Facebook Ads Source
Source: https://docs.popsink.com/connectors/source/facebook-ads



The Facebook Ads Source Connector ingests advertising data from the Meta Marketing API into Popsink pipelines. It retrieves campaigns, ad sets, and ads on a configurable schedule, enabling marketing analytics, budget tracking, and cross-channel reporting with fresh advertising data.

## Compatible versions

|                        |                                              |
| ---------------------- | -------------------------------------------- |
| **Meta Marketing API** | Graph API v21.0                              |
| **Authentication**     | OAuth 2.0 access token with `ads_read`       |
| **Accounts**           | Ad accounts reachable by the authorised user |

<Note>The Graph API version is pinned by the connector. Meta supports each version for about two years, and Popsink rolls it forward as part of connector upgrades.</Note>

## Key Features

* **Ad structure by default:** ingests **campaigns**, **ad sets**, and **ads** out of the box; the resource list can be customized.
* **Token-based authentication:** uses a Marketing API access token scoped to your ad account.
* **Scheduled ingestion:** flexible scheduling by minutes, hours, or days, with an immediate first run on creation.

## Prerequisites

* A Meta app with the **Marketing API** product added.
* A Marketing API **access token** with the `ads_read` permission for the target ad account.
* Your **ad account ID** (found in Meta Ads Manager).

## Configuration

| Field            | Required | Description                                           |
| ---------------- | -------- | ----------------------------------------------------- |
| **Access Token** | Yes      | Marketing API access token with `ads_read` permission |
| **Account ID**   | Yes      | Ad account ID, with or without the `act_` prefix      |
| **Schedule**     | Yes      | Sync frequency — see [Scheduling](#scheduling)        |

## Scheduling

API source connectors run on a schedule rather than streaming continuously. A first run is always triggered immediately after the connector is created; subsequent runs depend on the selected mode:

| Mode        | Behavior                                                      | Example                                      |
| ----------- | ------------------------------------------------------------- | -------------------------------------------- |
| **days**    | Runs every N days at 00:00 UTC + offset (in minutes)          | Offset 30 → every day at 00:30 UTC           |
| **hours**   | Runs every N hours at :00 + offset (in minutes)               | Offset 15 → 00:15, 01:15, 02:15, …           |
| **minutes** | Runs every N minutes, anchored on the connector creation time | Created 10:03, interval 10 → 10:13, 10:23, … |


# Firestore Source
Source: https://docs.popsink.com/connectors/source/firestore



The Firestore Source Connector allows you to easily ingest from the Firestore databases using a Google Cloud service account and whitelist specific Firestore collections that you want to access. This document will guide you through the necessary steps to get started with the Firestore Connector.

## Compatible versions

|                     |                                         |
| ------------------- | --------------------------------------- |
| **Cloud Firestore** | Native mode                             |
| **API**             | Firestore API v1                        |
| **Authentication**  | Google Cloud service account key (JSON) |

<Note>Datastore mode databases are not supported.</Note>

## Prerequisites

* A Firestore database with existing collections.
* A Google Cloud service account with the appropriate Firestore database permissions and the JSON key file associated with the service account.


# Google Ads Source
Source: https://docs.popsink.com/connectors/source/google-ads



The Google Ads Source Connector ingests advertising data from the Google Ads API into Popsink pipelines. It retrieves campaign and performance data on a configurable schedule, enabling marketing analytics, spend monitoring, and cross-channel attribution with fresh advertising data.

## Compatible versions

|                    |                                     |
| ------------------ | ----------------------------------- |
| **Google Ads API** | v20                                 |
| **Authentication** | OAuth 2.0 with a developer token    |
| **Accounts**       | Manager (MCC) and standard accounts |

<Note>The API version is pinned by the connector and rolled forward as Google retires older versions.</Note>

## Key Features

* **Campaign data by default:** retrieves campaign attributes out of the box; queries can be customized using GAQL (Google Ads Query Language).
* **OAuth2 authentication:** secure access through your own Google Cloud OAuth application.
* **Scheduled ingestion:** flexible scheduling by minutes, hours, or days, with an immediate first run on creation.

## Prerequisites

* A Google Ads account and its **customer ID**.
* A Google Ads API **developer token** (requested from a Google Ads manager account under **Tools & settings → API Center**).
* A Google Cloud project with the Google Ads API enabled and an **OAuth2 client** (client ID and client secret).
* An OAuth2 **refresh token** authorized for the Google Ads account (obtained through the OAuth consent flow with the `https://www.googleapis.com/auth/adwords` scope).

## Configuration

| Field               | Required | Description                                     |
| ------------------- | -------- | ----------------------------------------------- |
| **Customer ID**     | Yes      | Google Ads customer ID (e.g. `123-456-7890`)    |
| **Developer Token** | Yes      | Google Ads API developer token                  |
| **Client ID**       | Yes      | OAuth2 client ID from your Google Cloud project |
| **Client Secret**   | Yes      | OAuth2 client secret                            |
| **Refresh Token**   | Yes      | OAuth2 refresh token authorized for the account |
| **Schedule**        | Yes      | Sync frequency — see [Scheduling](#scheduling)  |

## Scheduling

API source connectors run on a schedule rather than streaming continuously. A first run is always triggered immediately after the connector is created; subsequent runs depend on the selected mode:

| Mode        | Behavior                                                      | Example                                      |
| ----------- | ------------------------------------------------------------- | -------------------------------------------- |
| **days**    | Runs every N days at 00:00 UTC + offset (in minutes)          | Offset 30 → every day at 00:30 UTC           |
| **hours**   | Runs every N hours at :00 + offset (in minutes)               | Offset 15 → 00:15, 01:15, 02:15, …           |
| **minutes** | Runs every N minutes, anchored on the connector creation time | Created 10:03, interval 10 → 10:13, 10:23, … |


# HubSpot Source
Source: https://docs.popsink.com/connectors/source/hubspot



The HubSpot Source Connector ingests CRM data from HubSpot — contacts, companies, deals, tickets, and custom objects — into Popsink pipelines. It runs on a configurable schedule and performs incremental synchronization, so only records updated since the last run are delivered downstream. This is particularly valuable for consolidating customer data, building comprehensive customer profiles, and powering data-driven marketing and sales analytics.

## Compatible versions

|                    |                                                                    |
| ------------------ | ------------------------------------------------------------------ |
| **HubSpot API**    | CRM API v3                                                         |
| **Authentication** | Private app access token                                           |
| **Objects**        | Standard and custom CRM objects exposed by the v3 object endpoints |

<Note>OAuth apps and the legacy API-key authentication are not supported — create a private app instead.</Note>

## Key Features

* **Incremental synchronization:** the connector tracks each object's `hs_lastmodifieddate` cursor across runs and only delivers new or updated records.
* **Selectable CRM objects:** choose which objects to ingest (e.g. `contacts`, `companies`, `deals`); standard and custom objects are discovered automatically.
* **Scheduled ingestion:** flexible scheduling by minutes, hours, or days, with an immediate first run on creation.

## Prerequisites

* A HubSpot account with permission to create private apps.
* A **private app access token**: in HubSpot, go to **Settings → Integrations → Private apps**, create a private app, grant it read scopes for the CRM objects you want to ingest (e.g. `crm.objects.contacts.read`, `crm.objects.companies.read`, `crm.objects.deals.read`), and copy the access token.

## Configuration

| Field            | Required | Description                                    |
| ---------------- | -------- | ---------------------------------------------- |
| **Access Token** | Yes      | The private app access token                   |
| **Schedule**     | Yes      | Sync frequency — see [Scheduling](#scheduling) |

## Scheduling

API source connectors run on a schedule rather than streaming continuously. A first run is always triggered immediately after the connector is created; subsequent runs depend on the selected mode:

| Mode        | Behavior                                                      | Example                                      |
| ----------- | ------------------------------------------------------------- | -------------------------------------------- |
| **days**    | Runs every N days at 00:00 UTC + offset (in minutes)          | Offset 30 → every day at 00:30 UTC           |
| **hours**   | Runs every N hours at :00 + offset (in minutes)               | Offset 15 → 00:15, 01:15, 02:15, …           |
| **minutes** | Runs every N minutes, anchored on the connector creation time | Created 10:03, interval 10 → 10:13, 10:23, … |


# IBMi (AS/400) Source
Source: https://docs.popsink.com/connectors/source/ibmi



The IBMi (AS/400) Source Connector integrates with IBM i, formerly known as IBM AS/400 and iSeries, a powerful and reliable midrange computer system widely used in enterprise environments. This connector enables the extraction and streaming of data from IBMi systems, which are known for hosting critical business applications and databases in industries such as manufacturing, finance, and healthcare. It allows organizations to tap into their legacy IBMi data, including information from DB2 for i databases and various file systems. By connecting IBMi to modern data pipelines, this connector facilitates data modernization efforts, enabling real-time analytics, cross-platform data integration, and the ability to leverage legacy data in cloud-based or hybrid environments.

## Table of Contents

1. [Compatible versions](#compatible-versions)
2. [Key Features](#key-features)
3. [How It Works](#how-it-works)
4. [Core Concepts](#core-concepts)
5. [Prerequisites](#prerequisites)
6. [Configuration](#configuration)
7. [Enabling Journaling](#enabling-journaling)
8. [Granting User Permissions](#granting-user-permissions)
9. [Enabling Incremental Load](#enabling-incremental-load)
10. [Estimating Change Volume](#estimating-change-volume)
11. [Housekeeping](#housekeeping)

## Compatible versions

|                    |                                                                                        |
| ------------------ | -------------------------------------------------------------------------------------- |
| **IBM i (OS/400)** | 7.3, 7.4, 7.5, 7.6                                                                     |
| **Database**       | Db2 for i, as shipped with the OS release                                              |
| **Connectivity**   | JTOpen (JT400) 21.0.4 over the IBM i host servers — nothing installed on the partition |
| **Journals**       | Journaled physical files with `*BOTH` images                                           |
| **CDC engine**     | Debezium 3.5, Popsink build of the Db2 for i connector                                 |

<Note>IBM i 7.2 and earlier are out of IBM support and are not covered. Date and time columns are decoded from `QSYS2.SYSCOLUMNS2`, whose DDS format columns were added in IBM i 7.3 TR11 / 7.4 TR5; below those Technology Refresh levels the decoder assumes the `*ISO` format, so files using `*MDY`, `*EUR` or another DDS format need a TR at or above those levels.</Note>

## Key Features

* **Real-time Change Data Capture (CDC):** the connector reads native IBM i journal binaries directly, capturing and streaming changes (inserts, updates, and deletes) as they occur, with end-to-end latency typically between 50–200 ms from commit to publication.
* **Agentless:** no software needs to be installed on the IBM i system. The connector accesses journals remotely via JT400, keeping SQL engine involvement minimal.
* **Low System Overhead:** lightweight binary parsing keeps CPU impact low on the source partition — typically under 2% at low volumes (\<10k changes/min), \~5% at moderate volumes (\~50k changes/min), and 10–15% at high volumes (100k+ changes/min) — leaving headroom for business-critical workloads.
* **High Throughput:** continuous streaming with exact receiver offset tracking sustains more than 50k changes/sec on mid-range Power9 systems and remains stable under burst load.
* **Initial Load:** the connector automatically performs an initial full table load using JDBC + SQL snapshots.
* **Fault-tolerant:** the connector tracks exact journal receiver offsets and resumes replication from the last known position in case of failures, ensuring data consistency and minimal downtime.

## How It Works

The connector combines two mechanisms:

1. **Snapshots:** initial full loads are performed over JDBC using SQL.
2. **Change capture:** ongoing changes are read directly from the native IBM i journal binaries attached to your physical files, via a remote procedure call (RPC) approach over JT400. Rather than polling journals through SQL, the connector streams raw journal entries continuously and tracks exact receiver offsets, minimizing load on the SQL engine and transferring changes in a compact binary format.

## Core Concepts

* **Journal Receiver:** the physical storage object containing change records with before-and-after data images, metadata, timestamps, and transaction information.
* **Journal:** the database tracking mechanism that records changes to specific database objects, maintains metadata, and points to Journal Receivers.
* **Journal Library:** the repository storing journal definitions and organizing journal objects.
* **Journal Receiver Library:** a separate storage location for Journal Receivers, enabling better organization and maintenance.

## Prerequisites

* The tables you want to ingest must be journaled. [See Enabling Journaling](#enabling-journaling)
* You will need a user with the necessary permissions. [See Granting User Permissions](#granting-user-permissions)
* For incremental load support, a signal table must be created and journaled. [See Enabling Incremental Load](#enabling-incremental-load)
* Network access from Popsink to the IBM i host on the ports listed below. [See Network Access](#network-access)

### Network Access

The connector reaches the IBM i system over JT400, which uses the DB2 for i (DRDA/DDM) listener and the IBM i host servers. Allow outbound TCP traffic from Popsink to the source host on the following ports:

| Port   | Service                             | Purpose                                                   |
| ------ | ----------------------------------- | --------------------------------------------------------- |
| `446`  | DRDA / DDM                          | DB2 for i over TCP/IP — used for JDBC snapshots           |
| `449`  | Server mapper (`as-svrmap`)         | Resolves the host server ports the client must connect to |
| `8471` | Database server (`as-database`)     | SQL and catalog access                                    |
| `8473` | File server (`as-file`)             | Access to journal receiver objects                        |
| `8475` | Remote command server (`as-rmtcmd`) | Runs the journal retrieval program calls                  |
| `8476` | Sign-on server (`as-signon`)        | User profile authentication                               |

<Note>
  If the system is configured for TLS, use `448` for the secure DDM/DRDA listener and the `947x` range (`9471`, `9473`, `9475`, `9476`) instead of `847x` for the host servers.
</Note>

If the IBM i system is only reachable through a bastion host, see [Connecting through an SSH Tunnel](#connecting-through-an-ssh-tunnel).

## Configuration

| Field                | Required | Description                                                                         |
| -------------------- | -------- | ----------------------------------------------------------------------------------- |
| **Host**             | Yes      | Hostname or IP address of the IBM i system                                          |
| **Port**             | —        | The connector uses port `446` for IBM i access (fixed, not exposed as a form field) |
| **User / Password**  | Yes      | IBM i user profile and password                                                     |
| **Schema (Library)** | Yes      | IBM i library containing the tables to replicate (e.g. `MYLIB`)                     |

### Advanced Options

| Field                    | Default | Description                                                                                                |
| ------------------------ | ------- | ---------------------------------------------------------------------------------------------------------- |
| **Initial Load**         | `true`  | Snapshot existing rows before streaming journal changes                                                    |
| **Poll Interval (ms)**   | `2000`  | How often the connector polls for new journal activity                                                     |
| **Unicode Escape**       | `false` | Apply Unicode escape handling for special characters in field names                                        |
| **Transaction Buffered** | `false` | Buffer transactions and emit changes only after commit, avoiding partial or uncommitted rows downstream    |
| **Incremental Load**     | `false` | Enable incremental load via the signal table — see [Enabling Incremental Load](#enabling-incremental-load) |

## Connecting through an SSH Tunnel

If the database is only reachable through a bastion host, enable **SSH Tunnel** and provide:

| Field               | Required | Description                                     |
| ------------------- | -------- | ----------------------------------------------- |
| **SSH Host**        | Yes      | Hostname or IP of the bastion host              |
| **SSH Port**        | Yes      | SSH port (default `22`)                         |
| **SSH User**        | Yes      | User to authenticate as on the bastion          |
| **SSH Private Key** | Yes      | **Base64-encoded** private key for the SSH user |

### Table Selection

After a successful credential check, pick the tables to replicate from the discovered list. Only **journaled** tables can be selected — the list indicates the journaling status of each table. The selection is stored as a comma-separated whitelist.

## Enabling Journaling

To capture changes, the tables you want to track must be journaled. If your tables are already journaled, you can skip to [Granting User Permissions](#granting-user-permissions).

1. **Create a Journal Receiver:**
   ```
   CRTJRNRCV <RECEIVER_LIB>/<RECEIVER_NAME>
   ```

2. **Create a Journal attached to the Receiver:**
   ```
   CRTJRN <JOURNAL_LIB>/<JOURNAL_NAME> <RECEIVER_LIB>/<RECEIVER_NAME>
   ```

3. **Start journaling the tables you want to track:**
   ```
   STRJRNPF <FILE_LIB>/<FILE_NAME> <JOURNAL_LIB>/<JOURNAL_NAME>
   ```

4. **Enable before/after image capture (recommended):**
   ```
   CHGJRNOBJ OBJ((<FILE_LIB>/<FILE_NAME> *FILE)) ATR(*IMAGES) IMAGES(*BOTH)
   ```
   Capturing both before and after images is required to correctly handle updates to Primary Key columns.

## Granting User Permissions

The connector's user profile needs the following authorities. No SQL-level SELECT authority is needed on system UDTFs.

```
GRTOBJAUT OBJ(<JOURNAL_LIB>) OBJTYPE(*LIB) USER(popsink) AUT(*EXECUTE)
GRTOBJAUT OBJ(<JOURNAL_LIB>/*ALL) OBJTYPE(*JRNRCV) USER(popsink) AUT(*USE)
GRTOBJAUT OBJ(<JOURNAL_LIB>/<JOURNAL_NAME>) OBJTYPE(*JRN) USER(popsink) AUT(*USE *OBJEXIST)
GRTOBJAUT OBJ(<RECEIVER_LIB>) OBJTYPE(*LIB) USER(popsink) AUT(*EXECUTE)
GRTOBJAUT OBJ(<RECEIVER_LIB>/*ALL) OBJTYPE(*FILE) USER(popsink) AUT(*USE)
```

In summary, the user requires:

* **\*EXECUTE** authority on the journal library and the receiver library
* **\*USE** and **\*OBJEXIST** authority on the journal object
* **\*USE** authority on the journal receivers

## Enabling Incremental Load

To enable incremental load, the connector uses a signal table:

1. **Create the signal table:**
   ```sql theme={null}
   CREATE TABLE SCHEMA.POP_SIGNAL (
     ID VARCHAR(128) NOT NULL,
     TYPE VARCHAR(32) NOT NULL,
     DATA VARCHAR(2048)
   )
   ```

2. **Journal this table in the same journal as the tracked tables:**
   ```
   STRJRNPF <FILE_LIB>/POP_SIGNAL <JOURNAL_LIB>/<JOURNAL_NAME>
   ```

3. **Grant the user write permissions on this table** (this table only — no write access is needed anywhere else).

## Estimating Change Volume

You can estimate your daily change volume by counting journal entries over the last 24 hours:

```sql theme={null}
SELECT J.OBJECT,
  COUNT(CASE WHEN J.JOURNAL_ENTRY_TYPE = 'PT' THEN 1 END) AS INSERT_COUNT,
  COUNT(CASE WHEN J.JOURNAL_ENTRY_TYPE = 'UP' THEN 1 END) AS UPDATE_COUNT,
  COUNT(CASE WHEN J.JOURNAL_ENTRY_TYPE = 'DL' THEN 1 END) AS DELETE_COUNT
FROM TABLE(QSYS2.DISPLAY_JOURNAL('<JOURNAL_LIB>', '<JOURNAL_NAME>')) AS J
WHERE J.ENTRY_TIMESTAMP >= CURRENT_TIMESTAMP - 24 HOURS
GROUP BY J.OBJECT;
```

## Housekeeping

### Journal Receiver Retention

Journal receivers take up disk space, so it is best practice to delete old ones regularly. There is a tradeoff between retention and the operational resilience of your replication pipeline: longer retention gives you more time to recover from outages before having to resync, while shorter retention takes up less space. We recommend a maximum retention of 7 days, and a minimum of 1 day depending on your requirements.

1. **Delete journal receivers older than 7 days:**
   ```
   DLTJRNRCV JRNRCV(<RECEIVER_LIB>/<RECEIVER_NAME>) DLTOPT(*IGNINQMSG) SELECT(*OLD) RETENTION(7)
   ```
   Wildcard patterns are supported (e.g. `RECEIVER_NAME*`). You can schedule this command periodically using `WRKJOBSCDE`.

2. **Monitor active receivers:**
   ```
   WRKJRNRCV <JOURNAL_LIB>/<JOURNAL_NAME>
   ```

3. **Monitor receiver disk usage:**
   ```
   DSPJRNRCVA
   ```

### Stopping and Restarting CDC

1. **Stop CDC connectivity:**
   ```
   ENDTCPIFC INTERFACE(<CDC_SERVER_IP>)
   ```

2. **Restart CDC connectivity:**
   ```
   STRTCPIFC INTERFACE(<CDC_SERVER_IP>)
   ```

### Disabling Journaling

1. **Stop journaling a table:**
   ```
   ENDJRNPF FILE(<FILE_LIB>/<FILE_NAME>) JRN(<JOURNAL_LIB>/<JOURNAL_NAME>)
   ```


# IBM Z Source Connector
Source: https://docs.popsink.com/connectors/source/ibmz



The IBM Z Source Connector ingests data from IBM Z systems using a **split architecture**: a lightweight **Z Host Agent** that runs inside the mainframe environment and an **Off-host Worker** that runs in Popsink outside the host.

***

## Compatible versions

|                          |                                                                  |
| ------------------------ | ---------------------------------------------------------------- |
| **IBM Z / z/OS**         | Confirmed per deployment                                         |
| **Db2 for z/OS**         | Log capture through the Instrumentation Facility Interface (IFI) |
| **Z Host Agent runtime** | z/OS UNIX System Services (USS)                                  |
| **Deployment**           | Popsink Self-Hosted only                                         |

<Note>This connector runs an agent inside your mainframe environment, so supported z/OS and Db2 for z/OS levels are validated with you during onboarding. Get in touch with your release levels before planning a deployment.</Note>

## Overview

This connector enables reliable, secure extraction of dataset/log events from IBM Z with:

* **Z Host Agent (on Z)** – reads local sources and forwards events.
* **Off-host Worker (external)** – receives, processes, and delivers events into Popsink pipelines.

This split design isolates mainframe access and keeps sensitive credentials and on-host logic inside the Z environment.

***

## Architecture

### Components

**Z Host Agent**

* Runs in IBM Z (USS or native process).
* Reads configured Z sources (datasets, logs, SMF, etc.).
* Emits event batches over a secure channel.

**Off-host Worker**

* Runs outside Z (Popsink Self-Hosted).
* Writes to Popsink retention layer.
* Handles retries, DLQ, metrics.

### How It Works

1. Agent reads records locally.
2. Agent pushes batches over a secure connection to Worker.
3. Worker processes events and writes to destination.
4. Checkpoints are synced back to the agent.

***

## Z Host Agent

The Z Host Agent runs on IBM z/OS and is responsible for reliable Db2 log capture and delivery. It is intentionally lightweight, minimizing CPU usage while guaranteeing ordered and recoverable event flow.

### Responsibilities

**Read Db2 Logs via IFI**\
Streams log records directly from Db2 using IFI for near–real-time, low-overhead capture.

**Frame (Not Decode) Records**\
Packages raw log records for transport without decoding. Decoding happens off-host to reduce mainframe CPU usage and allow independent evolution of parsing logic.

**Partition Assignment**\
Deterministically assigns records to partitions to preserve ordering and enable parallel downstream processing.

**Maintain Offset Checkpoints**\
Tracks the current log position to ensure safe recovery and prevent data loss.

**Advance Checkpoint on ACK**\
Offsets advance only after worker acknowledgment, providing at-least-once delivery semantics.

**Replay from Archive Logs**\
Automatically resumes from Db2 archive logs when active logs are no longer available.

**Local Spooling**\
Buffers records locally during connectivity interruptions, ensuring continuous ingestion and controlled recovery.

***

## Prerequisites

* **Z Host Agent** must be installed on the IBM Z environment.
* Outbound connectivity from Z to Off-host Worker endpoint.
* Appropriate read access to mainframe sources.


# Kafka Source
Source: https://docs.popsink.com/connectors/source/kafka



The Kafka Source Connector integrates with Apache Kafka, a distributed, fault-tolerant, and highly-scalable event streaming platform designed for handling real-time data streams. It enables the efficient publishing, storing, and processing of large volumes of events, making it ideal for building data pipelines, messaging systems, and real-time analytics applications. Kafka is widely used across various industries for its high throughput, low latency, and strong durability guarantees.

## Compatible versions

|                          |                                                                                                                                                |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| **Apache Kafka**         | 2.1 and later, including 3.x and 4.x                                                                                                           |
| **Compatible platforms** | Confluent Platform & Confluent Cloud, Amazon MSK, Aiven for Apache Kafka, Redpanda, Azure Event Hubs (Kafka endpoint, Standard tier or higher) |
| **Protocol**             | Kafka wire protocol over `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, or `SASL_SSL`                                                                   |

<Note>Brokers older than 2.1 are not supported — the client negotiates API versions with the cluster and no longer implements the pre-2.1 protocol.</Note>

## Key Features

* **Continuous consumption:** the connector subscribes to your Kafka topics and streams events into Popsink as they are produced.
* **Broad authentication support:** PLAINTEXT, SSL, and SASL (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER, GSSAPI), including mutual TLS.
* **Message metadata:** optionally include Kafka message metadata (topic, partition, offset, timestamp) alongside the payload.

## Prerequisites

* A reachable Kafka cluster.
* Credentials with permission to **read** from the source topics (and to use the configured consumer group).

## Configuration

| Field                        | Required  | Description                                                                                            |
| ---------------------------- | --------- | ------------------------------------------------------------------------------------------------------ |
| **Bootstrap Servers**        | Yes       | Comma-separated `host:port` pairs (e.g. `kafka1.example.com:9092,kafka2.example.com:9092`)             |
| **Security Protocol**        | Yes       | `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, or `SASL_SSL`                                                    |
| **SASL Mechanism**           | With SASL | `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBEARER`, or `GSSAPI`                                  |
| **SASL Username / Password** | With SASL | Credentials for the chosen SASL mechanism                                                              |
| **Topic**                    | Yes       | Kafka topic to consume from                                                                            |
| **Consumer Group ID**        | No        | Consumer group used to track offsets; auto-generated when omitted                                      |
| **Include Metadata**         | No        | Include Kafka message metadata (topic, partition, offset, timestamp) in the records (default: `false`) |

## TLS and Mutual TLS

When connecting over `SSL` or `SASL_SSL`, you can additionally provide certificates (PEM or base64-encoded PEM):

| Field                  | Description                               |
| ---------------------- | ----------------------------------------- |
| **CA Certificate**     | CA certificate used to verify the brokers |
| **Client Certificate** | Client certificate for mutual TLS         |
| **Client Private Key** | Client private key for mutual TLS         |

## Security Protocols

| Protocol         | Encryption | Authentication                     |
| ---------------- | ---------- | ---------------------------------- |
| `PLAINTEXT`      | None       | None — development only            |
| `SSL`            | TLS        | TLS client certificates (optional) |
| `SASL_PLAINTEXT` | None       | SASL                               |
| `SASL_SSL`       | TLS        | SASL — recommended for production  |


# MongoDB Source
Source: https://docs.popsink.com/connectors/source/mongodb



The MongoDB Source Connector is a powerful and efficient solution designed to facilitate seamless data integration between your MongoDB databases and Popsink. This connector helps you synchronize with MongoDB in real-time, ensuring that your target systems remain updated with the latest information in real-time, driving business intelligence, analytics, and operational efficiency.

## Compatible versions

|                      |                                                                                       |
| -------------------- | ------------------------------------------------------------------------------------- |
| **MongoDB**          | 6.0, 7.0, 8.0                                                                         |
| **Topology**         | Replica set or sharded cluster — change streams are unavailable on standalone servers |
| **Managed services** | MongoDB Atlas                                                                         |
| **CDC engine**       | Debezium 3.5                                                                          |

<Note>MongoDB 5.0 and earlier are outside the tested matrix. Splitting change events larger than the 16 MB BSON limit requires MongoDB 6.0.9 or later.</Note>

## Prerequisites

* A `Replica Set` name: a replication set is a group of MongoDB instances sharing the same data and is defined in `replication.replSetName` for each instance in their respective configuration files ([see here](https://www.mongodb.com/docs/manual/replication/#replication-in-mongodb)).

* A user with:

  * `read` privileges on the `oplog` database.

  * `read` privileges on the `config` database.

  * `listDatabases` privileges.

  * Cluster-wide `find` and `changeStream` privileges.

## Key Features

* Real-time Change Data Capture (CDC): The MongoDB Source Connector employs a CDC mechanism using the native MongoDB logical decoding feature, capturing and streaming changes (inserts, updates, and deletes) as they occur in your database.

* Fault-tolerant and Scalable: The MongoDB Source Connector is built with fault tolerance and scalability in mind. It is capable of resuming data replication from the last known offset in case of failures, ensuring data consistency and minimal downtime.

* Initial Load: The connector automatically performs an initial full table load.

* Advanced Filtering: The MongoDB Source Connector provides a range of filtering options, including table and schema filters, allowing you to selectively replicate specific tables and schemas based on your needs.

## Security

The connector supports SSL/TLS encryption for secure communication between the connector and your MongoDB database.


# MSSQL Source
Source: https://docs.popsink.com/connectors/source/mssql



The MSSQL Source Connector is a powerful and efficient solution designed to facilitate seamless data integration between your Microsoft SQL Server (MSSQL) databases and Popsink. This connector helps you synchronize with Microsoft SQL Server in real-time, ensuring that your target systems remain updated with the latest information in real-time, driving business intelligence, analytics, and operational efficiency.

## Table of Contents

1. [Compatible versions](#compatible-versions)
2. [Key Features](#key-features)
3. [Security](#security)
4. [Prerequisites](#prerequisites)
5. [Configuration](#configuration)
6. [Creating a Dedicated Role](#creating-a-dedicated-role)
7. [Enabling CDC](#enabling-cdc)
8. [Housekeeping Scripts](#housekeeping-scripts)

## Compatible versions

|                      |                                                                                                 |
| -------------------- | ----------------------------------------------------------------------------------------------- |
| **SQL Server**       | 2017, 2019, 2022                                                                                |
| **Editions**         | Standard, Enterprise, Developer — CDC is available on Standard from SQL Server 2016 SP1 onwards |
| **Managed services** | Azure SQL Database, Azure SQL Managed Instance, Amazon RDS for SQL Server                       |
| **Requirements**     | SQL Server Agent running, CDC enabled per database and per table                                |
| **CDC engine**       | Debezium 3.5                                                                                    |

<Note>SQL Server 2016 is outside the tested matrix and works on a best-effort basis. SQL Server 2025 has not been validated yet — get in touch before planning an upgrade.</Note>

## Key Features

* **Real-time Change Data Capture (CDC):** the MSSQL Source Connector employs a CDC mechanism using the native Microsoft SQL Server logical decoding feature, capturing and streaming changes (inserts, updates, and deletes) as they occur in your database.
* **Fault-tolerant and Scalable:** The MSSQL Source Connector is built with fault tolerance and scalability in mind. It is capable of resuming data replication from the last known offset in case of failures, ensuring data consistency and minimal downtime.
* **Initial Load:** The connector automatically performs an initial full table load.
* **Advanced Filtering:** The MSSQL Source Connector provides a range of filtering options, including table and schema filters, allowing you to selectively replicate specific tables and schemas based on your needs.

## Security

The connector supports SSL/TLS encryption for secure communication between the connector and your Microsoft SQL Server database.

## Prerequisites

* You will need to have Change Data Capture enabled on the tables you want to ingest. [See Enabling CDC](#enabling-cdc)
* You will need a user with the necessary permissions. [See Creating a Dedicated Role](#creating-a-dedicated-role)

### Relevant Roles

* **sysadmin:** role required for enabling CDC at the database level.
* **db\_owner:** role required for enabling CDC at the table level.

## Configuration

| Field               | Required | Description                                                     |
| ------------------- | -------- | --------------------------------------------------------------- |
| **Host**            | Yes      | Hostname or IP address of the SQL Server                        |
| **Port**            | Yes      | TCP port (default `1433`)                                       |
| **Database**        | Yes      | Database name — a single database per connector for CDC capture |
| **User / Password** | Yes      | SQL Server login with the permissions described below           |

### Advanced Options

| Field                        | Default | Description                                                          |
| ---------------------------- | ------- | -------------------------------------------------------------------- |
| **Encrypt**                  | `true`  | JDBC connection encryption                                           |
| **Trust Server Certificate** | `true`  | Skips certificate verification — useful for self-signed certificates |
| **Initial Load**             | `true`  | Full snapshot of existing data before streaming changes              |

## Connecting through an SSH Tunnel

If the database is only reachable through a bastion host, enable **SSH Tunnel** and provide:

| Field               | Required | Description                                     |
| ------------------- | -------- | ----------------------------------------------- |
| **SSH Host**        | Yes      | Hostname or IP of the bastion host              |
| **SSH Port**        | Yes      | SSH port (default `22`)                         |
| **SSH User**        | Yes      | User to authenticate as on the bastion          |
| **SSH Private Key** | Yes      | **Base64-encoded** private key for the SSH user |

### Table Selection

After a successful credential check, pick the tables to replicate from the discovered list. The selection is stored as a comma-separated whitelist in `schema.table` format.

## Creating a Dedicated Role

This step creates a dedicated replication ROLE, assigns it to your USER and grants it the necessary permissions to work with

### Steps

1. **Create the Replication Role:**
   ```sql theme={null}
   USE [{{your_database}}];
   GO
   CREATE ROLE cdc_role;
   GO
   ```
2. **Assign the Replication Role to a User:**
   ```sql theme={null}
   USE [{{your_database}}];
   GO
   EXEC sp_addrolemember 'cdc_role', '{{your_user}}';
   GO
   ```
3. **Grant CDC Permissions:**
   ```sql theme={null}
   USE [{{your_database}}];
   GO
   GRANT EXECUTE ON SCHEMA::cdc TO cdc_role;
   GO
   ```
4. **Grant Permissions to Check Version, Encryption, and Track Schema:**
   ```sql theme={null}
   USE master;
   GO
   GRANT VIEW SERVER STATE TO cdc_role;
   GO

   USE {{your_database}};
   GO
   GRANT SELECT ON OBJECT::INFORMATION_SCHEMA.TABLES TO cdc_role;
   GO
   ```

## Enabling CDC

To enable CDC for your data, you need to do two things: 1) Enable CDC at the Database level, and 2) Enable CDC at the Table level.

### For Databases

1. **Enable CDC on a Database:**
   ```sql theme={null}
   USE [{{your_database}}];
   GO
   EXEC sys.sp_cdc_enable_db;
   GO
   ```

2. **Enable CDC on Multiple Databases:**
   ```sql theme={null}
   DECLARE @databases CURSOR;
   SET @databases = CURSOR FOR
   SELECT name FROM sys.databases WHERE name IN ({{your_database_list}});

   OPEN @databases;

   DECLARE @dbName NVARCHAR(50);

   FETCH NEXT FROM @databases INTO @dbName;

   WHILE @@FETCH_STATUS = 0
   BEGIN
       EXEC('USE ' + @dbName + '; EXEC sys.sp_cdc_enable_db;');
       FETCH NEXT FROM @databases INTO @dbName;
   END;

   CLOSE @databases;
   DEALLOCATE @databases;
   ```

### For Tables

1. **Enable CDC on a Table:**
   ```sql theme={null}
   USE [{{your_database}}];
   GO
   EXEC sys.sp_cdc_enable_table
       @source_schema = N'dbo',
       @source_name   = N'{{your_table}}',
       @role_name     = N'cdc_role', -- use @role_name = NULL if you haven't set up a role
       @supports_net_changes = 1;
   GO
   ```

2. **Enable CDC on multiple Tables:**
   ```sql theme={null}
   DECLARE @tables CURSOR;
   SET @tables = CURSOR FOR
   SELECT schema_name(t.schema_id) as schema_name, t.name as table_name
   FROM sys.tables t
   WHERE t.name IN ({{your_table_list}});

   OPEN @tables;

   DECLARE @schemaName NVARCHAR(50);
   DECLARE @tableName NVARCHAR(50);

   FETCH NEXT FROM @tables INTO @schemaName, @tableName;

   WHILE @@FETCH_STATUS = 0
   BEGIN
       EXEC('USE {{your_database}}; EXEC sys.sp_cdc_enable_table @source_schema = ''' + @schemaName + ''', @source_name = ''' + @tableName + ''', @role_name = ''cdc_user'' + @supports_net_changes = 1;');
       FETCH NEXT FROM @tables INTO @schemaName, @tableName;
   END;

   CLOSE @tables;
   DEALLOCATE @tables;
   ```

### For All

1. **Enable CDC on multiple Tables in multiple Databases:**
   ```sql theme={null}
   -- Enable CDC on all databases'
   DECLARE @dbName NVARCHAR(128);
   DECLARE @tableName NVARCHAR(128);
   DECLARE @sql NVARCHAR(MAX);

   -- Cursor for all databases
   DECLARE db_cursor CURSOR FOR
   SELECT name
   FROM sys.databases
   -- [Optional] WHERE name LIKE 'something-%';

   -- Open the cursor and fetch the first database
   OPEN db_cursor;
   FETCH NEXT FROM db_cursor INTO @dbName;

   -- Loop through all databases
   WHILE @@FETCH_STATUS = 0
   BEGIN
       -- Enable CDC on the database if not already enabled
       SET @sql = 'IF NOT EXISTS (SELECT 1 FROM sys.databases WHERE name = ''' + @dbName + ''' AND is_cdc_enabled = 1)
                   BEGIN
                       EXEC(''USE [' + @dbName + ']; EXEC sys.sp_cdc_enable_db'')
                   END';
       EXEC sp_executesql @sql;

       -- Cursor for all tables in the current database
       DECLARE table_cursor CURSOR FOR
       SELECT QUOTENAME(SCHEMA_NAME(schema_id)) + '.' + QUOTENAME(name)
       FROM sys.tables
       WHERE is_tracked_by_cdc = 0; -- Only select tables not already tracked by CDC, you may add more conditions

       -- Open the cursor for tables
       SET @sql = 'USE [' + @dbName + ']';
       EXEC sp_executesql @sql;

       OPEN table_cursor;
       FETCH NEXT FROM table_cursor INTO @tableName;

       -- Loop through all tables
       WHILE @@FETCH_STATUS = 0
       BEGIN
           -- Enable CDC on the table
           SET @sql = 'USE [' + @dbName + ']; EXEC sys.sp_cdc_enable_table @source_schema = N''' + SCHEMA_NAME(schema_id) + ''', @source_name = N''' + @tableName + ''', @role_name = NULL, @supports_net_changes = 1';
           EXEC sp_executesql @sql;

           -- Fetch the next table
           FETCH NEXT FROM table_cursor INTO @tableName;
       END;

       -- Close and deallocate the table cursor
       CLOSE table_cursor;
       DEALLOCATE table_cursor;

       -- Fetch the next database
       FETCH NEXT FROM db_cursor INTO @dbName;
   END;

   -- Close and deallocate the database cursor
   CLOSE db_cursor;
   DEALLOCATE db_cursor;
   ```

## Housekeeping Scripts

Here are a few useful scripts that can help you manage the CDC lifecycle.

### Purging Logs

Enabling Logs can take up space on your database so it's best practice to purge them regularly. There is a tradeoff between the log retention and the operational resilience of your replication pipeline. A longer retention means you will need more time to recover from outages before having to resync. Shorter retention takes up less space but gives you less time to recover.

1. **Purge CDC Logs:**
   ```sql theme={null}
   -- Set the database and the retention period
   USE YourDatabaseName;
   GO

   DECLARE @retentionDays INT = 7; -- Set the retention period in days

   -- Convert retention days to a cutoff LSN
   DECLARE @retentionDateTime DATETIME = DATEADD(DAY, -@retentionDays, GETDATE());
   DECLARE @cutoffLSN BINARY(10);

   -- Retrieve the minimum LSN based on the retention date
   SELECT @cutoffLSN = sys.fn_cdc_map_time_to_lsn('smallest greater than or equal', @retentionDateTime);

   -- Check if the cutoff LSN is valid
   IF @cutoffLSN IS NOT NULL
   BEGIN
       DECLARE @capture_instance NVARCHAR(200);
       DECLARE @sql NVARCHAR(MAX);

       -- Cursor to go through each capture instance
       DECLARE instance_cursor CURSOR FOR
       SELECT capture_instance
       FROM cdc.change_tables;

       OPEN instance_cursor;

       FETCH NEXT FROM instance_cursor INTO @capture_instance;

       WHILE @@FETCH_STATUS = 0
       BEGIN
           -- Generate the cleanup command for each capture instance
           SET @sql = N'EXEC sys.sp_cdc_cleanup_change_table @capture_instance = ''' + @capture_instance + ''', @low_water_mark = ''' + CONVERT(NVARCHAR(100), @cutoffLSN, 1) + ''';';
       
           -- Execute the cleanup command
           EXEC sp_executesql @sql;

           FETCH NEXT FROM instance_cursor INTO @capture_instance;
       END;

       CLOSE instance_cursor;
       DEALLOCATE instance_cursor;

       PRINT 'CDC logs purged successfully up to ' + CONVERT(NVARCHAR(30), @retentionDateTime, 121);
   END
   ELSE
   BEGIN
       PRINT 'No valid LSN found for the specified retention period.';
   END
   GO
   ```

### Check if CDC is Enabled

1. **Check if it is Enabled on a Database:**
   ```sql theme={null}
   USE master
   GO
   select name, is_cdc_enabled
   from sys.databases
   where name = '{{your_database}}'
   GO
   ```
   0 means CDC is not enabled, 1 means it is.

2. **Check if it is Enabled on a Table:**
   ```sql theme={null}
   USE [{{your_database}}];
   GO
   select name,type,type_desc,is_tracked_by_cdc
   from sys.tables
   where name = ‘<table_name>’
   GO
   ```
   0 means CDC is not enabled, 1 means it is.

### Disable CDC

1. **Disable CDC on a Database:**
   ```sql theme={null}
   USE [{{your_database}}];
   GO
   EXEC sys.sp_cdc_disable_db;
   GO
   ```
2. **Disable CDC on a Table:**
   ```sql theme={null}
   USE [{{your_database}}];
   GO
   EXEC sys.sp_cdc_disable_table
       @source_schema = N'dbo',
       @source_name   = N'{{your_table}}',
       @capture_instance = N'dbo_{{your_table}}';
   GO
   ```


# MySQL Source
Source: https://docs.popsink.com/connectors/source/mysql



The MySQL Source Connector is a powerful and efficient solution designed to facilitate seamless data integration between your MySQL databases and Popsink. This connector helps you synchronize with MySQL in real-time, ensuring that your target systems remain updated with the latest information in real-time, driving business intelligence, analytics, and operational efficiency.

## Compatible versions

|                      |                                                                       |
| -------------------- | --------------------------------------------------------------------- |
| **MySQL**            | 8.0, 8.4, 9.0, 9.1                                                    |
| **Storage engine**   | InnoDB only                                                           |
| **Binary log**       | Row-based — `binlog_format=ROW`, `binlog_row_image=FULL`              |
| **Managed services** | Amazon RDS & Aurora MySQL, Google Cloud SQL, Azure Database for MySQL |
| **CDC engine**       | Debezium 3.5                                                          |

<Note>MySQL 5.7 has reached end of life and is not supported. MariaDB is a separate product and is not covered by this connector.</Note>

## Prerequisites

* Binary logging enabled with **row-based format**. [See](https://debezium.io/documentation/reference/stable/connectors/mysql.html#enable-mysql-binlog):

```
log_bin = ON
binlog_format = ROW
binlog_row_image = FULL
```

* A user with the following privileges on the target source(s):

```sql theme={null}
SELECT
RELOAD
SHOW DATABASES
REPLICATION SLAVE
REPLICATION CLIENT
```

* Tables must use the **InnoDB** storage engine — MyISAM and other engines do not support transactional CDC reliably. Non-InnoDB tables are disabled in the table selection.

## Key Features

* Real-time Change Data Capture (CDC): the MySQL Source Connector employs a CDC mechanism using the native MySQL binlog, capturing and streaming changes (inserts, updates, and deletes) as they occur in your database.

* Fault-tolerant and Scalable: The MySQL Source Connector is built with fault tolerance and scalability in mind. It is capable of resuming data replication from the last known offset in case of failures, ensuring data consistency and minimal downtime.

* Initial Load: The connector automatically performs an initial full table load.

* Advanced Filtering: The MySQL Source Connector provides a range of filtering options, including table and schema filters, allowing you to selectively replicate specific tables and schemas based on your needs.

## Configuration

| Field               | Required | Description                                  |
| ------------------- | -------- | -------------------------------------------- |
| **Host**            | Yes      | Hostname or IP address of the MySQL server   |
| **Port**            | Yes      | TCP port (default `3306`)                    |
| **Database**        | Yes      | Database name to connect to                  |
| **User / Password** | Yes      | Credentials with replication and read access |

### Advanced Options

| Field                        | Default | Description                                                                                     |
| ---------------------------- | ------- | ----------------------------------------------------------------------------------------------- |
| **Use TLS**                  | `true`  | Encrypts the connection to MySQL                                                                |
| **Trust Server Certificate** | `true`  | Skips certificate verification — useful for self-signed certificates                            |
| **Initial Load**             | `true`  | Full snapshot of existing data before streaming binlog events; recommended for first-time setup |

## Connecting through an SSH Tunnel

If the database is only reachable through a bastion host, enable **SSH Tunnel** and provide:

| Field               | Required | Description                                     |
| ------------------- | -------- | ----------------------------------------------- |
| **SSH Host**        | Yes      | Hostname or IP of the bastion host              |
| **SSH Port**        | Yes      | SSH port (default `22`)                         |
| **SSH User**        | Yes      | User to authenticate as on the bastion          |
| **SSH Private Key** | Yes      | **Base64-encoded** private key for the SSH user |

## Table Selection

After a successful credential check, pick the tables to replicate from the discovered list. The selection is stored as a comma-separated whitelist.

## Security

The connector supports SSL/TLS encryption for secure communication between the connector and your MySQL database.


# Oracle Source
Source: https://docs.popsink.com/connectors/source/oracle



The Oracle Source Connector is an advanced tool designed to synchronize Oracle databases with Popsink efficiently and in real-time. It enables the real-time transfer of data, ensuring that the latest database changes are mirrored to the target systems. This capability is crucial for maintaining up-to-date business intelligence, enabling real-time analytics, and enhancing operational workflows.

## Compatible versions

|                      |                                                            |
| -------------------- | ---------------------------------------------------------- |
| **Oracle Database**  | 19c, 21c, 23ai, 26ai                                       |
| **Capture method**   | LogMiner, reading the online redo and archived redo logs   |
| **Deployments**      | Single instance and RAC; multitenant (CDB/PDB) and non-CDB |
| **Managed services** | Amazon RDS for Oracle, Oracle Base Database Service (OCI)  |
| **CDC engine**       | Debezium 3.5                                               |

<Note>Oracle 12c and 11g are not supported. On 23ai and 26ai, capture is available through LogMiner only.</Note>

## **Prerequisites**

Before deploying the Oracle Source Connector, certain prerequisites must be met to ensure a smooth integration process:

1. **Whitelisting Popsink's IP Address**: Ensure that Popsink's IP address is whitelisted to allow uninterrupted communication between Oracle databases and Popsink.
2. **Database Log Mode Configuration**: The Oracle database Archive Mode should be set to ArchiveLog.
3. **Supplemental Logging**: Supplemental Logs should be enable on the source you wish to replicate. This can me done at Database or Table Level :

   * At Database Level

   ```sql theme={null}
   ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
   ```

   * At Table Level

   ```sql theme={null}
   ALTER TABLE **{db.name}** ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
   ```

   > Note: If using AWS RDS, additional steps may be required. Please consult the relevant AWS RDS documentation and do feel free to reach out for help.

## **Required Permissions**

Although you *could* use a **`superuser`**, we do recommend setting up dedicated users for security reasons. The user account that the Oracle Source Connector utilizes must have sufficient privileges. The following SQL statements outline the necessary permissions:

```sql theme={null}
GRANT CREATE SESSION TO {username} CONTAINER=ALL;
GRANT SET CONTAINER TO {username} CONTAINER=ALL;
GRANT SELECT ON V_$DATABASE to {username} CONTAINER=ALL;
GRANT FLASHBACK ANY TABLE TO {username} CONTAINER=ALL;
GRANT SELECT ANY TABLE TO {username} CONTAINER=ALL;
GRANT SELECT_CATALOG_ROLE TO {username} CONTAINER=ALL;
GRANT EXECUTE_CATALOG_ROLE TO {username} CONTAINER=ALL;
GRANT SELECT ANY TRANSACTION TO {username} CONTAINER=ALL;
GRANT LOGMINING TO {username} CONTAINER=ALL;
GRANT CREATE TABLE TO {username} CONTAINER=ALL;
GRANT LOCK ANY TABLE TO {username} CONTAINER=ALL;
GRANT CREATE SEQUENCE TO {username} CONTAINER=ALL;
GRANT EXECUTE ON DBMS_LOGMNR TO {username} CONTAINER=ALL;
GRANT EXECUTE ON DBMS_LOGMNR_D TO {username} CONTAINER=ALL;
GRANT SELECT ON V_$LOG TO {username} CONTAINER=ALL;
GRANT SELECT ON V_$LOG_HISTORY TO {username} CONTAINER=ALL;
GRANT SELECT ON V_$LOGMNR_LOGS TO {username} CONTAINER=ALL;
GRANT SELECT ON V_$LOGMNR_CONTENTS TO {username} CONTAINER=ALL;
GRANT SELECT ON V_$LOGMNR_PARAMETERS TO {username} CONTAINER=ALL;
GRANT SELECT ON V_$LOGFILE TO {username} CONTAINER=ALL;
GRANT SELECT ON V_$ARCHIVED_LOG TO {username} CONTAINER=ALL;
GRANT SELECT ON V_$ARCHIVE_DEST_STATUS TO {username} CONTAINER=ALL;
GRANT SELECT ON V_$TRANSACTION TO {username} CONTAINER=ALL;
```

Replace **`{username}`** with the actual username being used for the connection.

## Steps

Go to: **[Sources](https://app.popsink.com/sources) -**> [**Create New**](https://app.popsink.com/sources/add) then select the **Oracle** connector and **Continue**

In the *Credentials* sections fill in the required information.

### **`host`**

* **Description**: The hostname or IP address of the Oracle database server.
* **Type**: **`string`**
* **Example**: **`oracle-db-server.example.com`**

### **`port`**

* **Description**: The port number on which the Oracle database server is listening.
* **Type**: **`integer`**
* **Default**: **`1521`** (default port for Oracle database)
* **Example**: **`1521`**

### **`user`**

* **Description**: The username used to connect to the Oracle database.
* **Type**: **`string`**
* **Example**: **`popsink_user`**

### **`password`**

* **Description**: The password associated with the specified user for authentication.
* **Type**: **`string`**
* **Example**: **`p@ssw0rd123`**

### **`service_name`** or **`sid`**

Provide **exactly one** of the two:

* **`service_name`**: The Oracle service name to connect to.
  * **Type**: **`string`**
  * **Example**: **`ORCLPDB1`**
* **`sid`**: The Oracle System Identifier, for legacy databases that are not service-name addressable.
  * **Type**: **`string`**
  * **Example**: **`ORCL`**

### Advanced Options

| Field                  | Default | Description                                             |
| ---------------------- | ------- | ------------------------------------------------------- |
| **Poll Interval (ms)** | `2000`  | How often the connector polls for new redo log activity |
| **Initial Load**       | `true`  | Snapshot existing rows before streaming changes         |

## Connecting through an SSH Tunnel

If the database is only reachable through a bastion host, enable **SSH Tunnel** and provide:

| Field               | Required | Description                                     |
| ------------------- | -------- | ----------------------------------------------- |
| **SSH Host**        | Yes      | Hostname or IP of the bastion host              |
| **SSH Port**        | Yes      | SSH port (default `22`)                         |
| **SSH User**        | Yes      | User to authenticate as on the bastion          |
| **SSH Private Key** | Yes      | **Base64-encoded** private key for the SSH user |

### Table Selection

After a successful credential check, pick the tables to replicate from the discovered list. The selection is stored as a comma-separated whitelist in `SCHEMA.TABLE` format.

You can use **Check Credentials** at any moment to verify the validity of your inputs.

Once the validity check has passed, the next step is the standard *Informations* page where you may give the connector a name and select which of your teams should own this connector.

Create the connector and you are now done. The connector may take a few minutes to synchronize with the source database.

## **Conclusion**

The Oracle Source Connector is an essential component for creating a reliable data pipeline between Oracle databases and Popsink. By following the guidelines provided in this document, you can ensure a successful setup and enjoy a robust, real-time data integration experience.


# Source Connectors Overview
Source: https://docs.popsink.com/connectors/source/overview

Discover Popsink source connectors and learn how to capture data from your systems in real time.

Popsink source connectors allow you to ingest change events from your operational systems and SaaS tools with minimal setup. This overview explains the common concepts you will find in each connector guide and links to the detailed documentation for every supported source.

## Before you begin

* Verify that the systems you want to connect are reachable from your Popsink deployment.
* Review the permissions required by the connector you selected. Most databases need replication or CDC privileges.
* Identify the tables, collections, or topics you plan to replicate so you can configure filters precisely.

## How each connector guide is structured

Every connector page follows the same flow:

1. **Prerequisites** – Network and security configuration required before connecting.
2. **Setup** – Authentication steps, object selection, and CDC configuration.
3. **Monitoring** – Guidance to track connector health through Popsink alerting and metrics.

## Explore connector families

Choose the family that matches your source to access the detailed guides:

### Databases

[Database connectors](./databases) cover relational engines, analytical warehouses, and document stores.

* [Postgres](./postgres)
* [MySQL](./mysql)
* [MSSQL](./mssql)
* [Oracle](./oracle)
* [IBM i](./ibmi)
* [Snowflake](./snowflake)
* [BigQuery](./bigquery)
* [MongoDB](./mongodb)
* [Firestore](./firestore)

### Streaming

[Streaming connectors](./streaming) ingest events from your message buses and real-time producers.

* [Kafka](./kafka)
* [Event Hub](./eventhub)
* [Datagen](./datagen)
* [Webhook](./webhook)

### SaaS & Applications

[SaaS & Applications connectors](./saas-applications) synchronize your business tools and custom integrations.

* [Generic Application](./application)
* [Attio](./attio)
* [HubSpot](./hubspot)
* [Salesforce](./salesforce)

Can't find your source? Contact [support@popsink.com](mailto:support@popsink.com) to discuss a custom integration.


# Pipedrive Source
Source: https://docs.popsink.com/connectors/source/pipedrive



The Pipedrive Source Connector ingests CRM data from Pipedrive — deals, persons, organizations, activities, and more — into Popsink pipelines. It runs on a configurable schedule and performs incremental synchronization, so only records updated since the last run are delivered downstream. Use it to feed sales analytics, forecasting models, and revenue dashboards with up-to-date pipeline data.

## Compatible versions

|                    |                 |
| ------------------ | --------------- |
| **Pipedrive API**  | API v1          |
| **Authentication** | API token       |
| **Protocol**       | REST over HTTPS |

## Key Features

* **Incremental synchronization:** the connector tracks each resource's `update_time` cursor across runs and only delivers new or updated records.
* **Selectable resources:** choose which Pipedrive resources to ingest (e.g. `deals`, `persons`, `organizations`, `activities`); available resources are discovered automatically.
* **Scheduled ingestion:** flexible scheduling by minutes, hours, or days, with an immediate first run on creation.

## Prerequisites

* A Pipedrive account with API access.
* A **personal API token**: in Pipedrive, go to **Settings → Personal preferences → API** and copy your token.

## Configuration

| Field              | Required | Description                                                                    |
| ------------------ | -------- | ------------------------------------------------------------------------------ |
| **Company Domain** | Yes      | Your Pipedrive company domain (e.g. `mycompany` for `mycompany.pipedrive.com`) |
| **API Token**      | Yes      | Your personal API token                                                        |
| **Schedule**       | Yes      | Sync frequency — see [Scheduling](#scheduling)                                 |

## Scheduling

API source connectors run on a schedule rather than streaming continuously. A first run is always triggered immediately after the connector is created; subsequent runs depend on the selected mode:

| Mode        | Behavior                                                      | Example                                      |
| ----------- | ------------------------------------------------------------- | -------------------------------------------- |
| **days**    | Runs every N days at 00:00 UTC + offset (in minutes)          | Offset 30 → every day at 00:30 UTC           |
| **hours**   | Runs every N hours at :00 + offset (in minutes)               | Offset 15 → 00:15, 01:15, 02:15, …           |
| **minutes** | Runs every N minutes, anchored on the connector creation time | Created 10:03, interval 10 → 10:13, 10:23, … |


# Postgres Source
Source: https://docs.popsink.com/connectors/source/postgres



The Postgres Source Connector is a powerful and efficient solution designed to facilitate seamless data integration between your PostgreSQL databases and Popsink. This connector helps you synchronize with PostgreSQL in real-time, ensuring that your target systems remain updated with the latest information in real-time, driving business intelligence, analytics, and operational efficiency.

## Compatible versions

|                             |                                                                                 |
| --------------------------- | ------------------------------------------------------------------------------- |
| **PostgreSQL**              | 14, 15, 16, 17, 18                                                              |
| **Logical decoding plugin** | `pgoutput` — built into PostgreSQL, nothing to install                          |
| **Managed services**        | Amazon RDS & Aurora PostgreSQL, Google Cloud SQL, Azure Database for PostgreSQL |
| **CDC engine**              | Debezium 3.5                                                                    |

<Note>PostgreSQL 13 and earlier are outside the tested matrix and are not supported. On managed services, use the provider's own replication role rather than a superuser.</Note>

## Prerequisites

* Your Postgres should be in `logical` mode (use `SHOW wal_level`).

* A user with `REPLICATION` rights on the tables you wish to connect to. One way to do this is:

```sql theme={null}
CREATE ROLE replication_role REPLICATION;
GRANT replication_role TO <your_user>;
```

* Are you using PostgreSQL on Heroku? Here’s a great guide to [migrating](https://towardsdatascience.com/migrating-from-heroku-postgres-to-amazon-rds-2e738e7730e5).

* Each table should have its `REPLICA IDENTITY` set to `FULL`

```sql theme={null}
ALTER TABLE mytable REPLICA IDENTITY FULL
```

* You have to create publications for your tables

```sql theme={null}
CREATE PUBLICATION popsink FOR TABLE <your table list here>;
```

* The user should have the rights to list tables.

## Key Features

* Real-time Change Data Capture (CDC): The Postgres Source Connector employs a CDC mechanism using the native PostgreSQL logical decoding feature, capturing and streaming changes (inserts, updates, and deletes) as they occur in your database.

* Schema Evolution and Compatibility: This connector supports schema evolution, automatically detecting changes in the PostgreSQL database schema and adapting the data stream accordingly.

* Fault-tolerant and Scalable: The Postgres Source Connector is built with fault tolerance and scalability in mind. It is capable of resuming data replication from the last known offset in case of failures, ensuring data consistency and minimal downtime.

* Initial Load: The connector automatically performs an initial full table load.

* Advanced Filtering: The Postgres Source Connector provides a range of filtering options, including table and schema filters, allowing you to selectively replicate specific tables and schemas based on your needs.

## Configuration

| Field               | Required | Description                                                  |
| ------------------- | -------- | ------------------------------------------------------------ |
| **Host**            | Yes      | Hostname or IP address of the PostgreSQL server              |
| **Port**            | Yes      | TCP port (default `5432`)                                    |
| **Database**        | Yes      | Database name to connect to                                  |
| **Schema**          | Yes      | Default schema used when resolving tables (default `public`) |
| **User / Password** | Yes      | Credentials with replication and metadata access             |

### Advanced Options

| Field                     | Default | Description                                                                                   |
| ------------------------- | ------- | --------------------------------------------------------------------------------------------- |
| **Publication Name**      | —       | Name of the PostgreSQL publication listing the tables to replicate                            |
| **Replication Slot Name** | —       | Logical replication slot name; use a dedicated name per connector instance to avoid conflicts |
| **Initial Load**          | `true`  | Snapshot existing rows before streaming changes                                               |

## Connecting through an SSH Tunnel

If the database is only reachable through a bastion host, enable **SSH Tunnel** and provide:

| Field               | Required | Description                                     |
| ------------------- | -------- | ----------------------------------------------- |
| **SSH Host**        | Yes      | Hostname or IP of the bastion host              |
| **SSH Port**        | Yes      | SSH port (default `22`)                         |
| **SSH User**        | Yes      | User to authenticate as on the bastion          |
| **SSH Private Key** | Yes      | **Base64-encoded** private key for the SSH user |

## Table Selection

After a successful credential check, pick the tables to replicate from the discovered list. The selection is stored as a comma-separated whitelist in `schema.table` format.

## Security

The connector supports SSL/TLS encryption for secure communication between the connector and your PostgreSQL database.


# REST API Source
Source: https://docs.popsink.com/connectors/source/rest-api



The REST API Source Connector lets you ingest data from **any REST API** into Popsink pipelines by providing a declarative JSON configuration. It is the generic counterpart to the dedicated API connectors (Zendesk, Shopify, Pipedrive, …): if a SaaS platform doesn't have its own connector yet, you can usually integrate it in minutes with this one. The configuration follows the [dlt `rest_api_source` schema](https://dlthub.com/docs/dlt-ecosystem/verified-sources/rest_api), describing the API client (base URL, authentication, pagination) and the resources (endpoints) to ingest.

## Compatible versions

|                          |                                                                                   |
| ------------------------ | --------------------------------------------------------------------------------- |
| **Any REST API**         | HTTP/1.1 or HTTP/2 over TLS, JSON responses                                       |
| **Authentication**       | Bearer token, API key (header or query), HTTP basic, OAuth 2.0 client credentials |
| **Configuration schema** | dlt `rest_api_source`, dlt 1.24                                                   |

<Note>The configuration follows the dlt `rest_api_source` schema. When Popsink upgrades dlt, existing configurations keep working — the schema is backwards compatible within the 1.x line.</Note>

## Key Features

* **Connect to any REST API:** base URL, authentication, pagination, and endpoints are all declared in a single JSON configuration.
* **Multiple auth styles:** bearer tokens, API keys (header or query), HTTP basic, and OAuth2 client credentials.
* **Incremental synchronization:** declare a cursor field per resource and only new or updated records are delivered on each run.
* **Scheduled ingestion:** flexible scheduling by minutes, hours, or days, with an immediate first run on creation.

## Configuration

| Field             | Required | Description                                                          |
| ----------------- | -------- | -------------------------------------------------------------------- |
| **Configuration** | Yes      | A JSON object following the dlt `rest_api_source` schema — see below |
| **Schedule**      | Yes      | Sync frequency — see [Scheduling](#scheduling)                       |

### Configuration structure

The configuration JSON has two main blocks:

* **`client`** — how to reach and authenticate against the API: `base_url`, `auth`, optional `paginator` and `headers`.
* **`resources`** — an array of endpoints to ingest. Each resource becomes a stream/table, with its `path`, a `data_selector` pointing at the records in the response, optional query `params`, and an optional `incremental` cursor.

### Example

Ingesting two resources from an API authenticated with a bearer token, with cursor-based incremental sync on `updated_at`:

```json theme={null}
{
  "client": {
    "base_url": "https://api.example.com/v1",
    "auth": {
      "type": "bearer",
      "token": "YOUR_API_TOKEN"
    },
    "paginator": {
      "type": "json_link",
      "next_url_path": "paging.next"
    }
  },
  "resources": [
    {
      "name": "customers",
      "endpoint": {
        "path": "customers",
        "data_selector": "data",
        "params": { "per_page": 100 },
        "incremental": {
          "cursor_path": "updated_at",
          "initial_value": "1970-01-01T00:00:00Z"
        }
      }
    },
    {
      "name": "invoices",
      "endpoint": {
        "path": "invoices",
        "data_selector": "data"
      }
    }
  ]
}
```

Refer to the [dlt REST API source documentation](https://dlthub.com/docs/dlt-ecosystem/verified-sources/rest_api) for the full schema, including all supported authentication types and paginators.

## Scheduling

API source connectors run on a schedule rather than streaming continuously. A first run is always triggered immediately after the connector is created; subsequent runs depend on the selected mode:

| Mode        | Behavior                                                      | Example                                      |
| ----------- | ------------------------------------------------------------- | -------------------------------------------- |
| **days**    | Runs every N days at 00:00 UTC + offset (in minutes)          | Offset 30 → every day at 00:30 UTC           |
| **hours**   | Runs every N hours at :00 + offset (in minutes)               | Offset 15 → 00:15, 01:15, 02:15, …           |
| **minutes** | Runs every N minutes, anchored on the connector creation time | Created 10:03, interval 10 → 10:13, 10:23, … |


# Source SaaS & Application Connectors
Source: https://docs.popsink.com/connectors/source/saas-applications

Discover Popsink source connectors dedicated to SaaS platforms and custom applications.

These connectors capture changes from SaaS applications and business services. They cover CRM solutions as well as your in-house apps.

> Read the [source connectors overview](./overview) to understand the permission model and monitoring best practices applied to every integration.

## Custom applications

* [Generic Application](./application) – Integrate your own applications through the Popsink API.
* [REST API](./rest-api) – Connect to any REST API with a declarative JSON configuration.

## Business SaaS

* [Attio](./attio) – Synchronize Attio CRM objects in real time.
* [HubSpot](./hubspot) – Capture HubSpot contact, deal, and engagement updates.
* [Salesforce](./salesforce) – Replicate Salesforce data using native CDC.
* [SAP](./sap) – Capture changes from SAP via Operational Data Provisioning and OData.
* [Zendesk](./zendesk) – Ingest tickets, users, and organizations incrementally.
* [Shopify](./shopify) – Sync orders, products, and customers from your store.
* [Pipedrive](./pipedrive) – Ingest deals, persons, organizations, and activities.

## Ads & Marketing

* [Google Ads](./google-ads) – Retrieve campaign and performance data via GAQL.
* [Facebook Ads](./facebook-ads) – Ingest campaigns, ad sets, and ads from the Meta Marketing API.

## Files & Collaboration

* [SharePoint](./sharepoint) – Ingest SharePoint Online site lists via Microsoft Entra ID.

Need another SaaS integration? [support@popsink.com](mailto:support@popsink.com) can help assess feasibility.


# Salesforce Source
Source: https://docs.popsink.com/connectors/source/salesforce



The Salesforce Source Connector integrates with Salesforce, a leading cloud-based Customer Relationship Management (CRM) platform. Salesforce offers a comprehensive suite of applications for sales, customer service, marketing automation, analytics, and application development. This connector enables real-time extraction and streaming of data from various Salesforce objects, including accounts, contacts, leads, opportunities, custom objects, and more. It supports both bulk and incremental data synchronization, allowing for efficient data transfer and up-to-date information flows. The Salesforce connector is particularly valuable for organizations looking to centralize their customer data, perform advanced analytics, create 360-degree customer views, or integrate Salesforce data with other business systems. It enables data-driven decision making across departments by making Salesforce data readily available for business intelligence, machine learning, and other data processing pipelines.

## Compatible versions

|                    |                                                                                                                    |
| ------------------ | ------------------------------------------------------------------------------------------------------------------ |
| **Salesforce**     | Any edition with API access — Enterprise, Unlimited, Performance, Developer (Professional requires the API add-on) |
| **APIs**           | REST API and Bulk API 2.0, plus PushTopic for streaming                                                            |
| **Authentication** | OAuth 2.0 connected app                                                                                            |

<Note>Salesforce retires older API versions roughly every three years; the connector targets a current version and is rolled forward as part of connector upgrades.</Note>

## Permissions

* To display objects
* To display attributes of objects
* To list and get records
* To list, read and create PushTopic


# SAP Source (ODP / OData)
Source: https://docs.popsink.com/connectors/source/sap

Source connector to capture Change Data Capture (CDC) from SAP using Operational Data Provisioning and OData services.

The **SAP Source Connector** enables continuous ingestion of changes from SAP operational systems using **Operational Data Provisioning (ODP)** with the **Delta Queue (ODQ)** and exposes them via **OData services**. This connector is designed for reliability, supportability, and near real-time replication into Popsink pipelines.

***

## Compatible versions

|                   |                                                                             |
| ----------------- | --------------------------------------------------------------------------- |
| **SAP systems**   | Any system exposing Operational Data Provisioning (ODP) through SAP Gateway |
| **Extraction**    | ODP with the Delta Queue (ODQ) — SAPI extractors and CDS views              |
| **Protocol**      | OData over HTTPS (port `443`)                                               |
| **Compatibility** | Confirmed per deployment                                                    |

<Note>ODP availability depends on your SAP release, add-on level, and SAP Gateway (`SAP_GWFND`) version, so compatibility is validated with you during onboarding.</Note>

## Prerequisites

Before configuring this connector, ensure the following are in place:

* **SAP System Access**\
  A reachable SAP system with Operational Data Provisioning (ODP) configured.

* **ODP / Delta Queue Support**\
  ODP extractors configured with Delta Queue enabled. Verify that extractors (SAPI or CDS) support delta semantics.

* **OData Exposure**\
  ODP providers must be published through **SAP Gateway** as **OData services**.

* **Technical User**\
  A dedicated technical user with privileges for:
  * Extracting ODP data
  * Monitoring and managing ODQ subscriptions and retention

* **Network Connectivity**\
  Outbound HTTPS (443) connectivity from Popsink to the SAP Gateway endpoints.

***

## Key Features

* **Near Real-Time CDC**\
  Captures incremental changes using ODP delta tokens instead of periodic batch pulls.

* **Standard SAP Mechanisms**\
  Built on SAP-supported ODP and OData standards for stability and upgrade safety.

* **Secure HTTP/OData Consumption**\
  Uses HTTP/OData for reliable and firewall-friendly communication.

* **Initial Load + Delta Sync**\
  Supports initial full extraction followed by efficient delta streaming.

***

## Security

The connector uses HTTPS to securely communicate with SAP OData endpoints.

**Authentication & Permissions:**

* Technical user credentials (username/password or OAuth) configured in Popsink.
* User must have:
  * Authorization to read ODP extractors
  * Rights to monitor and manage ODQ
* Follow least privilege principles.

***

## Configuration

To set up the SAP Source Connector:

### 1. Create a New Source

1. In the Popsink console, navigate to **Connectors → Sources**.
2. Click **New Source** and select **SAP Source (ODP / OData)**.

### 2. Provide Connection Details

| Field                   | Description                                          |
| ----------------------- | ---------------------------------------------------- |
| **SAP Gateway URL**     | Base URL for the SAP Gateway exposing OData services |
| **Technical Username**  | User with ODP extract + ODQ monitoring rights        |
| **Password / Secret**   | Secure credential for the user                       |
| **OData Provider Path** | Path to specific ODP provider(s) in SAP Gateway      |
| **TLS/SSL Config**      | Optional TLS settings if required                    |

### 3. Configure Stream Definition

1. Select the OData providers (extractors) you want to capture.
2. For each provider, define:
   * Initial load behavior
   * Delta token handling strategy
   * Optional filters if supported by the OData service

### 4. Validate & Launch

1. Test connectivity.
2. Save and start the source.
3. Confirm that the initial load completes and delta ingestion begins.

***

## Monitoring

Monitor connector health through the Popsink UI and alerts:

* **Throughput & Lag**\
  Track events/s and delta latency.

* **ODQ Queue Health**\
  Ensure subscriptions, retention, and error counts remain within expected bounds.

* **Errors & Retries**\
  Configure alerts for authentication issues, extraction failures, and schema changes.

* **Audit Logs**\
  Collect logs for troubleshooting and traceability.

***

## Example Flow

```mermaid theme={null}
flowchart LR
    SAP["SAP System"]
    ODP["ODP / ODQ"]
    OData["SAP Gateway (OData)"]
    Popsink["Popsink SAP Source"]
    Target["Target System"]

    SAP --> ODP --> OData --> Popsink --> Target
```


# SharePoint Source
Source: https://docs.popsink.com/connectors/source/sharepoint



The SharePoint Source Connector ingests data from a SharePoint Online site — site lists and their contents — into Popsink pipelines via the SharePoint REST API. It authenticates through a Microsoft Entra ID (Azure AD) application and runs on a configurable schedule, making collaborative and operational data from SharePoint available for analytics and integration.

## Compatible versions

|                    |                                                                        |
| ------------------ | ---------------------------------------------------------------------- |
| **SharePoint**     | SharePoint Online (Microsoft 365)                                      |
| **API**            | SharePoint REST API (`/_api`)                                          |
| **Authentication** | Microsoft Entra ID OAuth 2.0 client credentials (`/oauth2/v2.0/token`) |

<Note>SharePoint Server on-premises is not supported — the connector authenticates against the Entra ID token endpoint.</Note>

## Key Features

* **Site lists by default:** ingests the site's **lists** out of the box; the resource list can be customized.
* **Entra ID authentication:** secure app-only access using the OAuth2 client credentials flow — no user account required.
* **Scheduled ingestion:** flexible scheduling by minutes, hours, or days, with an immediate first run on creation.

## Prerequisites

* A SharePoint Online site you want to ingest from.
* A Microsoft Entra ID **app registration** with:
  * A **client secret**.
  * The **`Sites.Read.All`** application permission granted and admin-consented.
* The app must be authorized to access SharePoint site content in your tenant.

## Configuration

| Field             | Required | Description                                                              |
| ----------------- | -------- | ------------------------------------------------------------------------ |
| **Site URL**      | Yes      | SharePoint site URL (e.g. `https://contoso.sharepoint.com/sites/mysite`) |
| **Tenant ID**     | Yes      | Microsoft Entra ID tenant ID                                             |
| **Client ID**     | Yes      | Application (client) ID of the app registration                          |
| **Client Secret** | Yes      | Client secret generated for the app registration                         |
| **Schedule**      | Yes      | Sync frequency — see [Scheduling](#scheduling)                           |

## Scheduling

API source connectors run on a schedule rather than streaming continuously. A first run is always triggered immediately after the connector is created; subsequent runs depend on the selected mode:

| Mode        | Behavior                                                      | Example                                      |
| ----------- | ------------------------------------------------------------- | -------------------------------------------- |
| **days**    | Runs every N days at 00:00 UTC + offset (in minutes)          | Offset 30 → every day at 00:30 UTC           |
| **hours**   | Runs every N hours at :00 + offset (in minutes)               | Offset 15 → 00:15, 01:15, 02:15, …           |
| **minutes** | Runs every N minutes, anchored on the connector creation time | Created 10:03, interval 10 → 10:13, 10:23, … |


# Shopify Source
Source: https://docs.popsink.com/connectors/source/shopify



The Shopify Source Connector ingests commerce data from your Shopify store — orders, products, and customers — into Popsink pipelines through the Shopify Admin API. It runs on a configurable schedule, making your storefront data available for revenue analytics, inventory monitoring, and customer insights.

## Compatible versions

|                       |                                   |
| --------------------- | --------------------------------- |
| **Shopify Admin API** | REST Admin API, version `2024-01` |
| **Authentication**    | Custom app access token           |
| **Plans**             | Any plan that allows custom apps  |

<Note>The API version is pinned by the connector. Shopify keeps each REST Admin API version supported for at least 12 months, and Popsink rolls it forward as part of connector upgrades.</Note>

## Key Features

* **Core commerce resources:** ingests **orders**, **products**, and **customers** by default; the resource list can be customized.
* **Admin API integration:** uses a custom app access token scoped to exactly the data you allow.
* **Scheduled ingestion:** flexible scheduling by minutes, hours, or days, with an immediate first run on creation.

## Prerequisites

* A Shopify store with permission to create custom apps.
* An **Admin API access token**: in your Shopify admin, go to **Settings → Apps and sales channels → Develop apps**, create a custom app, grant it the Admin API access scopes `read_orders`, `read_products`, and `read_customers`, then install the app and copy the access token.

## Configuration

| Field            | Required | Description                                           |
| ---------------- | -------- | ----------------------------------------------------- |
| **Shop URL**     | Yes      | Your store URL (e.g. `https://mystore.myshopify.com`) |
| **Access Token** | Yes      | The custom app's Admin API access token               |
| **Schedule**     | Yes      | Sync frequency — see [Scheduling](#scheduling)        |

## Scheduling

API source connectors run on a schedule rather than streaming continuously. A first run is always triggered immediately after the connector is created; subsequent runs depend on the selected mode:

| Mode        | Behavior                                                      | Example                                      |
| ----------- | ------------------------------------------------------------- | -------------------------------------------- |
| **days**    | Runs every N days at 00:00 UTC + offset (in minutes)          | Offset 30 → every day at 00:30 UTC           |
| **hours**   | Runs every N hours at :00 + offset (in minutes)               | Offset 15 → 00:15, 01:15, 02:15, …           |
| **minutes** | Runs every N minutes, anchored on the connector creation time | Created 10:03, interval 10 → 10:13, 10:23, … |


# Snowflake Source
Source: https://docs.popsink.com/connectors/source/snowflake



The Snowflake Source Connector streams changes from Snowflake tables into Popsink pipelines using the native `CHANGES` clause together with Time Travel. It performs an initial full load of each selected table, then polls for incremental changes (inserts, updates, and deletes) every 10 minutes, making your warehouse data available downstream without manual exports.

## Compatible versions

|                           |                                                                 |
| ------------------------- | --------------------------------------------------------------- |
| **Snowflake**             | Managed service — Standard edition or higher                    |
| **Required features**     | Change tracking (`CHANGES`) and Time Travel                     |
| **Time Travel retention** | At least 1 day — the Standard default of 24 hours is sufficient |
| **Authentication**        | Key-pair (RSA 2048) — password authentication is not supported  |

<Note>If the connector stays offline for more than 12 hours it falls back to a full reload, because the Time Travel window may no longer cover the last processed position.</Note>

## Key Features

* **Native change tracking:** uses Snowflake's `CHANGES (INFORMATION => DEFAULT)` clause with Time Travel — no triggers or staging objects in your account.
* **Initial Load:** the connector automatically performs an initial full table load, then switches to incremental polling.
* **Schema evolution:** the table schema is re-introspected on every poll cycle, and the stream schema is updated automatically when columns change.
* **Resilient:** if the worker is offline longer than 12 hours, the connector automatically falls back to a full reload on restart, protecting against Time Travel window expiry.

## Prerequisites

1. A **dedicated user** with read access on the source database and schema.
2. **Key-pair authentication** enabled for that user. Only key-pair authentication is supported — password authentication is intentionally disabled.
3. **Change tracking** enabled on the tables you want to replicate:
   ```sql theme={null}
   ALTER TABLE <your_table> SET CHANGE_TRACKING = TRUE;
   ```
4. The account's **Time Travel retention** must cover the poll interval (the standard tier default of 24 hours is sufficient).

## Setting up Key-Pair Authentication

1. Generate an RSA key pair (no passphrase):
   ```bash theme={null}
   openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt
   openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
   ```
2. Register the public key on the Snowflake user:
   ```sql theme={null}
   ALTER USER <your_user> SET RSA_PUBLIC_KEY='<public_key_body>';
   ```

## Configuration

| Field                 | Required | Description                                                                                                                                |
| --------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| **Account**           | Yes      | Snowflake account identifier (e.g. `xy12345.eu-west-1`)                                                                                    |
| **User**              | Yes      | Snowflake user configured for key-pair authentication                                                                                      |
| **Database / Schema** | Yes      | Database and schema containing the tables to replicate                                                                                     |
| **Private Key**       | Yes      | Base64 body of the PEM PKCS8 private key **without** the `-----BEGIN/END PRIVATE KEY-----` headers — the connector adds them automatically |
| **Warehouse**         | No       | Warehouse to use for queries; the user's default warehouse is used when omitted                                                            |

## Table Selection

After entering credentials, select tables from the discovered list. For each selected table the connector:

1. **Initial load:** reads all existing rows.
2. **CDC polling:** every 10 minutes, queries `CHANGES (INFORMATION => DEFAULT) AT (TIMESTAMP => ...)` for inserts, updates, and deletes.

## Schema Handling

Snowflake column types are mapped as follows:

| Snowflake type                         | Stream type                      |
| -------------------------------------- | -------------------------------- |
| `VARCHAR` / `CHAR` / `TEXT` / `BINARY` | `string`                         |
| `NUMBER` / `INTEGER` / `DECIMAL`       | `float`                          |
| `FLOAT` / `DOUBLE` / `REAL`            | `double`                         |
| `BOOLEAN`                              | `boolean`                        |
| `DATE` / `TIMESTAMP` (all variants)    | `long` (Unix epoch milliseconds) |
| `VARIANT` / `OBJECT` / `ARRAY`         | `string` (JSON-serialized)       |


# Source Streaming Connectors
Source: https://docs.popsink.com/connectors/source/streaming

Configure Popsink source connectors for event streaming platforms and webhook ingestion.

Connectors in this category funnel continuous event streams into Popsink. They let you attach message buses, brokers, and application producers to our platform.

> Need a refresher on the core concepts? Read the [source connectors overview](./overview) before configuring your first stream.

## Streaming platforms

* [Kafka](./kafka) – Subscribe to your existing Kafka topics through managed consumer groups.
* [Event Hub](./eventhub) – Connect Azure Event Hubs to relay low-latency events.
* [Datagen](./datagen) – Simulate test data to validate pipelines in isolated environments.

## Event ingestion

* [Webhook](./webhook) – Receive inbound HTTP events and trigger Popsink transformations.

Using another streaming platform? [Contact us](mailto:support@popsink.com) to explore a custom connector.


# Webhook Source
Source: https://docs.popsink.com/connectors/source/webhook



The Webhook Source Connector allows you to directly send data to Popsink using webhooks. It is the most efficient way to build a direct integration between a streaming service and Popsink. Start by creating a connector then you will be able to register topics/schemas to send data to using you application.

## Compatible versions

|              |                                   |
| ------------ | --------------------------------- |
| **Protocol** | HTTP/1.1 over TLS 1.2 or later    |
| **Payload**  | JSON                              |
| **Clients**  | Any HTTP client — no SDK required |


# Zendesk Source
Source: https://docs.popsink.com/connectors/source/zendesk



The Zendesk Source Connector ingests support data from your Zendesk instance — tickets, users, organizations, and any other standard resource — into Popsink pipelines. It runs on a configurable schedule and performs incremental synchronization, so only records updated since the last run are delivered downstream. This is ideal for powering support analytics, SLA dashboards, and customer 360 initiatives with fresh helpdesk data.

## Compatible versions

|                 |                                   |
| --------------- | --------------------------------- |
| **Zendesk API** | Support API v2                    |
| **Products**    | Zendesk Suite and Zendesk Support |
| **Protocol**    | REST over HTTPS                   |

## Key Features

* **Incremental synchronization:** the connector tracks each resource's `updated_at` cursor across runs and only delivers new or updated records.
* **Selectable resources:** choose which Zendesk resources to ingest (e.g. `tickets`, `users`, `organizations`); available resources are discovered automatically.
* **Scheduled ingestion:** flexible scheduling by minutes, hours, or days, with an immediate first run on creation.

## Prerequisites

* A Zendesk account with API access enabled.
* A Zendesk **API token**: in Zendesk Admin Center, go to **Apps and integrations → APIs → Zendesk API**, enable Token Access, and add an API token.

## Configuration

| Field         | Required | Description                                                           |
| ------------- | -------- | --------------------------------------------------------------------- |
| **Subdomain** | Yes      | Your Zendesk subdomain (e.g. `mycompany` for `mycompany.zendesk.com`) |
| **Email**     | Yes      | Agent email address associated with the API token                     |
| **API Token** | Yes      | The Zendesk API token                                                 |
| **Schedule**  | Yes      | Sync frequency — see [Scheduling](#scheduling)                        |

## Scheduling

API source connectors run on a schedule rather than streaming continuously. A first run is always triggered immediately after the connector is created; subsequent runs depend on the selected mode:

| Mode        | Behavior                                                      | Example                                      |
| ----------- | ------------------------------------------------------------- | -------------------------------------------- |
| **days**    | Runs every N days at 00:00 UTC + offset (in minutes)          | Offset 30 → every day at 00:30 UTC           |
| **hours**   | Runs every N hours at :00 + offset (in minutes)               | Offset 15 → 00:15, 01:15, 02:15, …           |
| **minutes** | Runs every N minutes, anchored on the connector creation time | Created 10:03, interval 10 → 10:13, 10:23, … |


# Airtable Target
Source: https://docs.popsink.com/connectors/target/airtable

Configure and operate the Airtable target connector to sync Popsink with your collaborative bases.

The Airtable target connector surfaces your Popsink data in Airtable bases while preserving business structure and the platform's real-time collaboration.

## Compatible versions

|                    |                                                                              |
| ------------------ | ---------------------------------------------------------------------------- |
| **Airtable API**   | Web API v0                                                                   |
| **Authentication** | Personal access token                                                        |
| **Field types**    | Types exposed by the Web API, such as `singleLineText`, `number`, and `date` |

## Prerequisites

* **Airtable access:** an account with the "Creator" role on the target workspace plus an API key or PAT token with the `data.records:write` scope.
* **Application parameters:** identify the `baseId`, `tableId`, and target fields, and enable the REST API for the workspace.
* **Platform quotas:** monitor the 50,000-record-per-table limit, 5 requests per second, and the 100 MB maximum attachment size.
* **Governance:** log connector usage in your processing register and validate GDPR compliance for personal data.

## Configuration

<Tabs>
  <Tab title="Scheduled sync">
    <Steps>
      <Step title="Create the Airtable integration">
        In Popsink, select Airtable as the target and link the PAT token stored in your secrets manager. Verify the token is scoped to the intended workspace.
      </Step>

      <Step title="Map the fields">
        Map each Popsink field to the matching Airtable field. Use column types such as `singleLineText`, `number`, or `date` to stay compatible with the API.
      </Step>

      <Step title="Define the frequency">
        Set the load cadence (cron or interval). Enable stale-record purge to keep bases lightweight.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Near real-time updates">
    <Steps>
      <Step title="Enable webhook mode">
        Configure a Popsink trigger that sends mutations to the Airtable connector immediately after they are received.
      </Step>

      <Step title="Optimize batches">
        Tune `max_records_per_call` and `retry_on_rate_limit` to absorb bursts while respecting API quotas.
      </Step>

      <Step title="Secure credentials">
        Rotate the PAT token every 90 days and restrict its usage with Airtable access controls.
      </Step>
    </Steps>
  </Tab>
</Tabs>

## Monitoring

* Analyze the Popsink dashboard to track API response times, the number of records created or updated, and retry attempts.
* Enable alert webhooks when the Airtable API returns `429 Too Many Requests` or `422 Unprocessable Entity`.
* Archive transformation logs to audit changes applied to critical fields.

## Best practices

* Simplify the Airtable views used by Popsink to reduce write latency.
* Document field mappings in a dictionary shared with business teams.
* Automate the removal of obsolete attachments to respect storage limits.

<Callout type="warning">
  Do not exceed 10 "Lookup" or "Rollup" columns in automatically fed tables—the API can reject updates if dependencies become cyclical.
</Callout>

## FAQ

<AccordionGroup>
  <Accordion title="How do I manage record key conflicts?">
    Use the external key column (`recordId`) provided by Popsink. If a conflict occurs, choose the `merge` option to preserve Airtable comments.
  </Accordion>

  <Accordion title="Can I write to multiple tables?">
    Yes—create one target per table with dedicated filters and reuse the same PAT token when permissions allow.
  </Accordion>

  <Accordion title="How do I track validation errors?">
    Review the Popsink log for details about the rejected request, then replay the call with the Airtable API tester to identify the invalid field.
  </Accordion>
</AccordionGroup>

<CardGroup>
  <Card title="Alerting Popsink" href="/alerting">
    Trigger notifications whenever an Airtable limit is reached.
  </Card>

  <Card title="Target dashboard" href="/target">
    Visualize global metrics for all destination connectors.
  </Card>

  <Card title="Pipeline templates" href="/quickstart">
    Use reference examples to industrialize your Airtable synchronizations.
  </Card>
</CardGroup>


# Application Target
Source: https://docs.popsink.com/connectors/target/application



The Application Target Connector allows you to consume data from Popsink using Python, Node, Go, Java or other programming languages. It is the most efficient way to build a direct integration between your application and Popsink. Start by creating a connector then you will be able to subscribe to existing topics/schemas to receive data in your application.

## Compatible versions

|                        |                                                                                              |
| ---------------------- | -------------------------------------------------------------------------------------------- |
| **Protocol**           | Kafka wire protocol — any client library (Python, Node.js, Go, Java, …)                      |
| **Client requirement** | Kafka client supporting `SASL_SSL` and API version negotiation (Kafka 2.1 protocol or later) |
| **Schema registry**    | Confluent-compatible Schema Registry API                                                     |
| **Transport**          | TLS 1.2 or later                                                                             |


# BigQuery Target
Source: https://docs.popsink.com/connectors/target/bigquery



The BigQuery Target Connector enables seamless integration with Google BigQuery, a fully-managed, serverless data warehouse that enables super-fast SQL queries using the processing power of Google's infrastructure. This connector streams data from Popsink directly into BigQuery, automatically handling schema mapping, data type conversion, and table creation. It is particularly useful for organizations looking to perform large-scale analytics or combine streaming data with historical data for comprehensive business intelligence.

## Compatible versions

|                     |                                                                              |
| ------------------- | ---------------------------------------------------------------------------- |
| **Google BigQuery** | Managed service — no version to pin                                          |
| **API**             | BigQuery API v2 with the Storage Write API                                   |
| **Location**        | The target dataset must already exist — Popsink creates tables, not datasets |

## Key Features

* **Automatic table management:** tables are created automatically inside your dataset; schema evolution is handled as the source schema changes.
* **History + latest state:** every change event is preserved in a history table, with an upsert view exposing the current state per primary key.
* **CDC replication:** primary keys configured in the subscription mapping are used to apply updates and deletes correctly.

## Prerequisites

* A GCP project with BigQuery enabled and an **existing dataset** — Popsink creates tables, not datasets.
* A **service account** with the following roles on the dataset/project:
  * **BigQuery Data Editor** (`roles/bigquery.dataEditor`) — create tables and insert data
  * **BigQuery Job User** (`roles/bigquery.jobUser`) — run load jobs

## Configuration

| Field               | Required | Description                                                                       |
| ------------------- | -------- | --------------------------------------------------------------------------------- |
| **Service Account** | Yes      | GCP service account key in JSON format — paste the full content of the key file   |
| **Project**         | No       | GCP project ID; auto-extracted from the service account's `project_id` when empty |
| **Dataset**         | Yes      | Existing BigQuery dataset where tables will be created                            |

## How It Works

For each subscription, Popsink creates:

1. A **history table** (`{table}_history`) containing all CDC events with metadata columns.
2. An **upsert view** (`{table}`) exposing the latest state per primary key.

Schema evolution is handled automatically — new columns are added to the BigQuery table as the source schema evolves.


# ClickHouse Target
Source: https://docs.popsink.com/connectors/target/clickhouse



Welcome to the ClickHouse Target Connector documentation. This guide is designed to help you effectively integrate and utilize ClickHouse as a data sink for your CDC (Change Data Capture) data pipelines. The ClickHouse Target Connector allows you to seamlessly transfer data from your tools, applications and services to a ClickHouse database, leveraging its high performance for analytics and large-scale data storage.

## Compatible versions

|                   |                                                          |
| ----------------- | -------------------------------------------------------- |
| **ClickHouse**    | ClickHouse Cloud and self-managed servers                |
| **Table engine**  | `ReplacingMergeTree` for incremental updates and deletes |
| **Protocol**      | HTTPS interface — port `8443` on ClickHouse Cloud        |
| **Compatibility** | Confirmed per deployment                                 |

### Overview

ClickHouse is an open-source columnar database management system that is optimized for OLAP (Online Analytical Processing) scenarios. It is designed to process billions of rows and gigabytes of data per second with low latency. Using the ClickHouse Target Connector, you can push data from various sources into your ClickHouse instance, making it available for analysis and reporting.

### Features

* **High Performance**: Utilizes ClickHouse's capabilities for handling large volumes of data with high insertion speeds.
* **Real-time Ingestion**: Popsink sinks data in real-time to your Clickhouse tables.
* **Data Consistency**: Includes features for handling duplicates and ensuring data consistency.
* **Incremental Writes**: Popsink leverages Change Data Capture and ClickHouse’s ReplacingMergeTree to write incremental updates and deletes to your tables. You can read more about forcing deduplication in the [FINAL modifier documentation](https://clickhouse.com/docs/en/sql-reference/statements/select/from#final-modifier).

### Prerequisites

Before setting up the ClickHouse Target Connector, ensure you have the following:

* A running ClickHouse server accessible from the service.
* A user with appropriate credentials and permissions to access and write to the ClickHouse database. Should you wish to create a new user, here is how you could do so:

```sql theme={null}
CREATE ROLE POPSINK_ROLE;
GRANT SELECT, CREATE, SHOW, INSERT  ON <my_database>.* TO POPSINK_ROLE;

CREATE USER <popsink_user> IDENTIFIED WITH sha256_password BY '<my_password>'
GRANT POPSINK_ROLE TO <popsink_user>;
```

### Configuration

To configure the ClickHouse Target Connector, follow these steps:

* **Username**: if you followed the Prerequisites, this will be `<popsink_user>`. Otherwise use the username you wish to use.
* **Password**: if you followed the Prerequisites, this will be `<my_password>`. Otherwise use the corresponding password for the username in the previous step.
* **Host**: You can find the Host in the “Connect” option of your Clickhouse Console, it should look like this: `6dw0f8sj4.us-east-1.aws.clickhouse.cloud`
* **Port**: For Clickhouse Cloud this is generally `8443`
* **Database Name**: if you followed the Prerequisites, this will be `<my_database>`. Otherwise specify the name of the database you wish to write in.


# CosmosDB Target
Source: https://docs.popsink.com/connectors/target/cosmosdb



The CosmosDB Target Connector facilitates seamless integration with Azure Cosmos DB, Microsoft's globally distributed, multi-model database service. This connector enables real-time streaming of data from Popsink into Cosmos DB containers, supporting various APIs including SQL, MongoDB, Cassandra, Gremlin, and Table. It handles automatic schema mapping, data type conversion, and can create or update containers as needed. The CosmosDB connector is particularly valuable for applications requiring low-latency data access at global scale, such as IoT telemetry, real-time analytics, and personalization engines. It allows organizations to leverage Cosmos DB's multi-region writes, automatic indexing, and flexible consistency models while maintaining efficient data pipelines from diverse sources through Popsink. This connector is ideal for businesses looking to build highly responsive and globally distributed applications with real-time data synchronization capabilities.

## Compatible versions

|                     |                                     |
| ------------------- | ----------------------------------- |
| **Azure Cosmos DB** | Managed service — no version to pin |
| **API**             | NoSQL (Core) API                    |
| **Compatibility**   | Confirmed per deployment            |


# Target Data Lakes & Streaming Connectors
Source: https://docs.popsink.com/connectors/target/data-lakes-streaming

Route Popsink pipelines into your data lakehouses and streaming services.

Use these connectors to feed large-scale storage platforms or broadcast real-time events. They provide flexible options to expose your transformed data.

> Revisit the [target connectors overview](./overview) to master the shared partitioning, schema, and monitoring parameters across all destinations.

## Data lakes

* [Iceberg](./iceberg) – Publish Iceberg tables optimized for ACID queries.
* [Hudi](./hudi) – Maintain Hudi datasets with precise version control.
* [Unity Catalog (Delta Lake)](./deltalake) – Write Delta tables to Azure Blob Storage and register them in Databricks Unity Catalog.

## Event distribution

* [Kafka](./kafka) – Relay output events to an Apache Kafka cluster (Kafka-to-Kafka pipelines).
* [Pub/Sub](./pubsub) – Deliver Popsink events to Google Cloud Pub/Sub.
* [Webhook](./webhook) – Send outbound HTTP notifications to your downstream services.

Need a specific lake or streaming integration? [support@popsink.com](mailto:support@popsink.com) is ready to help.


# Target Database Connectors
Source: https://docs.popsink.com/connectors/target/databases

Explore Popsink target connectors for transactional, analytical, and NoSQL databases.

The target connectors in this family make your Popsink data available in the databases your teams already use. Choose the technology that aligns with your analytical or operational needs.

> Before creating a destination, review the [target connectors overview](./overview) to revisit the shared throughput, schema, and governance parameters.

## Relational databases & warehouses

* [Snowflake](./snowflake) – Feed your Snowflake warehouses with analytics-ready pipelines.
* [Postgres](./postgres) – Publish transformed data into dedicated Postgres schemas.
* [MySQL](./mysql) – Push output tables into your MySQL or MariaDB instances.
* [Oracle](./oracle) – Synchronize datasets to Oracle Database securely.
* [Teradata](./teradata) – Load analytical tables optimized for your Teradata workloads.
* [IBM i](./ibmi) – Expose Popsink results in DB2 for i for your legacy applications.
* [MSSQL](./mssql) – Distribute Popsink transformations to Microsoft SQL Server.
* [ClickHouse](./clickhouse) – Deliver high-frequency data into ClickHouse.

## NoSQL, vector, and cloud services

* [BigQuery](./bigquery) – Make Popsink datasets available in BigQuery for serverless analytics.
* [MongoDB](./mongodb) – Write into MongoDB Atlas or self-hosted collections.
* [Cosmos DB](./cosmosdb) – Synchronize multi-model Azure Cosmos DB containers.
* [Pinecone](./pinecone) – Keep your Pinecone vector indexes aligned with Popsink pipelines.
* [Elasticsearch](./elasticsearch) – Index change events into Elasticsearch for search and analytics.

Don't see your database yet? [support@popsink.com](mailto:support@popsink.com) can share the roadmap or scope a dedicated connector.


# Unity Catalog (Delta Lake) Target
Source: https://docs.popsink.com/connectors/target/deltalake



The Unity Catalog Target Connector writes your pipeline data to Delta Lake tables stored in Azure Blob Storage and registers them in a Databricks Unity Catalog catalog and schema. Data is written in Delta format with full ACID compliance and schema evolution, and tables are queryable from any Databricks workspace once registered. The connector is delivered as a native worker — no Spark cluster and no SQL warehouse are required on your side — streaming change events directly into your lakehouse.

Because the worker writes Delta files directly to Azure Blob Storage (outside Databricks compute) and registers them as **external** tables, two requirements are easy to miss: the metastore must allow **External Data Access**, and the Service Principal needs the **external-table** privileges. Both are detailed below.

## Table of Contents

1. [Compatible versions](#compatible-versions)
2. [Key Features](#key-features)
3. [Prerequisites](#prerequisites)
4. [Storage Authentication](#storage-authentication)
5. [Databricks Authentication](#databricks-authentication)
6. [External Data Access](#external-data-access)
7. [Catalog and Schema](#catalog-and-schema)

## Compatible versions

|                       |                                                                  |
| --------------------- | ---------------------------------------------------------------- |
| **Delta Lake**        | Tables written with delta-rs 0.22                                |
| **Databricks**        | Any workspace with Unity Catalog enabled                         |
| **Unity Catalog API** | REST API 2.1, with External Data Access enabled on the metastore |
| **Storage**           | Azure Blob Storage / ADLS Gen2                                   |

<Note>No Spark cluster or SQL warehouse is needed in the write path — the worker writes Delta files directly and registers external tables.</Note>

## Key Features

* **Delta Lake format:** data is written as Delta tables with ACID transactions and schema evolution.
* **Unity Catalog registration:** each subscription's target table is automatically created or updated under the catalog and schema you provide, making it instantly queryable from Databricks.
* **Native streaming worker:** change events are streamed directly into Azure Blob Storage without a Spark job or SQL API in the write path.
* **Secure by design:** Databricks access uses OAuth Machine-to-Machine with a Service Principal — no Personal Access Tokens.

## Prerequisites

* An Azure Blob Storage account with an existing container for the Delta data.
* A Databricks workspace with Unity Catalog enabled.
* An existing Unity Catalog **catalog** and **schema** — the connector registers tables but does not create the catalog or schema.
* **External Data Access** enabled on the metastore (per-metastore, admin-only). See [External Data Access](#external-data-access).
* A Databricks **Service Principal** with an OAuth secret and the required privileges on the target catalog and schema. Because the connector registers **external** tables, those are `USE CATALOG`, `USE SCHEMA`, `CREATE EXTERNAL TABLE` and `EXTERNAL USE SCHEMA`.

## Storage Authentication

Two authentication methods are supported for the underlying Azure Blob Storage. Pick the one that matches how you administer the storage account.

### Option A — Connection String

Authenticate with an Azure Blob Storage connection string, found in the Azure Portal under **Storage Account → Access keys**.

| Field                 | Description                                                                                                |
| --------------------- | ---------------------------------------------------------------------------------------------------------- |
| **Connection String** | e.g. `DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=...;EndpointSuffix=core.windows.net` |
| **Container Name**    | Existing container where Delta table data will be written (e.g. `my-lakehouse`)                            |

### Option B — Service Principal (SPN)

Authenticate with an Azure Active Directory Service Principal. The Service Principal must have the **Storage Blob Data Contributor** role on the storage account.

| Field                    | Description                                                       |
| ------------------------ | ----------------------------------------------------------------- |
| **Storage Account Name** | Name of the Azure Storage account (e.g. `mystorageaccount`)       |
| **Tenant ID**            | Azure AD tenant ID (Azure Portal → Microsoft Entra ID → Overview) |
| **Client ID**            | Application (client) ID of the registered app                     |
| **Client Secret**        | A client secret generated for the registered app                  |
| **Container Name**       | Existing container where Delta table data will be written         |

## Databricks Authentication

The connector authenticates to Databricks via **OAuth Machine-to-Machine (M2M)** using a Databricks Service Principal: it exchanges the Service Principal's client ID and secret for a short-lived access token at the workspace token endpoint, then registers each target table under your catalog and schema.

<Note>Personal Access Tokens (PATs) are not supported by this connector. Databricks deprecates PATs for new integrations — use a Service Principal with an OAuth secret.</Note>

### Setting up the Service Principal

1. In your Databricks workspace, open **Settings → Identity and access → Service principals** and create a new Service Principal (or pick an existing one).
2. Under the Service Principal, generate an **OAuth secret**. Copy the displayed **client ID** and **secret** — the secret is shown only once.
3. Grant the Service Principal the Unity Catalog privileges needed to register **external** tables on the target catalog and schema:

```sql theme={null}
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<service_principal>`;
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal>`;
GRANT CREATE EXTERNAL TABLE ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal>`;
GRANT EXTERNAL USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal>`;
```

`USE CATALOG` / `USE SCHEMA` make the catalog and schema visible; `CREATE EXTERNAL TABLE` allows registering externally-written Delta files; `EXTERNAL USE SCHEMA` lets a non-Databricks engine operate on the schema's tables. Popsink's credential check verifies these at configuration time when the Service Principal can read its own effective permissions.

| Field                        | Description                                                                                       |
| ---------------------------- | ------------------------------------------------------------------------------------------------- |
| **Workspace URL**            | Base URL of your Databricks workspace (e.g. `https://adb-1234567890123456.7.azuredatabricks.net`) |
| **Databricks Client ID**     | The Service Principal's application ID (UUID)                                                     |
| **Databricks Client Secret** | The OAuth secret generated for the Service Principal                                              |

## External Data Access

The worker writes Delta files directly to your storage container and registers them as **external** tables via the Unity Catalog REST API. Because the write happens outside Databricks compute, the metastore must have the **External Data Access** flag enabled. It is a **per-metastore** setting that only a **metastore admin** can change. When it is disabled, table registration fails with a `403 PERMISSION_DENIED` (`EXTERNAL_ACCESS_DISABLED_ON_METASTORE`).

Enable it one of two ways:

* **UI:** Catalog → Metastore details → *External data access* → enable.
* **API:**

```bash theme={null}
PATCH /api/2.1/unity-catalog/metastores/{metastore_id}
{ "external_access_enabled": true }
```

<Note>
  Popsink's credential check fails fast at configuration time if External Data Access is disabled or the external-table grants are missing, so you can fix these before the connector goes live rather than discovering them at runtime.
</Note>

## Catalog and Schema

| Field            | Description                                                         |
| ---------------- | ------------------------------------------------------------------- |
| **Catalog Name** | Unity Catalog catalog where tables will be registered (e.g. `main`) |
| **Schema Name**  | Schema (database) under that catalog (e.g. `bronze`)                |

Both must exist before the connector starts. Tables are created or updated using each subscription's target table name within this catalog and schema.


# Elasticsearch Target
Source: https://docs.popsink.com/connectors/target/elasticsearch



The Elasticsearch Target Connector enables your pipelines to deliver data from Popsink directly into Elasticsearch indices. Elasticsearch is a distributed search and analytics engine widely used for full-text search, log analytics, and real-time application monitoring. By streaming change events into Elasticsearch, this connector keeps your indices continuously up to date with your source systems, powering search experiences and analytics dashboards with fresh data.

## Table of Contents

1. [Compatible versions](#compatible-versions)
2. [Key Features](#key-features)
3. [Prerequisites](#prerequisites)
4. [Configuration](#configuration)
5. [Authentication Methods](#authentication-methods)

## Compatible versions

|                   |                                |
| ----------------- | ------------------------------ |
| **Elasticsearch** | 8.x and 9.x                    |
| **Deployments**   | Self-managed and Elastic Cloud |
| **Client**        | Elasticsearch Java client 9.4  |

<Note>Elasticsearch 7.x and OpenSearch are not supported — the client speaks the 8.x/9.x REST API.</Note>

## Key Features

* **Real-time indexing:** change events are streamed into Elasticsearch indices as they occur, keeping search results current.
* **Flexible authentication:** supports basic authentication, API keys, or no authentication for local development.
* **TLS support:** connects over HTTPS with optional certificate verification control.

## Prerequisites

* A reachable Elasticsearch cluster (self-managed or Elastic Cloud).
* A user or API key with **write permissions** on the target indices.

## Configuration

| Field                     | Required       | Description                                                                                                        |
| ------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------ |
| **URL**                   | Yes            | Full URL of the Elasticsearch cluster, including protocol and port (e.g. `https://my-cluster.es.example.com:9243`) |
| **Authentication Method** | Yes            | `basic` (default), `api_key`, or `none`                                                                            |
| **Username / Password**   | With `basic`   | Credentials of an Elasticsearch user with write permissions on the target indices                                  |
| **API Key**               | With `api_key` | Base64-encoded API key                                                                                             |
| **Verify SSL**            | No             | Whether to verify TLS certificates (default: `true`)                                                               |

## Authentication Methods

| Method       | Description                                         |
| ------------ | --------------------------------------------------- |
| **basic**    | Username and password authentication                |
| **api\_key** | API key authentication — recommended for production |
| **none**     | No authentication — local development only          |

### Creating an API Key

You can create a Base64-encoded API key using the [Elasticsearch Create API Key API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html):

```json theme={null}
POST /_security/api_key
{
  "name": "popsink-target",
  "role_descriptors": {
    "popsink_writer": {
      "indices": [
        { "names": ["*"], "privileges": ["create_index", "write", "view_index_metadata"] }
      ]
    }
  }
}
```

Use the `encoded` value returned by the API as the connector's API Key.


# Hubspot Target
Source: https://docs.popsink.com/connectors/target/hubspot



The Hubspot Target Connector enables seamless integration between Popsink and HubSpot, a leading inbound marketing, sales, and customer service platform. This connector allows you to stream data from Popsink directly into various HubSpot objects such as contacts, companies, deals, tickets, and custom objects. It supports both batch updates and real-time data synchronization, automatically handling data mapping and property creation or updates within HubSpot. The Hubspot connector is particularly valuable for organizations looking to enrich their CRM data with information from other systems, automate marketing and sales processes based on real-time data insights, or maintain a unified customer view across multiple platforms. It enables marketing, sales, and customer service teams to leverage up-to-date data for personalized engagement, lead scoring, and customer lifecycle management, while maintaining efficient data flows from various sources through Popsink.

## Compatible versions

|                    |                                                                    |
| ------------------ | ------------------------------------------------------------------ |
| **HubSpot API**    | CRM API v3                                                         |
| **Authentication** | Private app access token                                           |
| **Objects**        | Standard and custom CRM objects exposed by the v3 object endpoints |


# Hudi Target
Source: https://docs.popsink.com/connectors/target/hudi



The Hudi Target Connector facilitates integration with Apache Hudi (Hadoop Upserts Deletes and Incrementals), an open-source data management framework for incrementally processing and managing big data. This connector enables streaming data from Popsink directly into Hudi datasets, supporting both batch and near real-time data ingestion. It handles upserts, deletes, and incremental data processing efficiently, automatically managing schema evolution and data consistency. The Hudi connector is particularly useful for organizations building large-scale data lakes or implementing a lakehouse architecture, as it combines the flexibility of a data lake with database-like ACID transactions and incremental processing capabilities. It enables data teams to implement change data capture (CDC), create time travel queries, and perform efficient snapshot and incremental data retrieval. This connector is ideal for use cases requiring strict data quality and consistency, such as regulatory compliance, audit trails, or building scalable, real-time data pipelines while maintaining historical data versions.

## Compatible versions

|                   |                                        |
| ----------------- | -------------------------------------- |
| **Apache Hudi**   | Copy-on-Write and Merge-on-Read tables |
| **Compatibility** | Confirmed per deployment               |

<Note>Supported Hudi and query-engine versions are validated with you before deployment — get in touch with your stack details.</Note>


# IBMi (AS/400) Target
Source: https://docs.popsink.com/connectors/target/ibmi



The IBMi (AS/400) Target Connector enables seamless integration between Popsink and IBM i systems, formerly known as AS/400 and iSeries. This connector allows you to stream data from Popsink directly into IBMi databases, physical files, or data areas. It supports both batch updates and near real-time data synchronization, automatically handling data type conversion and file or table creation or updates within the IBMi environment. The IBMi connector is particularly valuable for organizations looking to modernize their legacy systems while maintaining critical business operations on the IBMi platform. It enables real-time data integration between modern cloud or on-premises applications and IBMi systems, facilitating scenarios such as operational data stores, data warehousing, or hybrid cloud architectures. This connector is ideal for businesses in industries like manufacturing, finance, and healthcare that rely on IBMi for core operations but need to integrate with contemporary data ecosystems, enabling them to leverage the reliability and security of IBMi while benefiting from modern data streaming and processing capabilities.

## Compatible versions

|                    |                                                                  |
| ------------------ | ---------------------------------------------------------------- |
| **IBM i (OS/400)** | 7.3, 7.4, 7.5, 7.6                                               |
| **Database**       | Db2 for i, as shipped with the OS release                        |
| **Connectivity**   | JTOpen (JT400) over the IBM i host servers, writing through JDBC |

<Note>IBM i 7.2 and earlier are out of IBM support and are not covered.</Note>


# Iceberg Target
Source: https://docs.popsink.com/connectors/target/iceberg



The Iceberg Target Connector writes your pipeline data to Apache Iceberg tables stored in Azure Blob Storage. Apache Iceberg is an open table format designed for huge analytic datasets, providing ACID transactions, time travel queries, partition evolution, and schema evolution without affecting concurrent reads. This connector streams change events from Popsink directly into your data lake in Iceberg format, making it ideal for lakehouse architectures, large-scale analytics, and machine learning pipelines.

## Compatible versions

|                    |                                                                                          |
| ------------------ | ---------------------------------------------------------------------------------------- |
| **Apache Iceberg** | Table format specification v2                                                            |
| **Writer**         | PyIceberg 0.11                                                                           |
| **Storage**        | Azure Blob Storage / ADLS Gen2                                                           |
| **Readers**        | Any engine that reads Iceberg v2 tables — Spark, Trino, Snowflake, Databricks, DuckDB, … |

## Key Features

* **Iceberg open table format:** full ACID compliance, time travel, and schema evolution on your data lake.
* **Streaming ingestion:** change events are written continuously — no batch jobs to schedule.
* **Flexible Azure authentication:** connection string or Service Principal (SPN), with optional proxy support for restricted networks.

## Prerequisites

* An Azure Blob Storage account with an **existing container** for the Iceberg data.
* Storage credentials: either the storage account **connection string**, or a **Service Principal** with the **Storage Blob Data Contributor** role on the storage account.

## Storage Authentication

Two authentication methods are supported. Pick the one that matches how you administer the storage account.

### Option A — Connection String

Authenticate with an Azure Blob Storage connection string, found in the Azure Portal under **Storage Account → Access keys**.

| Field                 | Description                                                                                                |
| --------------------- | ---------------------------------------------------------------------------------------------------------- |
| **Connection String** | e.g. `DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=...;EndpointSuffix=core.windows.net` |
| **Container Name**    | Existing container where Iceberg table data will be written                                                |

### Option B — Service Principal (SPN)

Authenticate with an Azure Active Directory Service Principal. The Service Principal must have the **Storage Blob Data Contributor** role on the storage account.

| Field                    | Description                                                       |
| ------------------------ | ----------------------------------------------------------------- |
| **Storage Account Name** | Name of the Azure Storage account (e.g. `mystorageaccount`)       |
| **Tenant ID**            | Azure AD tenant ID (Azure Portal → Microsoft Entra ID → Overview) |
| **Client ID**            | Application (client) ID of the registered app                     |
| **Client Secret**        | A client secret generated for the registered app                  |
| **Container Name**       | Existing container where Iceberg table data will be written       |

## Advanced Configuration

| Field          | Required | Description                                                                                                                                                                                                                   |
| -------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **HTTP Proxy** | No       | Optional HTTP(S) proxy URL for the Entra ID token endpoint (`login.microsoftonline.com`). Storage and local traffic stay direct. Only used with SPN authentication — connection-string authentication makes no Entra ID call. |


# Kafka Target
Source: https://docs.popsink.com/connectors/target/kafka



The Kafka Target Connector defines the Apache Kafka cluster where your pipelines write output events, enabling Kafka-to-Kafka replication through Popsink. Use it to relay, fan out, or bridge event streams between clusters — for example from an on-premises cluster to a cloud-managed one — while applying Popsink transformations in between. This target only stores the broker connection and authentication for the sink cluster: source topics and consumer groups are configured on the Kafka source connector and on your subscriptions.

## Table of Contents

1. [Compatible versions](#compatible-versions)
2. [Key Features](#key-features)
3. [Prerequisites](#prerequisites)
4. [Configuration](#configuration)
5. [Security Protocols](#security-protocols)

## Compatible versions

|                          |                                                                                                                                                |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| **Apache Kafka**         | 2.1 and later, including 3.x and 4.x                                                                                                           |
| **Compatible platforms** | Confluent Platform & Confluent Cloud, Amazon MSK, Aiven for Apache Kafka, Redpanda, Azure Event Hubs (Kafka endpoint, Standard tier or higher) |
| **Protocol**             | Kafka wire protocol over `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, or `SASL_SSL`                                                                   |

<Note>Brokers older than 2.1 are not supported — the client negotiates API versions with the cluster and no longer implements the pre-2.1 protocol.</Note>

## Key Features

* **Kafka-to-Kafka pipelines:** bridge event streams between clusters with Popsink transformations applied in flight.
* **Broad authentication support:** PLAINTEXT, SSL, and SASL (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER, GSSAPI).
* **Topic passthrough:** output topic names follow your subscription configuration — no renaming is imposed by the target.

## Prerequisites

* A reachable Kafka cluster to use as the sink.
* Credentials with permission to **produce** to the target topics (and to create them, if they don't pre-exist and auto-creation is disabled on the cluster).

## Configuration

| Field                        | Required  | Description                                                                                                       |
| ---------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------- |
| **Bootstrap Servers**        | Yes       | Comma-separated `host:port` pairs for the target cluster (e.g. `kafka1.example.com:9092,kafka2.example.com:9092`) |
| **Security Protocol**        | Yes       | `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, or `SASL_SSL`                                                               |
| **SASL Mechanism**           | With SASL | `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBEARER`, or `GSSAPI`                                             |
| **SASL Username / Password** | With SASL | Credentials for the chosen SASL mechanism                                                                         |

## Security Protocols

| Protocol         | Encryption | Authentication                     |
| ---------------- | ---------- | ---------------------------------- |
| `PLAINTEXT`      | None       | None — development only            |
| `SSL`            | TLS        | TLS client certificates (optional) |
| `SASL_PLAINTEXT` | None       | SASL                               |
| `SASL_SSL`       | TLS        | SASL — recommended for production  |


# MongoDB Target
Source: https://docs.popsink.com/connectors/target/mongodb



The MongoDB Target Connector allows you to stream data from Popsink into a MongoDB database. This connector is ideal for applications that require a flexible, scalable NoSQL database to store and manage large volumes of semi-structured or unstructured data. By configuring this connector, you can seamlessly integrate Popsink data with your MongoDB collections, enabling real-time updates and synchronization. It's particularly useful for developers and data engineers who need to build dynamic applications, perform real-time analytics, or maintain a consistent data pipeline across various sources. The integration supports various data formats and can handle high throughput, ensuring efficient and reliable data flow into your MongoDB database.

## Compatible versions

|                   |                                |
| ----------------- | ------------------------------ |
| **MongoDB**       | 6.0, 7.0, 8.0                  |
| **Deployments**   | Self-managed and MongoDB Atlas |
| **Compatibility** | Confirmed per deployment       |


# MSSQL Target
Source: https://docs.popsink.com/connectors/target/mssql



The SQL Server Target Connector enables you to stream data from Popsink directly into a SQL Server database. This connector is ideal for scenarios where you need to store and manage relational data in a SQL Server environment. It supports various data formats and allows for real-time updates to SQL Server tables. By configuring this connector, you can ensure that your SQL Server database is continuously synchronized with data from Popsink, facilitating data integration, reporting, and analysis. It's particularly valuable for applications that rely on structured data and require robust querying capabilities provided by SQL Server.

***

# 🗃️ SQL Server Target Connection Setup Guide

This guide provides step-by-step instructions to configure a **SQL Server** as a **target connection** in Popsink, allowing you to stream and load processed data into your Microsoft SQL Server instance.

***

## Compatible versions

|                      |                                                                           |
| -------------------- | ------------------------------------------------------------------------- |
| **SQL Server**       | 2017, 2019, 2022                                                          |
| **Managed services** | Azure SQL Database, Azure SQL Managed Instance, Amazon RDS for SQL Server |
| **Driver**           | Microsoft JDBC Driver 13.4 for SQL Server                                 |

<Note>SQL Server 2016 works on a best-effort basis. SQL Server 2025 has not been validated yet.</Note>

## ✅ Prerequisites

Before configuring the target, ensure you have the following:

* A running Microsoft SQL Server instance (on-premise or cloud-hosted)
* A user account with `CREATE TABLE`, `INSERT` and `UPDATE` privileges on the target database
* Network access from Popsink to the SQL Server host (ensure firewall & security group rules allow inbound traffic)

***

## 🔌 Connection Parameters

When setting up the target in Popsink, you'll need to provide the following:

| Parameter    | Description                                                               |
| ------------ | ------------------------------------------------------------------------- |
| **Host**     | The IP address or DNS name of your SQL Server (e.g., `sql.mycompany.com`) |
| **Port**     | The port number used by SQL Server (default: `1433`)                      |
| **Database** | Name of the target database where data will be written                    |
| **Username** | SQL Server login with appropriate privileges                              |
| **Password** | Password for the specified username                                       |

***

## 🏗️ Setting Up the Connection in Popsink

1. **Navigate to Targets**
   In the Popsink dashboard, go to **Connections** > **Targets** > **Add New Target**.

2. **Choose SQL Server**
   Select **Microsoft SQL Server** from the list of target types.

3. **Enter Connection Details**
   Fill in the form using the connection parameters listed above.

4. **Test Connection**
   Click **Test Connection** to verify that Popsink can reach your SQL Server.

5. **Save**
   Once validated, click **Save**. You can now use this target in your pipelines.

***

## 🧩 Table & Schema Considerations

* Popsink supports **automatic table creation**.
* Source data models must have primary keys if **upserts** are enabled.

***

## 🔐 Security Notes

* Ensure the SQL Server user has **least privilege access** needed to insert and update data only in the required schema/tables.
* For cloud-hosted SQL Server (e.g., Azure SQL Database), ensure Popsink IPs are whitelisted.

***

## 🛠️ Troubleshooting

| Issue                  | Suggested Resolution                                            |
| ---------------------- | --------------------------------------------------------------- |
| Cannot connect to host | Verify firewall rules, DNS resolution, and network reachability |
| Login failed for user  | Confirm username/password and SQL Server authentication mode    |


# MySQL Target
Source: https://docs.popsink.com/connectors/target/mysql



The MySQL Target Connector allows you to transfer data from Popsink into a MySQL database. This connector is well-suited for managing and analyzing structured data using MySQL's relational database capabilities. It supports real-time and batch data updates, ensuring that your MySQL tables are kept current with data from Popsink. You can configure the connector to handle different data formats and specify how data should be mapped to your database schema. This integration is ideal for applications requiring reliable, scalable database solutions for transaction management, reporting, and data analytics.

## Compatible versions

|                      |                                                                       |
| -------------------- | --------------------------------------------------------------------- |
| **MySQL**            | 8.0, 8.4, 9.x                                                         |
| **Storage engine**   | InnoDB                                                                |
| **Managed services** | Amazon RDS & Aurora MySQL, Google Cloud SQL, Azure Database for MySQL |
| **Driver**           | MySQL Connector/J 9.7                                                 |

<Note>MySQL 5.7 has reached end of life and is not covered. MariaDB is a separate product and is not supported.</Note>


# Oracle Target
Source: https://docs.popsink.com/connectors/target/oracle



The Oracle Target Connector enables seamless data transfer from Popsink into an Oracle Database. This connector is ideal for managing and analyzing structured data within Oracle's robust relational database system. It supports both real-time and batch data updates, allowing you to keep your Oracle tables synchronized with Popsink data. The connector facilitates integration by mapping data to your Oracle schema and handling various data formats.

## Compatible versions

|                      |                                                            |
| -------------------- | ---------------------------------------------------------- |
| **Oracle Database**  | 12.1 and later — validated on 19c and 21c                  |
| **Deployments**      | Single instance and RAC; multitenant (CDB/PDB) and non-CDB |
| **Managed services** | Amazon RDS for Oracle, Oracle Base Database Service (OCI)  |
| **Client**           | Oracle Instant Client 21.4                                 |

## Key Features

* **CDC replication:** applies inserts, updates, and deletes from your pipelines to Oracle tables, keeping them in sync with the source.
* **Automatic type mapping:** Popsink handles data type mapping to Oracle formats including `VARCHAR2`, `NUMBER`, `DATE`, and `CLOB`.

## Prerequisites

* A reachable Oracle database.
* A user with **write permissions** on the target schema: `CREATE TABLE`, `INSERT`, `UPDATE`, and `DELETE`.
* **Primary keys configured in the subscription mapping:** Oracle is a JDBC target — primary keys are required so that CDC updates and deletes can be applied correctly.

## Configuration

| Field           | Required | Description                                          |
| --------------- | -------- | ---------------------------------------------------- |
| **Host**        | Yes      | Hostname or IP address of the Oracle server          |
| **Port**        | Yes      | Port the Oracle listener is on (default `1521`)      |
| **Database**    | Yes      | Oracle database name (e.g. `ORCL`)                   |
| **Server Name** | Yes      | Oracle service name (e.g. `ORCLPDB1`)                |
| **Server ID**   | Yes      | Oracle system identifier (SID)                       |
| **User**        | Yes      | Username with write permissions on the target tables |
| **Password**    | Yes      | Password for the Oracle user                         |


# Target Connectors Overview
Source: https://docs.popsink.com/connectors/target/overview

Learn how Popsink delivers real-time data to your destinations and explore each available target connector.

Target connectors push the data captured by Popsink into your downstream systems with low latency. Use this overview to understand the configuration principles shared by every target and quickly navigate to the detailed guide for your destination.

## Deployment checklist

* Confirm that the Popsink deployment has network access to the destination service.
* Prepare credentials with the correct roles or API scopes to write data in the target system.
* Decide how you want Popsink to structure datasets, tables, or collections before activating a pipeline.

## Configuration patterns

All target connector guides explain:

1. **Authentication** – How to securely provide service accounts, API keys, or OAuth tokens.
2. **Schema management** – Recommendations for managing table creation, schema drift, and conflict resolution.
3. **Operational guidance** – Monitoring tips and error handling practices for production workloads.

## Explore destination families

Select the category that matches your target system to access the detailed guides:

### Databases

[Database connectors](./databases) cover relational engines, analytical warehouses, and NoSQL stores.

* [Snowflake](./snowflake)
* [Postgres](./postgres)
* [MySQL](./mysql)
* [Oracle](./oracle)
* [Teradata](./teradata)
* [IBM i](./ibmi)
* [MSSQL](./mssql)
* [ClickHouse](./clickhouse)
* [BigQuery](./bigquery)
* [MongoDB](./mongodb)
* [Cosmos DB](./cosmosdb)
* [Pinecone](./pinecone)

### Data Lakes & Streaming

[Data Lakes & Streaming connectors](./data-lakes-streaming) route your pipelines to lakehouse formats or event buses.

* [Iceberg](./iceberg)
* [Hudi](./hudi)
* [Delta Lake](./deltalake)
* [Pub/Sub](./pubsub)
* [Webhook](./webhook)

### SaaS & Applications

[SaaS & Applications connectors](./saas-applications) deliver Popsink data back into your business tools and custom applications.

* [Generic Application](./application)
* [Slack](./slack)
* [HubSpot](./hubspot)
* [Airtable](./airtable)

Need a specific destination? Contact [support@popsink.com](mailto:support@popsink.com) to discuss a new connector.


# Pinecone Target
Source: https://docs.popsink.com/connectors/target/pinecone



The Pinecone Target Connector allows you to stream data from Popsink into Pinecone, a vector database designed for managing and searching high-dimensional data. This connector is ideal for use cases involving machine learning models, recommendation systems, and semantic search applications. By configuring this connector, you can efficiently integrate Popsink data into Pinecone, facilitating real-time updates and ensuring that your vector database remains current with the latest data. This integration supports various data formats and allows for efficient management of vector embeddings, enhancing your ability to perform advanced search and retrieval operations.

## Compatible versions

|                   |                                                    |
| ----------------- | -------------------------------------------------- |
| **Pinecone**      | Managed service — serverless and pod-based indexes |
| **API**           | Pinecone Data Plane API over HTTPS                 |
| **Compatibility** | Confirmed per deployment                           |


# Postgres Target
Source: https://docs.popsink.com/connectors/target/postgres

Operational guide for connecting Popsink to a PostgreSQL database in batch or streaming mode.

The Popsink Postgres target connector consolidates transactional and analytical data in a PostgreSQL database while honoring your organization's security and governance requirements.

## Compatible versions

|                      |                                                                                 |
| -------------------- | ------------------------------------------------------------------------------- |
| **PostgreSQL**       | 13, 14, 15, 16, 17, 18                                                          |
| **Managed services** | Amazon RDS & Aurora PostgreSQL, Google Cloud SQL, Azure Database for PostgreSQL |
| **Driver**           | PostgreSQL JDBC 42.7                                                            |

<Note>PostgreSQL 12 still works but has reached end of life upstream and is no longer covered.</Note>

## Prerequisites

* **Roles and permissions:** a PostgreSQL role with `CREATE`, `INSERT`, `UPDATE`, `DELETE`, and `ALTER` rights on the target schema, plus permission to execute functions required for `COPY` or `INSERT` operations.
* **Infrastructure settings:** outbound network access from Popsink to port 5432 (or your custom port) with TLS enabled and root certificates imported. See [Compatible versions](#compatible-versions) for the supported PostgreSQL releases.
* **Quotas and limits:** provision disk capacity for at least 150% of the expected daily volume and verify concurrent connection limits (`max_connections`) to accommodate Popsink workers.
* **Monitoring and audits:** enable `pg_stat_statements` and error logging to trace connector activity.

## Configuration

<Tabs>
  <Tab title="Batch ingestion">
    <Steps>
      <Step title="Declare the destination">
        Provide the host, port, database, and target schema in Popsink. Attach a secret containing the user, password, and SSL string (`sslmode=require`) if needed.
      </Step>

      <Step title="Configure the write mode">
        Choose `COPY` for large volumes or `INSERT` for smaller datasets. Configure the maximum batch size (`batch_size`) and duplicate key handling (`upsert_strategy`).
      </Step>

      <Step title="Schedule loads">
        Define the sync cadence and retention rules for intermediate files. Ensure maintenance windows do not overlap with scheduled batches.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Streaming ingestion">
    <Steps>
      <Step title="Enable the real-time channel">
        Select streaming mode in Popsink and increase the allowed connection pool on PostgreSQL to handle long-lived sessions.
      </Step>

      <Step title="Configure the buffer">
        Adjust the memory buffer size (`stream_buffer_mb`) and commit interval (`commit_interval_ms`) to balance latency and cost. Enable acknowledgements to guarantee idempotency.
      </Step>

      <Step title="Secure the streams">
        Apply streaming-specific roles, enable automatic secret rotation, and restrict access to target tables with RLS policies when available.
      </Step>
    </Steps>
  </Tab>
</Tabs>

## Monitoring

* Instrument a dashboard built on `pg_stat_activity`, lock durations (`pg_locks`), and Popsink metrics (throughput, latency, errors).
* Enable job failure notifications in Popsink and forward PostgreSQL logs to your SIEM.
* Configure disk quotas and WAL usage alerts to prevent replication stoppages.

## Best practices

* Normalize data types (UUID, JSONB, timestamp with time zone) to avoid expensive conversions.
* Document target tables with SQL comments to streamline maintenance.
* Orchestrate schema changes outside of loading windows.

<Callout type="warning">
  Avoid setting `synchronous_commit=off` on the Popsink session: if a failure occurs, you may lose transactions that were confirmed on the source.
</Callout>

## FAQ

<AccordionGroup>
  <Accordion title="What if a batch fails because of a unique constraint?">
    Enable `upsert` mode with `ON CONFLICT DO UPDATE` and use a versioning column (`updated_at`) to resolve collisions.
  </Accordion>

  <Accordion title="How do I handle partitioned tables?">
    Declare the parent table in Popsink and ensure the PostgreSQL partitioning strategy routes rows automatically via triggers or `PARTITION BY`.
  </Accordion>

  <Accordion title="Which logs should I review first?">
    Start with Popsink logs to identify the failing step, then correlate with `postgresql.log` and `pg_stat_activity` for SQL details.
  </Accordion>
</AccordionGroup>

<CardGroup>
  <Card title="Monitor targets" href="/target">
    Understand metrics shared across all target connectors.
  </Card>

  <Card title="Alerting Popsink" href="/alerting">
    Configure notifications and webhooks when incidents occur.
  </Card>

  <Card title="Start a pipeline" href="/quickstart">
    End-to-end example to validate your Postgres configuration.
  </Card>
</CardGroup>


# Pubsub Target
Source: https://docs.popsink.com/connectors/target/pubsub



The Pubsub Target Connector enables data streaming from Popsink to a Pub/Sub messaging system, such as Google Cloud Pub/Sub or a similar publish-subscribe service. This connector facilitates real-time data distribution by allowing you to publish messages or events to a specific topic or channel within the Pub/Sub system. By configuring this connector, you can ensure that data from Popsink is delivered in near real-time to subscribers or downstream systems. It's particularly useful for applications requiring asynchronous data processing, event-driven architectures, or scalable message distribution.

## Compatible versions

|                          |                                         |
| ------------------------ | --------------------------------------- |
| **Google Cloud Pub/Sub** | Managed service — no version to pin     |
| **API**                  | Pub/Sub API v1                          |
| **Authentication**       | Google Cloud service account key (JSON) |
| **Compatibility**        | Confirmed per deployment                |


# Target SaaS & Application Connectors
Source: https://docs.popsink.com/connectors/target/saas-applications

Deliver Popsink data products into SaaS tools and custom applications.

This family groups the destinations designed to reinject Popsink data into your business tools or internal applications.

> For shared authentication, governance, and monitoring parameters, refer to the [target connectors overview](./overview).

## Custom applications

* [Generic Application](./application) – Integrate Popsink with your applications through our REST API.

## Collaborative & CRM SaaS

* [Slack](./slack) – Send contextual notifications to your Slack channels.
* [HubSpot](./hubspot) – Enrich HubSpot CRM objects with consolidated data.
* [Airtable](./airtable) – Populate Airtable bases for your operational workflows.

Need to connect another application? [support@popsink.com](mailto:support@popsink.com) can help build a dedicated connector.


# Slack Target
Source: https://docs.popsink.com/connectors/target/slack



The Slack Target Connector enables you to send data from Popsink directly into Slack channels. This integration is useful for keeping teams informed in real time by streaming relevant data, alerts, or updates into their Slack workspace. You can configure the connector to post messages, update existing messages, or trigger specific Slack workflows based on the incoming data. It's particularly valuable for teams that use Slack for collaboration, incident management, or real-time communication.

## Compatible versions

|                    |                                 |
| ------------------ | ------------------------------- |
| **Slack**          | Slack Web API                   |
| **Authentication** | Slack app bot token (OAuth 2.0) |
| **Compatibility**  | Confirmed per deployment        |


# Snowflake Target
Source: https://docs.popsink.com/connectors/target/snowflake



The Snowflake Target Connector is a robust tool designed to facilitate the seamless integration of Change Data Capture (CDC) updates into Snowflake data warehouses. Leveraging Snowpipe streaming functionality, this connector efficiently streams CDC updates into Snowflake, where they are written as inserts. Subsequently, a dynamic table is employed to construct a copy of the source table, ensuring real-time data replication.

## Compatible versions

|                       |                                                                |
| --------------------- | -------------------------------------------------------------- |
| **Snowflake**         | Managed service — Standard edition or higher                   |
| **Required features** | Snowpipe Streaming and dynamic tables                          |
| **Authentication**    | Key-pair (RSA 2048) — password authentication is not supported |
| **Driver**            | Snowflake JDBC 4.3, Snowpipe Streaming SDK 1.6                 |

<Note>Dynamic tables require a running warehouse for their refresh; size it for your change volume.</Note>

## Key Features

* **CDC Streaming:** utilizes Snowpipe streaming to deliver real-time changes from the source system.
* **Insert Operation:** writes CDC updates as inserts into the target Snowflake database, preserving the full change history.
* **Dynamic Table:** constructs a dynamic table to mirror the source table, enabling easy access to replicated data.
* **Time Travel Support:** retains historical data for time travel queries, facilitating data analysis and auditing.
* **Key-Pair Authentication:** relies on Snowflake's key-pair authentication mechanism — no passwords.
* **Schema evolution:** new columns are added automatically as the source schema evolves.

## Prerequisites

* A Snowflake account where you wish to sync your data.
* A **role** with the required permissions on the target database and schema:
  * `USAGE` on the database and schema
  * `USAGE` on the warehouse used to run the connector
  * `CREATE TABLE` on the schema
  * `CREATE DYNAMIC TABLE` on the schema
  * `INSERT`, `UPDATE`, `DELETE` on target tables
* A **key pair** (no passphrase) for authentication. To set it up:

  1. Generate the key pair:
     ```bash theme={null}
     openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt
     openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
     ```
  2. Register the public key on the Snowflake user:
     ```sql theme={null}
     ALTER USER <your_user> SET RSA_PUBLIC_KEY='<public_key_body>';
     ```
  3. Copy the private key **without** the header and footer lines:
     ```
     -----BEGIN PRIVATE KEY-----
     -----END PRIVATE KEY-----
     ```

## Configuration

| Field                   | Required | Description                                                                                                                            |
| ----------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| **Account**             | Yes      | Snowflake account identifier (e.g. `myorg-myaccount` or `account.region.cloud`)                                                        |
| **User**                | Yes      | Username configured with key-pair authentication                                                                                       |
| **Database**            | Yes      | Database where tables will be created (e.g. `ANALYTICS_DB`)                                                                            |
| **Schema**              | Yes      | Schema within the database (e.g. `PUBLIC`)                                                                                             |
| **Role**                | Yes      | Role to use for the session — must have write access to the target schema                                                              |
| **Private Key**         | Yes      | The private key body, without headers                                                                                                  |
| **Enable Before/After** | Yes      | When enabled, stores both the **before** and **after** images of each change event, allowing field-level change tracking in the target |

## How It Works

For each subscription, Popsink creates tables in the target schema and applies CDC events to keep Snowflake in sync with the source. Updates are streamed as inserts via Snowpipe streaming, and a dynamic table exposes the latest state of each table. Schema evolution is handled automatically — new columns are added as the source schema evolves.


# Teradata Target
Source: https://docs.popsink.com/connectors/target/teradata

Deploying the Teradata target connector to load analytical volumes from Popsink.

The Teradata target connector powers your analytical environments by leveraging Teradata's MPP parallelization and optimization features.

## Compatible versions

|                      |                                                |
| -------------------- | ---------------------------------------------- |
| **Teradata Vantage** | 17.x and later                                 |
| **Protocol**         | Teradata SQL over port `1025` with TLS enabled |
| **Deployments**      | VantageCloud and on-premises                   |

## Prerequisites

* **Accounts and roles:** a Teradata user with `CREATE TABLE`, `INSERT`, `UPDATE`, `DELETE` privileges and access to the TPT utility when used.
* **Infrastructure:** network connectivity between Popsink and the Teradata port (`1025` by default) with TLS encryption enabled. See [Compatible versions](#compatible-versions) for the supported Vantage releases.
* **Capacity and quotas:** sufficient `AMP` allocation to absorb write spikes, session limits sized for Popsink jobs, and appropriately dimensioned permanent/swap space.
* **Monitoring:** access to the `DBC.ResUsage*` views and `DBC.QryLog` to collect performance and error metrics.

## Configuration

<Tabs>
  <Tab title="Batch loads">
    <Steps>
      <Step title="Declare the target">
        Enter the host, system (`DBCName`), and database name in Popsink. Attach a secret containing the service account and, if required, the client certificate.
      </Step>

      <Step title="Choose the write mode">
        Select the `TPT` utility for large volumes or `FastLoad` for empty tables. Configure the block size (`blocksize`) and `errorlimit` level.
      </Step>

      <Step title="Optimize batches">
        Configure the primary key or secondary indexes, enable multivalue compression, and define the automatic restart policy for failures.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Continuous flows">
    <Steps>
      <Step title="Enable streaming mode">
        Select the `continuous loads` option in Popsink and confirm that the `Teradata Parallel Transporter` component is sized for long-running sessions.
      </Step>

      <Step title="Manage latency">
        Tune the `checkpoint_interval` and `commit_frequency` parameters to balance throughput and AMP consumption.
      </Step>

      <Step title="Secure access">
        Restrict the Popsink role to target tables, enable LDAP/Kerberos authentication when available, and log requests through `DBC.DBQLogTbl`.
      </Step>
    </Steps>
  </Tab>
</Tabs>

## Monitoring

* Track CPU and I/O usage through `DBC.ResUsageScpu` and `DBC.ResUsageSvpr`, complemented by Popsink metrics (throughput, retries, latency).
* Configure alerts when `2689` (duplicate row) or `2641` (AMP down) errors exceed a defined threshold.
* Retain TPT and FastLoad logs to simplify diagnostics if a job aborts.

## Best practices

* Partition large tables with `PRIMARY INDEX` values aligned to the distribution key.
* Order columns to maximize block-level compression.
* Align Teradata maintenance windows with Popsink schedules to avoid collisions.

<Callout type="warning">
  Disable the `CHECKPOINT OFF` option only for exceptional jobs: without a recovery point, restarting a large load can severely impact production.
</Callout>

## FAQ

<AccordionGroup>
  <Accordion title="How should I handle duplicate errors (2689)?">
    Enable the Popsink upsert option or add a staging step with deduplication before the final insert.
  </Accordion>

  <Accordion title="What if an AMP becomes unavailable?">
    Pause the job, work with the DBA team to restore the AMP, then restart the connector once the cluster state is validated.
  </Accordion>

  <Accordion title="How do I audit writes?">
    Use `DBC.QryLog` to trace executed queries and export Popsink logs to your centralized SIEM solution.
  </Accordion>
</AccordionGroup>

<CardGroup>
  <Card title="Monitor targets" href="/target">
    Configure dashboards to track the health of Teradata and other destinations.
  </Card>

  <Card title="Alerting Popsink" href="/alerting">
    Receive alerts when TPT or FastLoad jobs fail.
  </Card>

  <Card title="Pipeline examples" href="/quickstart">
    Reuse a reference pipeline to accelerate your Teradata deployments.
  </Card>
</CardGroup>


# Webhook Target
Source: https://docs.popsink.com/connectors/target/webhook



The Webhook Target Connector delivers data from Popsink to any HTTP endpoint via webhook. It is the most efficient way to build a direct integration between Popsink and a downstream service: select existing streams from your Popsink catalog, and each event is forwarded as an HTTP request to your configured URL.

## Compatible versions

|              |                                           |
| ------------ | ----------------------------------------- |
| **Protocol** | HTTP/1.1 over TLS 1.2 or later            |
| **Payload**  | JSON                                      |
| **Endpoint** | Any HTTPS endpoint reachable from Popsink |

## Key Features

* **Push delivery:** events are forwarded to your endpoint as they flow through the pipeline — no polling required.
* **Flexible authentication:** token sent as a Bearer header or as a URL query parameter.
* **Configurable method:** deliver events with `POST` (payload in the body, recommended) or `GET`.

## Prerequisites

* An HTTP(S) endpoint reachable from Popsink that accepts incoming requests.

## Configuration

| Field                   | Required | Description                                                     |
| ----------------------- | -------- | --------------------------------------------------------------- |
| **Host**                | Yes      | Target webhook URL (e.g. `https://api.myservice.com/events`)    |
| **Token**               | No       | Authentication token, sent according to the authentication type |
| **Authentication Type** | Yes      | `BEARER` (default) or `URL` — see below                         |
| **Request Method**      | Yes      | `POST` (default, recommended) or `GET`                          |

## Authentication Types

| Type       | Description                                              |
| ---------- | -------------------------------------------------------- |
| **BEARER** | Token sent in the `Authorization: Bearer <token>` header |
| **URL**    | Token sent as a `?token=<token>` query parameter         |

## Request Methods

| Method   | Description                                          |
| -------- | ---------------------------------------------------- |
| **POST** | Event payload sent in the request body — recommended |
| **GET**  | Request sent without a body                          |


# Popsink Deployment Options
Source: https://docs.popsink.com/deployment/deployment



Popsink supports multiple deployment models so that you can run the same core engine, with the same features and performance, in the mode that fits your organization’s requirements. The primary deployment models are:

1. **SaaS (fully managed)**
2. **Bring Your Own Cloud (BYOC)**
3. **On-Premises (on-prem)**
4. **Air-Gapped / Isolated**

***

## Overview

Popsink’s deployment architecture is designed to adapt to different security postures, regulatory constraints, and operational preferences. Regardless of the model chosen, you benefit from the same core capabilities (e.g. data ingestion, replication, pipeline processing, observability). The choice of deployment impacts responsibilities for infrastructure, upgrades, connectivity, and control.

High level comparison:

| Deployment Mode       | Infrastructure Ownership                      | Connectivity Requirements                              | Upgrade / Patching Responsibility | Typical Use Cases                                   |
| --------------------- | --------------------------------------------- | ------------------------------------------------------ | --------------------------------- | --------------------------------------------------- |
| SaaS                  | Popsink-managed cloud                         | Outbound connectivity required                         | Popsink handles                   | Fast startup, low ops, non-restrictive environments |
| BYOC                  | Customer’s cloud account                      | Outbound/inbound as required                           | Shared (Popsink + Customer)       | Organizations wanting control over data, keys, cost |
| On-Premises           | Customer-managed data center or private cloud | Standard internal networking, limited external contact | Customer (with Popsink guidance)  | Enterprises with strict internal hosting needs      |
| Air-Gapped / Isolated | Customer-managed, isolated network            | No outbound connectivity allowed                       | Offline, signed update bundles    | Regulated environments, high-security zones         |

***

## Deployment Models — Detailed Descriptions

Below are more detailed descriptions of each deployment model, including features, constraints, and prerequisites.

### 1. SaaS (Fully Managed)

**Description**

In this model, Popsink hosts and manages the full stack (compute, storage, networking, orchestration) in a cloud environment. The user interacts via APIs, consoles, or UI but does not need to provision or manage infrastructure.

**Key Features & Benefits**

* Zero ops: Setup can be done in hours
* Automatic upgrades and patches
* Elastic scaling under predictable SLAs
* Data boundary controls: region selection, private networking options

**Responsibilities**

* Popsink: manage infrastructure, scaling, updates, high availability
* Customer: data modeling, configuration, access controls, inbound/outbound integration

<img alt="image" />

### 2. Bring Your Own Cloud (BYOC)

**Description**

This model installs Popsink within the customer’s own cloud account (e.g. AWS, GCP, Azure). All data, keys, and spend remain under the customer's control. The deployed components run within the customer’s VPC and use existing identity and key management systems.

**Key Features & Benefits**

* Data stays inside your VPC
* VPC-only traffic (internal networking)
* Single-tenant cost visibility
* More control over infrastructure and governance

**Responsibilities**

* Customer: provide the target cloud account, networking setup, VPC connectivity
* Popsink: deliver the deployment artifacts, containers, orchestration, guidance for upgrades, monitoring

<img alt="image" />

### 3. On-Premises (On-Prem)

**Description**

Popsink can run within a customer’s data center or private cloud environment, in a Kubernetes cluster.

**Key Features & Benefits**

* Supports Kubernetes or VM installations
* Integrates with enterprise SSO
* Minimal external dependencies

**Responsibilities**

* Customer: provide the physical/virtual infrastructure, storage, networking, identity, monitoring
* Popsink: provide the software, configuration, deployment instructions, integration with identity and observability systems

<img alt="image" />

### 4. Air-Gapped / Isolated

**Description**

In environments that prohibit outbound connectivity (e.g. high-security, classified, regulatory environments), Popsink supports an air-gapped deployment. All operations, updates, and communications happen without external network connectivity. Updates are delivered via signed offline bundles, and the environment supports strict change control and auditing.

**Key Features & Benefits**

* No outbound connectivity required
* Offline, signed update bundles (auditable)
* Controlled, auditable deployments
* End-to-end policy enforcement

**Responsibilities**

* Customer: maintain the isolated network, provide mechanisms for secure update transfer, ensure auditability
* Popsink: provide offline deployment artifacts, signature verification, rollback and upgrade procedures

<img alt="image" />


# Popsink Deployment Kubernetes Requirements
Source: https://docs.popsink.com/deployment/installation



## Prerequisites for Popsink Kubernetes Deployment

Before deploying Popsink on a Kubernetes cluster, ensure that your environment meets the following requirements.

### Mandatory Requirements

* **Kubernetes Cluster**

  * A Kubernetes cluster with a minimum of **2 worker nodes**
  * Each node must have at least:

    * **4 vCPUs**
    * **16 GB of RAM**
    * **amd64 architecture**

* **Ingress Controller**

  * An ingress controller must be installed and configured
  * Examples: **Traefik**, **Nginx Ingress Controller**, **Istio Ingress Gateway**

* **Storage**

  * SSD-based persistent storage
  * Minimum available disk space: **200 GB**

* **Helm**

  * **Helm 3** must be installed
  * Installation and upgrades of PopSink are performed using a **Helm chart**

### Optional Requirements

* **Object Storage**

  * An **S3-compatible storage** service (or equivalent)

* **Autoscaling**

  * **Kubernetes Cluster Autoscaler** (or equivalent node autoscaling solution)


# Install Popsink with Helm for partners
Source: https://docs.popsink.com/deployment/partners

Self-host the Popsink data plane on Kubernetes using the official Helm chart.

This page is intended for **integrators** who install Popsink in their own
Kubernetes cluster. The chart deploys a self-hosted **Popsink data plane** —
the runtime that moves data from your sources to your targets. It connects to
a **control plane** (managed by Popsink at `control-plane.popsink.com`, or
your own) which orchestrates and monitors the deployment.

## Architecture overview

The chart deploys the following components:

| Component          | Purpose                                                       | Default     |
| ------------------ | ------------------------------------------------------------- | ----------- |
| `data-plane`       | Main API + UI. Talks to the control plane, manages connectors | **enabled** |
| `tansu`            | Stateless Kafka-compatible broker, S3-backed                  | **enabled** |
| `karapace`         | Schema Registry (Avro / JSON Schema) for Tansu                | **enabled** |
| `metrics-exporter` | Exports pipeline & connector metrics to the control plane     | **enabled** |
| `kafka-ui`         | Web UI for Kafka inspection                                   | **enabled** |
| `postgresql`       | Bitnami sub-chart, in-cluster Postgres for the data-plane DB  | **enabled** |

You can disable any sub-component and bring your own:

* **External Kafka** — `tansu.enabled=false`, configure `defaultKafka.*`
* **External Schema Registry** — `karapace.enabled=false`, configure `schemaRegistry.*`
* **External PostgreSQL** — `postgresql.enabled=false`, configure `externalDatabase.*`

<Warning>
  The chart does **not** manage ingress. You must set up your own ingress
  controller (Traefik, NGINX, Istio…) and point a DNS record to it.
  See [Ingress](#ingress) below.
</Warning>

## Prerequisites

### Cluster

| Requirement       | Minimum                                                  |
| ----------------- | -------------------------------------------------------- |
| Kubernetes        | `1.23+`                                                  |
| Helm              | `3.8.0+`                                                 |
| Worker nodes      | 2 nodes, `amd64`                                         |
| Per-node          | 4 vCPU, 16 GB RAM                                        |
| Persistent volume | SSD-backed StorageClass, ≥ 200 GB available              |
| Ingress           | Traefik / NGINX / Istio (or any IngressClass-compatible) |
| Object storage    | S3-compatible bucket (recommended for Tansu)             |

### Access to Popsink images

All images are published to a private GAR registry:

```
europe-west1-docker.pkg.dev/popsink-common-438615/onprem
```

Popsink will provide you with a service-account JSON token to pull these
images.

<Tabs>
  <Tab title="Chart-managed pull secret">
    Let the chart create the `imagePullSecret` for you:

    ```yaml theme={null}
    imagePullSecret:
      create: true
      registry: europe-west1-docker.pkg.dev/popsink-common-438615/onprem
      token: |
        <base64-encoded service account JSON, or the JSON itself>
    ```
  </Tab>

  <Tab title="Bring your own pull secret">
    Create the secret yourself, then reference it via `global.imagePullSecrets`:

    ```bash theme={null}
    kubectl create secret docker-registry popsink-registry \
      --docker-server=europe-west1-docker.pkg.dev \
      --docker-username=_json_key \
      --docker-password="$(cat key.json)" \
      -n popsink
    ```

    ```yaml theme={null}
    global:
      imagePullSecrets:
        - popsink-registry
    ```
  </Tab>
</Tabs>

## Onboarding flow (control plane)

Before installing the chart, you need a **deployment ID** and a
**deployment JWT token** issued by the control plane.

<Steps>
  <Step title="Open the control plane">
    Log into `https://control-plane.popsink.com` (or your own control plane).
  </Step>

  <Step title="Create a self-hosted deployment">
    Open **Deployments → New deployment** and pick **Self-hosted**.
  </Step>

  <Step title="Fill the wizard">
    * Optional pre-fill of PostgreSQL credentials (not stored).
    * **Retention strategy** — pick S3-compatible storage (recommended) or
      PostgreSQL.
    * **FQDN** — the public URL where the data plane will be reachable
      (e.g. `https://popsink.your-company.com`).
  </Step>

  <Step title="Copy the values snippet">
    The wizard outputs a `values-control-plane.yaml` snippet containing
    `controlPlaneUrl`, `deploymentId` and `deploymentJwtToken`.
    **Use these values verbatim** — they identify and authenticate your data
    plane against the control plane.
  </Step>

  <Step title="Wait for connection">
    Keep the wizard open. After `helm install`, the control plane will switch
    from **Awaiting connection…** to **Connected** once the data plane
    registers.
  </Step>
</Steps>

## Required secrets to generate yourself

Beyond the values handed by the control plane, you must generate four secrets
locally:

| Secret                                                      | Format                                         | How to generate                                                                             |
| ----------------------------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------- |
| `adminCredentials.username` / `password`                    | any                                            | Pick a strong password                                                                      |
| `jwt.secret`                                                | random string ≥ 32 chars                       | `openssl rand -base64 48`                                                                   |
| `connectorConfigEncryptionKey.key`                          | URL-safe base64-encoded **32-byte** Fernet key | `python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"` |
| `postgresql.auth.password` (or `externalDatabase.password`) | any                                            | `openssl rand -base64 24`                                                                   |

<Warning>
  These four secrets encrypt connector credentials, sign user sessions and
  protect the admin login. **Lose them and you lose access to your stored
  connector configs**. Back them up in your secret manager.
</Warning>

If you prefer to manage secrets out of band (Vault, Sealed Secrets, External
Secrets Operator…), set the corresponding `existingSecret` field instead.

## Tansu storage (S3)

Tansu is a stateless Kafka broker that stores log segments in S3. You must
provide an S3-compatible bucket and a way to authenticate.

<CodeGroup>
  ```yaml AWS — IRSA (recommended on EKS) theme={null}
  tansu:
    storage:
      engine: "s3://my-popsink-bucket/kafka/"
      aws:
        region: eu-west-1
        irsaRoleArn:    arn:aws:iam::123456789012:role/popsink-tansu
        assumedRoleArn: arn:aws:iam::123456789012:role/popsink-tansu
  ```

  ```yaml AWS — static credentials theme={null}
  tansu:
    storage:
      engine: "s3://my-popsink-bucket/kafka/"
      aws:
        region: eu-west-1
        accessKeyId: AKIA…
        secretAccessKey: <secret>
  ```

  ```yaml MinIO / non-AWS S3 theme={null}
  tansu:
    storage:
      engine: "s3://my-popsink-bucket/kafka/"
      aws:
        region: us-east-1
        endpoint: http://minio.minio.svc.cluster.local:9000
        allowHttp: true
        accessKeyId: minio
        secretAccessKey: minio123
  ```
</CodeGroup>

The bucket must exist, be writable by the configured identity, and ideally
have **object versioning** enabled.

<Note>
  If you don't want Tansu, set `tansu.enabled=false` and `karapace.enabled=false`,
  then point the data plane at your existing Kafka:

  ```yaml theme={null}
  tansu:    { enabled: false }
  karapace: { enabled: false }

  defaultKafka:
    bootstrapServer: kafka.example.com:9093
    securityProtocol: SASL_SSL
    saslMechanism: SCRAM-SHA-512
    saslUsername: popsink
    saslPassword: <secret>
    caCert: |-
      -----BEGIN CERTIFICATE-----
      …
    cert: ""
    key:  ""

  schemaRegistry:
    url: https://schema-registry.example.com
    username: popsink
    password: <secret>
  ```
</Note>

## Ingress

The chart's built-in `ingress.enabled` flag is **off by default** and we
recommend leaving it that way: ingress in production usually needs
company-specific annotations (cert-manager, WAF, allow-lists). Provide your
own `Ingress` resource that exposes:

* the data-plane HTTP service (`<release>-data-plane`, port `80`),
* on the FQDN you declared in the control plane (`ingressUrl`).

Example with NGINX + cert-manager:

```yaml theme={null}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: popsink
  namespace: popsink
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts: [popsink.your-company.com]
      secretName: popsink-tls
  rules:
    - host: popsink.your-company.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: <release>-data-plane
                port: { number: 80 }
```

If you want the chart to render an `Ingress` for you, set `ingress.enabled=true`
and configure `ingress.hostname`, `ingress.ingressClassName`, `ingress.tls`.

<Tip>
  The value of **`ingressUrl`** in the chart **must** match the public URL the
  control plane redirects users to (it is also used for OAuth-style callbacks):

  ```yaml theme={null}
  ingressUrl: "https://popsink.your-company.com"
  ```
</Tip>

## Putting it together — minimal `values.yaml`

```yaml values.yaml theme={null}
# ───── Identity (from the control-plane wizard) ─────
controlPlaneUrl:         https://control-plane-api.popsink.com/api
controlPlaneFrontendUrl: https://control-plane.popsink.com
deploymentMode: SELF_HOSTED
deploymentId:   <uuid-from-wizard>
deploymentJwtToken:
  token: <jwt-from-wizard>

# ───── Public URL of this data plane ─────
ingressUrl: https://popsink.your-company.com

# ───── Image pull ─────
imagePullSecret:
  create: true
  token: <gar-service-account-json>

# ───── Secrets you generated ─────
adminCredentials:
  username: admin
  password: <strong-password>
jwt:
  secret: <openssl rand -base64 48>
connectorConfigEncryptionKey:
  key: <fernet-key>

# ───── In-cluster Postgres (default) ─────
postgresql:
  enabled: true
  auth:
    password: <strong-password>
  primary:
    persistence:
      size: 50Gi
      storageClass: <your-ssd-class>

# ───── Tansu + S3 ─────
tansu:
  enabled: true
  storage:
    engine: "s3://my-popsink-bucket/kafka/"
    aws:
      region: eu-west-1
      irsaRoleArn:    arn:aws:iam::123:role/popsink-tansu
      assumedRoleArn: arn:aws:iam::123:role/popsink-tansu

karapace:
  enabled: true
```

## Install

```bash theme={null}
# 1. Create the namespace
kubectl create namespace popsink

# 2. Install (chart is published as an OCI artifact)
helm install popsink \
  oci://ghcr.io/popsink/charts/data-plane \
  -n popsink \
  -f values.yaml \
  --version <chart-version>
```

<Tip>
  Use `helm search repo` or check the chart's `Chart.yaml` (`version:`) to pin
  the chart version. Pinning avoids surprise upgrades.
</Tip>

Watch the rollout:

```bash theme={null}
kubectl -n popsink get pods -w
```

When all pods are `Ready`, head back to the control-plane deployment page —
the status should flip from **Awaiting connection…** to **Connected** within
a minute. Open `ingressUrl` and log in with the `adminCredentials` you set.

## Production checklist

<AccordionGroup>
  <Accordion title="Versions & images">
    * Pin the chart version (`--version`) and the application image
      (`image.tag`) — never deploy `latest`.
  </Accordion>

  <Accordion title="State">
    * Use **external** PostgreSQL (`postgresql.enabled=false`,
      `externalDatabase.*`) backed by managed snapshots, not the in-cluster
      Bitnami chart.
    * Use **external** S3 with versioning + lifecycle policy for Tansu.
  </Accordion>

  <Accordion title="Resources & availability">
    * Set explicit `resources.requests/limits` on every component (defaults
      target small-medium clusters; tune for your workload).
    * Configure `replicaCount ≥ 2` on `data-plane` (default), keep
      `tansu.replicaCount=3` (default).
    * Enable `pdb.create: true` (default) — at least 1 pod stays during node
      drains.
  </Accordion>

  <Accordion title="Secrets">
    * Provide `imagePullSecrets` via a sealed secret / external secrets
      controller, not plain values.
    * Back up the four secrets (admin credentials, JWT secret, Fernet key,
      DB password) in your secret manager.
  </Accordion>

  <Accordion title="Safety switches">
    * Keep `allowDesignLogin: false` (default) and `pipelineMode: false`
      unless explicitly told otherwise — these are dev-only switches.
  </Accordion>

  <Accordion title="Network">
    * Ingress: TLS-only, cert-manager (or equivalent), restrict source ranges
      if your data plane is internet-exposed only for known IPs.
  </Accordion>
</AccordionGroup>

## Upgrade

```bash theme={null}
helm upgrade popsink \
  oci://ghcr.io/popsink/charts/data-plane \
  -n popsink \
  -f values.yaml \
  --version <new-chart-version>
```

The data plane runs DB migrations on startup. Watch a pod's logs:

```bash theme={null}
kubectl -n popsink logs -l app.kubernetes.io/component=data-plane -f
```

## Uninstall

```bash theme={null}
helm uninstall popsink -n popsink
```

<Warning>
  By default the **PostgreSQL PVC and Tansu/S3 data are kept**
  (`postgresql.primary.persistentVolumeClaimRetentionPolicy.whenDeleted=Retain`,
  S3 buckets are external). Delete them manually if you want a clean slate:

  ```bash theme={null}
  kubectl -n popsink delete pvc -l app.kubernetes.io/instance=popsink
  aws s3 rm s3://my-popsink-bucket/kafka/ --recursive   # ⚠ destroys all topics
  ```
</Warning>

## Troubleshooting

| Symptom                                                           | Likely cause                                                                                               |
| ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Control plane stays on "Awaiting connection…"                     | Wrong `controlPlaneUrl` / `deploymentId` / `deploymentJwtToken`, or egress to the control plane is blocked |
| `data-plane` pods crash-loop on `connector-config-encryption-key` | Fernet key is not a valid URL-safe base64-encoded 32-byte string                                           |
| `tansu` pods `CrashLoopBackOff` with S3 errors                    | `storage.engine`, `region`, or IAM role/credentials are wrong; bucket missing                              |
| `ImagePullBackOff` on every pod                                   | `imagePullSecret.token` not set / not a valid GAR service-account JSON                                     |
| Login at `ingressUrl` fails with redirect loop                    | `ingressUrl` doesn't match the public URL fronted by your ingress                                          |
| Workers (connectors) never become `LIVE`                          | The data-plane pod cannot reach Tansu / Karapace inside the cluster — check `NetworkPolicy`                |
| Ingress 502 on `/api/livez`                                       | Service name in your `Ingress` doesn't match `<release>-data-plane`                                        |

For deeper logs:

```bash theme={null}
kubectl -n popsink logs -l app.kubernetes.io/component=data-plane --tail=200
kubectl -n popsink logs -l app.kubernetes.io/component=tansu       --tail=200
kubectl -n popsink logs -l app.kubernetes.io/component=karapace    --tail=200
```

## Reference — most useful values

A full list lives in `values.yaml` (annotated with `@param` blocks). The most
common knobs are:

| Path                                            | What it does                              |
| ----------------------------------------------- | ----------------------------------------- |
| `controlPlaneUrl` / `controlPlaneFrontendUrl`   | Control plane API and UI URLs             |
| `deploymentId`, `deploymentJwtToken.token`      | Identity vs. control plane                |
| `ingressUrl`                                    | Public URL of this data plane             |
| `image.tag`                                     | Pin the data-plane image version          |
| `replicaCount`                                  | Data-plane API replicas (default 2)       |
| `resources`                                     | Data-plane CPU / memory requests / limits |
| `imagePullSecret.*` / `global.imagePullSecrets` | How to authenticate to the GAR registry   |
| `adminCredentials.*`                            | First admin login                         |
| `jwt.secret`                                    | Signs user session tokens                 |
| `connectorConfigEncryptionKey.key`              | Encrypts connector credentials at rest    |
| `tansu.enabled`, `tansu.storage.*`              | In-cluster Kafka or BYO; S3 backend       |
| `karapace.enabled` / `schemaRegistry.*`         | In-cluster Schema Registry or BYO         |
| `postgresql.enabled` / `externalDatabase.*`     | In-cluster Postgres or BYO                |
| `metricsExporter.*`                             | Pushes metrics to the control plane       |
| `kafkaUi.*`                                     | Optional Kafka inspection UI              |
| `pdb.create`                                    | PodDisruptionBudget (default `true`)      |
| `autoscaling.hpa.*`, `autoscaling.vpa.*`        | HPA / VPA — disabled by default           |

## Further reading

<CardGroup>
  <Card title="Installation prerequisites" icon="server" href="https://docs.popsink.com/deployment/installation">
    Hardware, ingress and storage requirements for a Popsink-ready cluster.
  </Card>

  <Card title="Self-hosted onboarding" icon="cloud-arrow-up" href="https://docs.popsink.com/deployment/selfhosted">
    The control-plane wizard that issues your `deploymentId` and JWT token.
  </Card>
</CardGroup>


# Popsink Kubernetes Deployment
Source: https://docs.popsink.com/deployment/selfhosted

Step-by-step guide to creating, configuring, and connecting a self-hosted deployment.

## Prerequisites

Ensure your infrastructure meets the following requirements before starting:

* **Kubernetes 1.23+**
* **Helm 3.8.0+**
* Support for a **Persistent Volume (PV) provisioner**

***

## 1. Create a New Deployment

1. Navigate to **Deployments** from the sidebar.
2. Click **New Deployment**.

<img alt="image" />

### Deployment Settings

Provide the required information:

* **Deployment name** – Choose a recognizable name for your environment.
* **Deployment URL** – Automatically generated from your deployment name.
* **Deployment method** – Select **Self-hosted**.

Click **Configure Self-Hosted** to continue.

> 💡 You can request a new region or provider if your preferred option is not listed.

<img alt="image" />

***

## 2. Configure the Deployment

This step bootstraps your deployment on a Kubernetes cluster using Helm.

> **Note:** The information entered here is only used to pre-fill your YAML in the next step and is not permanently stored. You can skip this step and manually edit the YAML if preferred.

### Optional Database Setup

You may enable a **PostgreSQL database** during setup if required by your deployment.

***

## 3. Select a Retention Strategy

Choose how your data will be stored:

* **S3-Compatible Storage** – Recommended for scalable object storage.
* **PostgreSQL Database** – Suitable for structured retention needs.

***

## 4. Configure the TLD (FQDN)

Provide the **Fully Qualified Domain Name (FQDN)** used to generate your Helm values.

> ⚠️ The chart does **not** manage ingress configuration. You must configure it separately.

**Example:**

```

demo-popsink.tld

```

Make sure to set the `ingressurl` field in your `values.yaml` to this FQDN.

Click **Create Chart** when ready.

<img alt="image" />

***

## 5. Install the Helm Chart

After generating the configuration, follow the provided commands.

<img alt="image" />

### Install Command

```bash theme={null}
helm install release-name oci://ghcr.io/popsink/charts/data-plane --version 0.1.0-alpha.8
```

***

## 6. Review and Customize Helm Values

A default configuration file is generated containing:

* Global FQDN
* PostgreSQL enablement flag
* Retention type
* Deployment metadata

Download or copy the configuration and adjust it as needed before applying it to your cluster.

### Required Helm Values

Set the following values **exactly as provided by Popsink** to allow the data-plane to register and communicate with the control-plane:

* `controlPlaneUrl`
* `controlPlaneFrontendUrl`
* `deploymentId`
* `deploymentJwtToken`

***

## Managing Secrets

Sensitive values (such as tokens or credentials) can be provided in two ways:

1. **Directly in `values.yaml`**
   Simple to configure, but ensure proper access control and avoid committing secrets to Git.

2. **Reference an existing Kubernetes Secret (Recommended for production)**
   Keeps sensitive data out of Helm files and aligns with Kubernetes best practices.

***

## Image Registry Authentication

Popsink distributes container images through a private registry.
To pull these images, you must configure authentication using an imagePullSecret.

Popsink will provide a registry token that you must add to your Helm configuration.

When finished, click **Mark as Complete**.

***

## 7. Await Connection

After installation, the platform waits for your self-hosted deployment to connect.
You will see an **"Awaiting connection…"** status until the cluster successfully registers.

<img alt="image" />

### Troubleshooting

If the connection is not established:

* Verify that the Helm installation completed successfully.
* Confirm your ingress and DNS are correctly configured.
* Ensure the cluster has outbound network access.

***

## Production Recommendations

For most production deployments, Popsink recommends:

* **Using S3-compatible object storage** for retention (scalable and durable).
* **Providing an external PostgreSQL database** instead of running Postgres inside the cluster.

These practices generally improve reliability, scalability, and operational simplicity.

***

## Advanced Configuration

### Custom JWT Secrets (Data-Plane)

For enhanced security and compliance, you can configure the deployment to **use your own JWT secrets** instead of the defaults.

This approach provides tighter control over:

* Token signing and rotation policies
* Secret ownership
* Security posture


# Single VM Deployment
Source: https://docs.popsink.com/deployment/singlevm

Deploy Popsink on a single virtual machine — ideal for evaluation and lightweight usage.

## Overview

The **Single VM** deployment packages the entire Popsink data-plane into a single virtual machine image. It is designed for:

* **Proof-of-concept and evaluation** — get Popsink running in your environment quickly, without managing a distributed infrastructure.
* **Lightweight workloads** — small teams or low-traffic environments where horizontal scalability is not a requirement.

> This deployment model is intentionally simple. For high-availability or large-scale production use, see the [Kubernetes deployment](./selfhosted) instead.

The VM is distributed as a pre-built image (`.qcow2` for KVM/QEMU, `.ova` for VMware vSphere). No Kubernetes knowledge is required to operate it.

***

## VM Platform

| Property              | Value                                                         |
| --------------------- | ------------------------------------------------------------- |
| **Operating system**  | Ubuntu 24.04 LTS "Noble Numbat"                               |
| **Architecture**      | x86-64 (amd64)                                                |
| **Base image**        | Ubuntu Server Cloud Image (`noble-server-cloudimg-amd64.img`) |
| **Container runtime** | k3s v1.35.5+k3s1 (single-node Kubernetes)                     |
| **Disk format**       | qcow2 (KVM/QEMU) / OVA (VMware vSphere)                       |
| **Disk size**         | 30 GiB (sparse — actual usage is much lower at rest)          |

All container images — k3s's own runtime images and the Popsink data-plane images alike — are bundled in the VM image. The VM never needs internet access to pull images, at first boot or afterwards. It does, however, need to reach the external dependencies listed below.

***

## Prerequisites

### Hardware

| Parameter | Minimum |
| --------- | ------- |
| CPU       | 4 vCPU  |
| RAM       | 16 GB   |
| Disk      | \~30 GB |

> Hypervisors that pre-allocate the full sparse image size require significantly more datastore space. Confirm the allocation behaviour of your target hypervisor before import.

### External Dependencies

The VM requires two external services to be reachable before it can start. These are **not** bundled in the image.

| Service                        | Requirement                                                                                           |
| ------------------------------ | ----------------------------------------------------------------------------------------------------- |
| **Database (PostgreSQL)**      | A database instance accessible from the VM. The VM will not start if unreachable.                     |
| **S3-compatible object store** | A bucket accessible from the VM (AWS S3, MinIO, or equivalent). The VM will not start if unreachable. |

Ensure outbound connectivity from the VM to both services before deployment.

### Supported Hypervisors

| Format                              | Target         |
| ----------------------------------- | -------------- |
| `popsink-deploy-vm-<version>.qcow2` | KVM/QEMU       |
| `popsink-deploy-vm-<version>.ova`   | VMware vSphere |

***

## Getting the VM Image

Popsink provides pre-built images for each release. Contact your Popsink representative to receive a download link and its accompanying `.sha256` checksum file.

Once you have both files, verify the image integrity before importing:

```bash theme={null}
sha256sum -c popsink-deploy-vm-<version>.<format>.sha256
```

Expected output: `popsink-deploy-vm-<version>.<format>: OK`

***

## Network Configuration

### Inbound

| Port | Protocol | Purpose                               |
| ---- | -------- | ------------------------------------- |
| 80   | TCP      | Popsink data-plane — all HTTP traffic |

Open port 80 from your internal network to the VM. TLS termination and HTTPS are your responsibility — the VM does not perform TLS.

### Outbound

| Destination                                             | Port                        | Required | Effect if unreachable                  |
| ------------------------------------------------------- | --------------------------- | -------- | -------------------------------------- |
| Your database instance (PostgreSQL)                     | Your DB port (default 5432) | **Hard** | VM fails to start                      |
| Your S3-compatible object store                         | TCP 443 (HTTPS)             | **Hard** | VM fails to start                      |
| Popsink control-plane (`control-plane-api.popsink.com`) | TCP 443 (HTTPS)             | Soft     | Pipelines run normally; UI logins fail |

Ensure the hard requirements are in place before booting the VM. The soft dependency (the Popsink control-plane) only affects UI logins — pipelines are unaffected.

### Outbound Proxy

If your network enforces an outbound proxy, set `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` in the configuration file (see [Configuration](#configuration)).

***

## Deployment

### KVM/QEMU

1. Build the `seed.iso` configuration image first — see [Cloud-Init](#cloud-init) below for step-by-step instructions.

2. Import the image and attach the `seed.iso` as a virtual CD-ROM in a single command. The example below uses `virt-install`; `virsh` and `virt-manager` are equally valid:

   ```bash theme={null}
   virt-install \
     --name popsink \
     --memory 16384 \
     --vcpus 4 \
     --disk path=popsink-deploy-vm-<version>.qcow2,format=qcow2 \
     --disk path=seed.iso,device=cdrom,readonly=yes \
     --import \
     --os-variant generic \
     --noautoconsole
   ```

3. Boot the VM. Cloud-init reads the ISO on first boot and configures the data-plane automatically.

### VMware vSphere

1. Build the `seed.iso` configuration image first — see [Cloud-Init](#cloud-init) below for step-by-step instructions.

2. In the vSphere UI, right-click the target datacenter or cluster → **Deploy OVF Template**.

3. Follow the wizard:
   * At the **Select networks** step, map the VM's network adapter to a port group that satisfies the inbound and outbound requirements listed above.
   * Confirm the hardware summary: 4 vCPU, 16 GB RAM, 30 GiB disk.
   * At the **Select storage** step, choose thin provisioning unless thick is required by your policy.

4. After the deploy task completes, **do not boot the VM yet**. Upload `seed.iso` to your vSphere datastore:
   * In the vSphere Client, navigate to the target datastore → **Files** tab → **Upload**.
   * Upload the `seed.iso` file you generated.

5. Attach `seed.iso` as a CD-ROM drive on the VM:
   * Right-click the VM → **Edit Settings**.
   * Click **Add Other Device** → **CD/DVD Drive**.
   * Select **Datastore ISO File** and browse to the uploaded `seed.iso`.
   * Ensure **Connect at Power On** is checked, then click **OK**.

6. Boot the VM. Cloud-init reads the ISO on first boot and configures the data-plane automatically.

***

## Configuration

The VM reads its configuration from `/etc/popsink/config.env` at first boot, then runs `popsink-configure.service` which applies the settings to the data-plane.

### Cloud-Init

Cloud-init is the standard Linux mechanism for configuring a VM on first boot. On both KVM/QEMU and VMware, configuration is delivered via a small ISO image — the `seed.iso` — which you attach as a virtual CD-ROM drive. The VM reads it automatically at boot; no manual console access is needed.

The `seed.iso` contains two plain-text files:

* **`meta-data`** — minimal instance identity (hostname and ID).
* **`user-data`** — the actual configuration in `#cloud-config` format.

#### Step 1 — Create a working directory

```bash theme={null}
mkdir popsink-cloudinit
cd popsink-cloudinit
```

#### Step 2 — Create `meta-data`

```
instance-id: popsink-vm
local-hostname: popsink-vm
```

Save this as a file named exactly `meta-data` (no extension).

#### Step 3 — Create `user-data`

Save the following as `user-data`, filling in all `REPLACE_WITH_*` placeholders with your values:

```yaml theme={null}
#cloud-config
write_files:
  - path: /etc/popsink/config.env
    owner: root:root
    permissions: '0600'
    content: |
      DB_BACKEND=postgres
      DB_HOST=db.example.com
      DB_PORT=5432
      DB_DATABASE=popsink
      DB_USER=popsink
      DB_PASSWORD=REPLACE_WITH_SECRET

      KORA_DB_BACKEND=postgres
      KORA_DB_HOST=db.example.com
      KORA_DB_PORT=5432
      KORA_DB_DATABASE=kora
      KORA_DB_USER=kora
      KORA_DB_PASSWORD=REPLACE_WITH_SECRET

      S3_BUCKET=s3://my-bucket/kafka/
      S3_REGION=eu-west-1
      S3_ACCESS_KEY_ID=REPLACE_WITH_KEY_ID
      S3_SECRET_ACCESS_KEY=REPLACE_WITH_SECRET

      DEPLOYMENT_ID=REPLACE_WITH_UUID
      DEPLOYMENT_JWT_TOKEN=REPLACE_WITH_SECRET
      INGRESS_URL=http://vm.example.com

      ADMIN_PANEL_USERNAME=admin
      ADMIN_PANEL_PASSWORD=REPLACE_WITH_SECRET

      JWT_SECRET=REPLACE_WITH_SECRET
      # Must be a 32-byte secret key encoded in URL-safe base64 format (a Fernet
      # key). Generate one with: openssl rand -base64 32 | tr '+/' '-_'
      CONNECTOR_CONFIG_ENCRYPTION_KEY=REPLACE_WITH_SECRET
```

See [Parameter Reference](#parameter-reference) for the description and constraints of each value.

> **Security:** This file contains secrets. Do not commit it to version control with real credentials. Keep it in a secure location and delete it once the VM is configured.

> **SSH access (optional):** No SSH keys are pre-provisioned in the image. To enable SSH, add an `ssh_authorized_keys` stanza to your `user-data` alongside the `write_files` block:
>
> ```yaml theme={null}
> ssh_authorized_keys:
>   - ssh-rsa AAAA...your-public-key...
> ```
>
> Without this, the VM is only accessible via the hypervisor console.

#### Static IP (optional)

By default the VM uses DHCP. If your environment requires a fixed IP address, create a third file named `network-config` in the same directory as `meta-data` and `user-data`:

```yaml theme={null}
version: 2
ethernets:
  eth0:
    addresses:
      - 192.168.1.50/24
    gateway4: 192.168.1.1
    nameservers:
      addresses: [192.168.1.1]
```

Replace `eth0` with the actual interface name for your VM (visible in the hypervisor console at first boot), and adjust the IP addresses for your network. If you create this file, include it when generating the ISO in Step 4 below.

#### Step 4 — Generate `seed.iso`

Choose the tool available on your machine. All three produce an equivalent result.

**Option A — `cloud-localds`** (Linux, simplest)

`cloud-localds` is purpose-built for this task and requires no flags:

```bash theme={null}
# Install on Ubuntu/Debian
sudo apt-get install cloud-image-utils

# Without static IP
cloud-localds seed.iso user-data meta-data

# With static IP (if you created a network-config file)
cloud-localds seed.iso user-data meta-data --network-config network-config
```

**Option B — `genisoimage` or `mkisofs`** (Linux)

`genisoimage` and `mkisofs` are interchangeable — use whichever is installed:

```bash theme={null}
# Install on Ubuntu/Debian
sudo apt-get install genisoimage

# Without static IP
genisoimage -output seed.iso -volid cidata -joliet -rock user-data meta-data

# With static IP (if you created a network-config file)
genisoimage -output seed.iso -volid cidata -joliet -rock user-data meta-data network-config
```

```bash theme={null}
# Install on RHEL/Fedora
sudo dnf install genisoimage

# Without static IP
mkisofs -output seed.iso -volid cidata -joliet -rock user-data meta-data

# With static IP
mkisofs -output seed.iso -volid cidata -joliet -rock user-data meta-data network-config
```

> The `-volid cidata` flag is mandatory. Cloud-init identifies the NoCloud datasource by looking for a volume labelled `cidata` — any other label will be ignored.

**Option C — `hdiutil`** (macOS)

`hdiutil` is built into macOS. It takes a directory rather than individual files, so run it from the parent directory. If you created a `network-config` file, simply place it in the directory alongside the other files — no additional flags are needed:

```bash theme={null}
hdiutil makehybrid -o seed.iso -hfs -joliet -iso -default-volume-name cidata popsink-cloudinit/
```

#### Step 5 — Attach as CD-ROM and boot

* **KVM/QEMU:** pass `seed.iso` as a virtual CD-ROM in the `virt-install` command (see the [KVM/QEMU deployment section](#kvmqemu) above).
* **VMware vSphere:** upload `seed.iso` to your datastore and attach it as a CD/DVD drive before booting (see the [VMware vSphere deployment section](#vmware-vsphere) above).

Cloud-init reads the ISO exactly once on first boot. The ISO is no longer needed afterwards and can be detached and deleted.

### Manual Configuration

Use this method only if your hypervisor cannot attach a CD-ROM drive, or if the VM has already booted without one. Deliver the configuration directly at the hypervisor console:

1. Create the configuration file:

   ```bash theme={null}
   sudo install -m 0600 -o root -g root /dev/null /etc/popsink/config.env
   sudo tee /etc/popsink/config.env <<'EOF'
   DB_BACKEND=postgres
   DB_HOST=db.example.com
   # ... (fill in all required parameters — see table below)
   EOF
   ```

2. Start the configure service:

   ```bash theme={null}
   sudo systemctl start popsink-configure.service
   ```

3. Verify success:

   ```bash theme={null}
   sudo journalctl -u popsink-configure.service
   ```

   A successful run ends with:

   ```
   Popsink cloud-init configuration applied successfully. Data-plane is running.
   ```

### Re-applying Configuration

After a successful first run, the configure service creates a marker file `/etc/popsink/.configured` and will not re-run on subsequent boots. If you need to update the configuration — for example, to correct a credential or change the database host — update the file and clear the marker:

1. Edit `/etc/popsink/config.env` with the new values.

2. Remove the marker and restart the service:

   ```bash theme={null}
   sudo rm /etc/popsink/.configured
   sudo systemctl restart popsink-configure.service
   ```

3. Verify success:

   ```bash theme={null}
   sudo journalctl -u popsink-configure.service
   ```

### Parameter Reference

#### Data-plane database (required)

| Parameter     | Required | Default | Description                                             |
| ------------- | -------- | ------- | ------------------------------------------------------- |
| `DB_BACKEND`  | Yes      | —       | Database backend: `postgres`. No default — must be set. |
| `DB_HOST`     | Yes      | —       | Database host (FQDN or IP)                              |
| `DB_PORT`     | No       | `5432`  | Database port                                           |
| `DB_DATABASE` | Yes      | —       | Database name.                                          |
| `DB_USER`     | Yes      | —       | Database username                                       |
| `DB_PASSWORD` | Yes      | —       | Database password                                       |

#### Kora Schema Registry Database (required)

Kora is Popsink's internal schema registry — it tracks the structure of the data flowing through your pipelines. It requires its own database (PostgreSQL), configured independently from the data-plane via `KORA_DB_BACKEND` — the two backends do not need to match. It can share the same database instance as the data-plane, using a separate database name (e.g. `kora`).

| Parameter          | Required | Default | Description                                                |
| ------------------ | -------- | ------- | ---------------------------------------------------------- |
| `KORA_DB_BACKEND`  | Yes      | —       | Database backend: `postgres` or. No default — must be set. |
| `KORA_DB_HOST`     | Yes      | —       | Kora database host (FQDN or IP)                            |
| `KORA_DB_PORT`     | No       | `5432`  | Port                                                       |
| `KORA_DB_DATABASE` | Yes      | —       | Database name.                                             |
| `KORA_DB_USER`     | Yes      | —       | Username                                                   |
| `KORA_DB_PASSWORD` | Yes      | —       | Password                                                   |

#### S3-Compatible Storage (required)

| Parameter              | Required | Default | Description                                                                                                         |
| ---------------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------------- |
| `S3_BUCKET`            | Yes      | —       | Full S3 URL including prefix: `s3://bucket/prefix/` — **not** just the bucket name                                  |
| `S3_REGION`            | Yes      | —       | AWS region (e.g. `eu-west-1`). Use any non-empty value for S3-compatible services that don't require a real region. |
| `S3_ACCESS_KEY_ID`     | Yes      | —       | Access key ID                                                                                                       |
| `S3_SECRET_ACCESS_KEY` | Yes      | —       | Secret access key                                                                                                   |
| `S3_ENDPOINT`          | No       | —       | Custom endpoint URL (e.g. MinIO). Omit for AWS S3.                                                                  |

#### Popsink Platform (required)

| Parameter              | Required | Description                                                                                   |
| ---------------------- | -------- | --------------------------------------------------------------------------------------------- |
| `DEPLOYMENT_ID`        | Yes      | UUID for this deployment — provided by Popsink                                                |
| `DEPLOYMENT_JWT_TOKEN` | Yes      | Deployment JWT token — provided by Popsink                                                    |
| `INGRESS_URL`          | Yes      | Display-only URL of this VM (e.g. `http://10.0.1.50`). Does **not** configure routing or TLS. |

#### Admin Credentials (required)

| Parameter              | Required | Description             |
| ---------------------- | -------- | ----------------------- |
| `ADMIN_PANEL_USERNAME` | Yes      | Admin UI login username |
| `ADMIN_PANEL_PASSWORD` | Yes      | Admin UI login password |

#### Security Keys (required)

| Parameter                         | Required | Description                                                                                                                                                                                                                                                                                                                         |
| --------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `JWT_SECRET`                      | Yes      | JWT signing secret. Changing this invalidates all active sessions.                                                                                                                                                                                                                                                                  |
| `CONNECTOR_CONFIG_ENCRYPTION_KEY` | Yes      | Encryption key for connector configurations. Must be a 32-byte secret key encoded in URL-safe base64 format (a Fernet key). Generate one with `openssl rand -base64 32 \| tr '+/' '-_'`. **Back this up before going live.** If lost, all connector configurations become permanently unreadable — Popsink cannot recover this key. |

#### Outbound Proxy (optional)

| Parameter     | Description                                           |
| ------------- | ----------------------------------------------------- |
| `HTTP_PROXY`  | HTTP proxy URL (e.g. `http://proxy.example.com:3128`) |
| `HTTPS_PROXY` | HTTPS proxy URL                                       |
| `NO_PROXY`    | Comma-separated list of hosts to bypass the proxy     |

***

## Health Verification

After the VM boots, verify each layer in order. Fix a failing layer before investigating the ones above it.

### Layer 1 — Data-Plane (network reachability)

From any host with network access to the VM:

```bash theme={null}
curl http://<vm-ip>/api/readyz
```

Expected: HTTP 200.

### Layer 2 — k3s Runtime

At the VM console or over SSH:

```bash theme={null}
systemctl status k3s
```

Expected: `Active: active (running)`. k3s may take a minute to reach this state on first boot.

### Layer 3 — Configuration Service

```bash theme={null}
journalctl -u popsink-configure.service
```

Look for the success message at the end of the output. If the service failed, the log identifies the cause (database unreachable, invalid S3 credentials, etc.).

### Layer 4 — Boot Check

```bash theme={null}
journalctl -u popsink-boot-check.service
```

* **No output:** configuration was present at boot — this layer is healthy.
* **Output present:** no configuration was found when the VM booted. Deliver the configuration file and re-run the configure service:
  ```bash theme={null}
  sudo rm -f /etc/popsink/.configured
  sudo systemctl start popsink-configure.service
  ```

***

## Troubleshooting

| Symptom                                                      | Likely cause                                          | Fix                                                                                                                                                                                      |
| ------------------------------------------------------------ | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `curl /api/readyz` — no response or connection refused       | Configuration service failed, or k3s still starting   | Check Layer 3 then Layer 2 in Health Verification above                                                                                                                                  |
| `popsink-configure.service` shows a database error           | `DB_HOST` unreachable, wrong port, or bad credentials | Test connectivity from the VM console: `nc -zv <DB_HOST> <DB_PORT>`; for PostgreSQL verify `pg_hba.conf` allows the VM's IP;                                                             |
| `popsink-configure.service` shows an S3 error                | Wrong `S3_BUCKET` format or unreachable endpoint      | `S3_BUCKET` must be a full URL — `s3://bucket/prefix/` not just `bucket`                                                                                                                 |
| VM booted but appears unconfigured                           | `seed.iso` not attached, or wrong volume label        | Check `journalctl -u popsink-boot-check.service`; verify the ISO was generated with `-volid cidata`; re-deliver and re-run (see [Re-applying Configuration](#re-applying-configuration)) |
| UI loads but all logins fail                                 | Popsink control-plane unreachable                     | Verify outbound HTTPS (tcp/443) from the VM to `control-plane-api.popsink.com`; check proxy settings if applicable                                                                       |
| Port 80 unreachable from another host, but the VM is healthy | Network firewall blocking inbound port 80             | Open TCP port 80 from your internal network to the VM's IP                                                                                                                               |

***

## Accessing Popsink

Once all health checks pass, open a browser and navigate to:

```
http://<vm-ip>/
```

Log in with the `ADMIN_PANEL_USERNAME` and `ADMIN_PANEL_PASSWORD` you set in the configuration.

***

## Updates

The VM is stateless — all data lives in your external database instance and S3 bucket. Updating Popsink means replacing the VM with a newer image; no in-place upgrades are required.

**Replacement procedure:**

1. Boot the new VM with the same configuration.
2. Confirm `curl http://<new-vm-ip>/api/readyz` returns HTTP 200.
3. Run a test pipeline end-to-end on the new VM.
4. Pause any active pipelines on the old VM, then switch network routing to the new VM.
5. Resume pipelines and decommission the old VM.

> The old and new VM can run against the same database and S3 bucket simultaneously — state is in the external systems, and concurrent access is safe. Do **not** serve production traffic from both VMs at the same time, as pipelines may execute in duplicate.

If the new VM fails verification, restore routing to the old VM. Its state is unaffected.

***

## Limitations

| Limitation              | Details                                             |
| ----------------------- | --------------------------------------------------- |
| No horizontal scaling   | Performance is bounded by a single machine.         |
| Single point of failure | If the VM goes down, the data-plane is unavailable. |
| No built-in TLS         | HTTPS termination is your responsibility.           |

For production-grade scalability and high availability, consider the [Kubernetes deployment](./selfhosted).


# On prem
Source: https://docs.popsink.com/on-prem/on-prem



## 📋 Table of Contents

1. [Quick Start](#quick-start)
2. [Prerequisites](#prerequisites)
3. [API Overview](#api-overview)
4. [Authentication Flow](#authentication-flow)
5. [Step-by-Step Tutorial](#step-by-step-tutorial)
   * [1. User Registration](#step-1-user-registration)
   * [2. User Login](#step-2-user-login)
   * [3. Environment Setup](#step-3-environment-setup)
   * [4. Team Creation](#step-4-team-creation)
   * [5. Team Member Management](#step-5-team-member-management)
   * [6. Pipeline Creation](#step-6-pipeline-creation)
   * [7. Pipeline Updates](#step-7-pipeline-updates)
   * [8. Pipeline Control](#step-8-pipeline-control)
6. [Testing Tools](#testing-tools)
7. [API Reference](#api-reference)
8. [Connector Configurations](#connector-configurations)
9. [Common Patterns](#common-patterns)
10. [Troubleshooting](#troubleshooting)

***

## 🚀 Quick Start

**For the impatient:** Here's the minimum you need to know:

```bash theme={null}
# 1. Create a user
curl -X POST "https://your-server/api/auth/register" \
  -H "Content-Type: application/json" \
  -d '{"email":"user@example.com","password":"secure123","is_active":true}' -k

# 2. Login to get your token
curl -X POST "https://your-server/api/auth/jwt/login" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d 'username=user@example.com&password=secure123' -k

# 3. Use the token in all subsequent requests
curl -X GET "https://your-server/api/pipelines/" \
  -H "Authorization: Bearer YOUR_TOKEN_HERE" -k
```

***

## ✅ Prerequisites

### What You Need

* **Access to a Popsink server** (e.g., `https://popsink.your-company.com`)
* **A terminal or API client** (curl, Postman, HTTPie, etc.)
* **5-15 minutes** to complete this guide

### Basic Concepts

| Concept          | Description                                         |
| ---------------- | --------------------------------------------------- |
| **Environment**  | A workspace that contains teams and their resources |
| **Team**         | A group of users working together on pipelines      |
| **Pipeline**     | A data processing flow from source to target        |
| **Connector**    | A connection to a data source or destination        |
| **Subscription** | Configuration for how data flows through a pipeline |
| **DataModel**    | Schema and configuration for data transformation    |

***

## 🔍 API Overview

### Base URL Structure

```
https://your-server/api/
```

### Authentication

All API endpoints (except registration and login) require a **Bearer token** in the `Authorization` header:

```
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
```

### Response Format

All responses are in **JSON** format:

```json theme={null}
{
  "id": "uuid-here",
  "name": "resource-name",
  "created_at": "2025-11-19T10:00:00Z"
}
```

### HTTP Status Codes

| Code  | Meaning          | When You'll See It                      |
| ----- | ---------------- | --------------------------------------- |
| `200` | Success          | GET, PATCH operations                   |
| `201` | Created          | POST operations that create resources   |
| `202` | Accepted         | Async operations (start/pause pipeline) |
| `204` | No Content       | DELETE operations                       |
| `400` | Bad Request      | Invalid input data                      |
| `401` | Unauthorized     | Missing or invalid token                |
| `403` | Forbidden        | Insufficient permissions                |
| `404` | Not Found        | Resource doesn't exist                  |
| `422` | Validation Error | Data doesn't meet requirements          |

***

## 🔐 Authentication Flow

```
┌─────────────┐
│   Register  │  POST /auth/register
└──────┬──────┘
       │
       ▼
┌─────────────┐
│    Login    │  POST /auth/jwt/login
└──────┬──────┘
       │
       ▼
┌─────────────┐
│  Get Token  │  Save this token!
└──────┬──────┘
       │
       ▼
┌─────────────┐
│  Use Token  │  All API requests
└─────────────┘
```

***

## 📖 Step-by-Step Tutorial

### Step 1: User Registration

**What:** Create your user account
**When:** First time using the API
**Required:** Email and password

#### Request

```bash theme={null}
curl -X POST "https://your-server/api/auth/register" \
  -H "Content-Type: application/json" \
  -d '{
    "email": "john.doe@company.com",
    "password": "MySecurePassword123!",
    "is_active": true,
    "is_superuser": false,
    "is_verified": false
  }' -k
```

#### Field Descriptions

| Field          | Type    | Required | Description                                       |
| -------------- | ------- | -------- | ------------------------------------------------- |
| `email`        | string  | ✅        | Your email address (must be unique)               |
| `password`     | string  | ✅        | Your password (min 8 characters recommended)      |
| `is_active`    | boolean | ✅        | Set to `true` to activate the account immediately |
| `is_superuser` | boolean | ⬜        | Admin privileges (default: `false`)               |
| `is_verified`  | boolean | ⬜        | Email verification status (default: `false`)      |

#### Response (201 Created)

```json theme={null}
{
  "id": "3b0c298a-062e-42d1-a01b-277fa66908da",
  "email": "john.doe@company.com",
  "is_active": true,
  "is_superuser": false,
  "is_verified": false,
  "active_env_id": null
}
```

#### 📝 Save This

```
USER_ID=3b0c298a-062e-42d1-a01b-277fa66908da
EMAIL=john.doe@company.com
PASSWORD=MySecurePassword123!
```

***

### Step 2: User Login

**What:** Authenticate and get your access token
**When:** Before making any authenticated API calls
**Token Lifetime:** Configurable (typically 24 hours)

#### Request

```bash theme={null}
curl -X POST "https://your-server/api/auth/jwt/login" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d 'username=john.doe@company.com&password=MySecurePassword123!' -k
```

#### Response (200 OK)

```json theme={null}
{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIzYjBjMjk4YS0wNjJlLTQyZDEtYTAxYi0yNzdmYTY2OTA4ZGEiLCJhdWQiOlsiZmFzdGFwaS11c2VyczphdXRoIl0sImV4cCI6MTcwMDAwMDAwMH0.signature",
  "token_type": "bearer"
}
```

#### 📝 Save This

```
TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
```

#### Using Your Token

From now on, include this header in **every request**:

```bash theme={null}
-H "Authorization: Bearer $TOKEN"
```

***

### Step 3: Environment Setup

**What:** Create a workspace for your teams and pipelines
**When:** After logging in, before creating teams
**Permissions:** Any authenticated user can create an environment

#### Request

```bash theme={null}
curl -X POST "https://your-server/api/envs/" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "production-environment",
    "description": "Production data processing environment",
    "use_retention": true
  }' -k
```

#### Field Descriptions

| Field           | Type    | Required | Description                                       |
| --------------- | ------- | -------- | ------------------------------------------------- |
| `name`          | string  | ✅        | Environment name (must be unique)                 |
| `description`   | string  | ⬜        | Description of the environment's purpose          |
| `use_retention` | boolean | ⬜        | Enable data retention policies (default: `false`) |

#### Response (201 Created)

```json theme={null}
{
  "id": "d455a981-be17-455e-a03c-4d9c3224db72",
  "name": "production-environment",
  "description": "Production data processing environment",
  "use_retention": true,
  "retention_configuration": null
}
```

#### 📝 Save This

```
ENV_ID=d455a981-be17-455e-a03c-4d9c3224db72
```

***

### Step 4: Team Creation

**What:** Create a team within your environment
**When:** After creating an environment
**Ownership:** The creator automatically becomes a team owner

#### Request

```bash theme={null}
curl -X POST "https://your-server/api/teams/" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Data Engineering Team",
    "description": "Team responsible for ETL pipelines"
  }' -k
```

#### Field Descriptions

| Field         | Type   | Required | Description                              |
| ------------- | ------ | -------- | ---------------------------------------- |
| `name`        | string | ✅        | Team name (unique within environment)    |
| `description` | string | ⬜        | Purpose and responsibilities of the team |

#### Response (201 Created)

```json theme={null}
{
  "id": "881e0fce-5caa-4f5c-82c6-5eb883c47100",
  "name": "Data Engineering Team",
  "description": "Team responsible for ETL pipelines",
  "env_id": "d455a981-be17-455e-a03c-4d9c3224db72"
}
```

#### 📝 Save This

```
TEAM_ID=881e0fce-5caa-4f5c-82c6-5eb883c47100
```

***

### Step 5: Team Member Management

**What:** Add users to your team
**When:** After creating a team
**Permissions:** Only team owners can add members

#### 5.1: Create Additional Users (Optional)

If you need to invite teammates, first create their accounts:

```bash theme={null}
curl -X POST "https://your-server/api/auth/register" \
  -H "Content-Type: application/json" \
  -d '{
    "email": "jane.smith@company.com",
    "password": "SecurePass456!",
    "is_active": true
  }' -k
```

**Response:**

```json theme={null}
{
  "id": "32d4b09d-4527-40d5-923b-e166d85cb459",
  "email": "jane.smith@company.com",
  "is_active": true
}
```

#### 5.2: Add Members to Team

```bash theme={null}
curl -X POST "https://your-server/api/teams/$TEAM_ID/members/bulk" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "members": [
      "32d4b09d-4527-40d5-923b-e166d85cb459"
    ]
  }' -k
```

#### Response (204 No Content)

Success! The members have been added to your team.

#### 5.3: List Team Members

```bash theme={null}
curl -X GET "https://your-server/api/teams/$TEAM_ID/members" \
  -H "Authorization: Bearer $TOKEN" -k
```

**Response:**

```json theme={null}
[
  {
    "id": "3b0c298a-062e-42d1-a01b-277fa66908da",
    "email": "john.doe@company.com",
    "role": "owner"
  },
  {
    "id": "32d4b09d-4527-40d5-923b-e166d85cb459",
    "email": "jane.smith@company.com",
    "role": "member"
  }
]
```

***

### Step 6: Pipeline Creation

**What:** Create a data pipeline with source and target connectors
**When:** After team setup
**Permissions:** Team members with write access

#### 🎯 Understanding Pipeline Structure (V2 - Flattened)

The new API uses a **flattened structure** instead of nested `json_configuration`. You can:

1. **Use existing connectors** by specifying their IDs
2. **Create new connectors** by providing name, type, and config
3. **Mix both** (e.g., existing source + new target)

#### Option A: Create Pipeline with New Connectors

```bash theme={null}
curl -X POST "https://your-server/api/pipelines/" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "kafka-to-oracle-pipeline",
    "team_id": "'$TEAM_ID'",

    "source_name": "kafka-events-source",
    "source_type": "KAFKA_SOURCE",
    "source_config": {
      "topic": "user-events",
      "bootstrap_servers": "kafka.company.com:9092",
      "consumer_group_id": "pipeline-consumer",
      "sasl_username": "kafka_user",
      "sasl_password": "kafka_pass",
      "sasl_mechanism": "PLAIN",
      "security_protocol": "SASL_SSL"
    },

    "target_name": "oracle-warehouse",
    "target_type": "ORACLE_TARGET",
    "target_config": {
      "host": "oracle.company.com",
      "port": "1521",
      "database": "ORCL",
      "server_name": "XE",
      "server_id": "oraclesrv01",
      "user": "etl_user",
      "password": "etl_pass"
    },

    "subscription_target_table_name": "events",
    "subscription_backfill": false,
    "subscription_mapper_config": [
      {
        "key": "user_id",
        "path": "$.user_id",
        "cast": "string",
        "primary_key": true,
        "nullable": false
      },
      {
        "key": "event_type",
        "path": "$.event_type",
        "cast": "string",
        "primary_key": false,
        "nullable": false
      }
    ]
  }' -k
```

#### Option B: Create Pipeline with Existing Connectors

```bash theme={null}
curl -X POST "https://your-server/api/pipelines/" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "reuse-connectors-pipeline",
    "team_id": "'$TEAM_ID'",

    "existing_source_id": "a1b2c3d4-e5f6-4a5b-8c9d-0e1f2a3b4c5d",
    "existing_target_id": "f6e5d4c3-b2a1-4c5d-9e8f-5d4c3b2a1f0e",

    "subscription_target_table_name": "events_copy",
    "subscription_backfill": true
  }' -k
```

#### Pipeline Configuration Fields

##### Core Fields

| Field     | Type   | Required | Description                                                                           |
| --------- | ------ | -------- | ------------------------------------------------------------------------------------- |
| `name`    | string | ✅        | Pipeline name (unique within team, alphanumeric with `-` and `_` only, max 255 chars) |
| `team_id` | UUID   | ✅        | Team that owns this pipeline                                                          |

##### Source Connector (Choose ONE approach)

**Approach 1: Use Existing Connector**

| Field                | Type | Description                        |
| -------------------- | ---- | ---------------------------------- |
| `existing_source_id` | UUID | ID of an existing source connector |

**Approach 2: Create New Connector**

| Field           | Type   | Description                                  |
| --------------- | ------ | -------------------------------------------- |
| `source_name`   | string | Name for the new source connector            |
| `source_type`   | string | Connector type: `KAFKA_SOURCE`               |
| `source_config` | object | Configuration specific to the connector type |

##### Target Connector (Choose ONE approach)

**Approach 1: Use Existing Connector**

| Field                | Type | Description                        |
| -------------------- | ---- | ---------------------------------- |
| `existing_target_id` | UUID | ID of an existing target connector |

**Approach 2: Create New Connector**

| Field           | Type   | Description                                     |
| --------------- | ------ | ----------------------------------------------- |
| `target_name`   | string | Name for the new target connector               |
| `target_type`   | string | Connector type: `KAFKA_TARGET`, `ORACLE_TARGET` |
| `target_config` | object | Configuration specific to the connector type    |

##### Data Model Configuration (Optional)

| Field                             | Type    | Description                       |
| --------------------------------- | ------- | --------------------------------- |
| `datamodel_source_topic`          | string  | Source topic for the data model   |
| `datamodel_target_topic`          | string  | Target topic for the data model   |
| `datamodel_error_table_enabled`   | boolean | Enable error table for data model |
| `datamodel_error_table_name`      | string  | Name of the error table           |
| `datamodel_error_table_target_id` | UUID    | Target connector for error table  |

##### Subscription Configuration (Optional)

| Field                                | Type    | Description                                            |
| ------------------------------------ | ------- | ------------------------------------------------------ |
| `subscription_target_table_name`     | string  | Target table name in the destination                   |
| `subscription_backfill`              | boolean | Whether to backfill historical data                    |
| `subscription_mapper_config`         | array   | Column mapping configuration (see Mapper Config below) |
| `subscription_consumer_id`           | string  | Consumer ID (auto-generated if not provided)           |
| `subscription_error_table_enabled`   | boolean | Enable error table for subscription                    |
| `subscription_error_table_name`      | string  | Name of the subscription error table                   |
| `subscription_error_table_target_id` | UUID    | Target connector for subscription errors               |

##### Mapper Configuration

Each mapper config entry:

| Field         | Type    | Description                                                             |
| ------------- | ------- | ----------------------------------------------------------------------- |
| `key`         | string  | Column name                                                             |
| `path`        | string  | JSON path to extract value (e.g., `$.field.nested`)                     |
| `static`      | string  | Static value (alternative to path)                                      |
| `cast`        | string  | Data type: `string`, `int`, `float`, `bool`, `date`, `datetime`, `time` |
| `cast_format` | string  | Format for casting (e.g., date format)                                  |
| `primary_key` | boolean | Is this a primary key column                                            |
| `nullable`    | boolean | Can this column be null                                                 |

#### Response (201 Created)

```json theme={null}
{
  "id": "5efd3d63-a6eb-4b0c-afe9-6ee1d8f5f820",
  "name": "kafka-to-oracle-pipeline",
  "state": "draft",
  "version": 1,
  "team_id": "881e0fce-5caa-4f5c-82c6-5eb883c47100",
  "team_name": "Data Engineering Team",

  "existing_source_id": null,
  "source_name": "kafka-events-source",
  "source_type": "KAFKA_SOURCE",
  "source_config": { "topic": "user-events", ... },

  "existing_target_id": null,
  "target_name": "oracle-warehouse",
  "target_type": "ORACLE_TARGET",
  "target_config": { "host": "oracle.company.com", ... },

  "subscription_target_table_name": "events",
  "subscription_backfill": false,
  "subscription_mapper_config": [...],

  "created_at": "2025-11-19T10:30:00Z",
  "updated_at": "2025-11-19T10:30:00Z"
}
```

#### 📝 Save This

```
PIPELINE_ID=5efd3d63-a6eb-4b0c-afe9-6ee1d8f5f820
```

#### 🔍 Available Connector Types

**Sources:**

* `KAFKA_SOURCE` - Apache Kafka

**Targets:**

* `KAFKA_TARGET` - Apache Kafka
* `ORACLE_TARGET` - Oracle database

**Jobs:**

* `JOB_SMT` - Single Message Transform (for data transformations)

***

### Step 7: Pipeline Updates

**What:** Modify an existing pipeline
**When:** Need to change configuration or settings
**Permissions:** Team members with write access

#### Update Pipeline Configuration

```bash theme={null}
curl -X PATCH "https://your-server/api/pipelines/$PIPELINE_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "updated-pipeline-name",
    "source_config": {
      "topic": "new-topic",
      "bootstrap_servers": "kafka.company.com:9092",
      "consumer_group_id": "updated-consumer"
    },
    "subscription_backfill": true
  }' -k
```

#### 💡 Update Patterns

**Pattern 1: Update Only Name**

```json theme={null}
{
  "name": "new-pipeline-name"
}
```

**Pattern 2: Switch to Existing Connector**

```json theme={null}
{
  "existing_source_id": "new-connector-uuid",
  "source_name": null,
  "source_type": null,
  "source_config": null
}
```

**Pattern 3: Update Subscription Config**

```json theme={null}
{
  "subscription_target_table_name": "new_table",
  "subscription_backfill": true,
  "subscription_mapper_config": [...]
}
```

#### Response (200 OK)

Returns the updated pipeline with all fields.

***

### Step 8: Pipeline Control

**What:** Start, pause, or check pipeline status
**When:** After pipeline creation and configuration
**States:** `draft` → `building` → `live` → `paused` / `error`

#### 8.1: Start Pipeline

```bash theme={null}
curl -X POST "https://your-server/api/pipelines/$PIPELINE_ID/start" \
  -H "Authorization: Bearer $TOKEN" -k
```

**Response (202 Accepted):**

```json theme={null}
"BUILDING"
```

The pipeline transitions through these states:

1. `draft` - Initial state, configuration in progress
2. `building` - Pipeline is being deployed
3. `live` - Pipeline is running and processing data

#### 8.2: Pause Pipeline

```bash theme={null}
curl -X POST "https://your-server/api/pipelines/$PIPELINE_ID/pause" \
  -H "Authorization: Bearer $TOKEN" -k
```

**Response (202 Accepted):**

```json theme={null}
"PAUSED"
```

#### 8.3: Check Pipeline Status

```bash theme={null}
curl -X GET "https://your-server/api/pipelines/$PIPELINE_ID" \
  -H "Authorization: Bearer $TOKEN" -k
```

**Response (200 OK):**

```json theme={null}
{
  "id": "5efd3d63-a6eb-4b0c-afe9-6ee1d8f5f820",
  "name": "kafka-to-oracle-pipeline",
  "state": "live",
  "version": 1,
  "created_at": "2025-11-19T10:30:00Z",
  "updated_at": "2025-11-19T10:45:00Z"
}
```

#### 8.4: Get Pipeline Logs (WebSocket)

```javascript theme={null}
// WebSocket connection example
const ws = new WebSocket(
  `wss://your-server/api/pipelines/${PIPELINE_ID}/logs`,
  ['Authorization', `Bearer ${TOKEN}`]
);

ws.onmessage = (event) => {
  console.log('Log:', event.data);
};
```

#### Pipeline State Diagram

```
     ┌──────┐
     │DRAFT │ (initial state)
     └───┬──┘
         │ start
         ▼
   ┌─────────┐
   │BUILDING │ (deployment in progress)
   └────┬────┘
        │ (automatic)
        ▼
     ┌──────┐
     │ LIVE │ (processing data)
     └───┬──┘
         │
         │ pause
         ▼
    ┌────────┐
    │ PAUSED │
    └───┬────┘
        │
      resume
        │
        ▼
   ┌─────────┐
   │BUILDING │
   └─────────┘
```

***

## 🛠️ Testing Tools

### Option 1: cURL (Command Line)

**Pros:** Available everywhere, scriptable
**Cons:** Verbose, requires manual token management

```bash theme={null}
# Set variables for easy reuse
export SERVER="https://your-server"
export TOKEN="your-token-here"

# Make a request
curl -X GET "$SERVER/api/pipelines/" \
  -H "Authorization: Bearer $TOKEN" -k
```

### Option 2: HTTPie (Command Line - User Friendly)

**Pros:** Simpler syntax, better output
**Cons:** Requires installation

```bash theme={null}
# Install
pip install httpie

# Use
http GET https://your-server/api/pipelines/ \
  Authorization:"Bearer $TOKEN"
```

### Option 3: Postman (GUI)

**Pros:** Visual interface, request collections
**Cons:** Requires download

1. Download Postman
2. Create a new request
3. Set method (GET, POST, etc.)
4. Enter URL: `https://your-server/api/pipelines/`
5. Add header: `Authorization: Bearer YOUR_TOKEN`
6. Add JSON body for POST/PATCH
7. Click "Send"

### Option 4: Python Script

**Pros:** Full programming capabilities
**Cons:** Requires Python knowledge

```python theme={null}
import requests

# Configuration
SERVER = "https://your-server"
TOKEN = "your-token-here"

headers = {
    "Authorization": f"Bearer {TOKEN}",
    "Content-Type": "application/json"
}

# Get all pipelines
response = requests.get(
    f"{SERVER}/api/pipelines/",
    headers=headers,
    verify=False  # Only for self-signed certificates
)

print(response.json())

# Create a pipeline
pipeline_data = {
    "name": "my-pipeline",
    "team_id": "your-team-id",
    "source_name": "my-source",
    "source_type": "KAFKA_SOURCE",
    "source_config": {...}
}

response = requests.post(
    f"{SERVER}/api/pipelines/",
    headers=headers,
    json=pipeline_data,
    verify=False
)

print(response.json())
```

***

## 📚 API Reference

### Authentication Endpoints

| Method | Endpoint                | Description                |
| ------ | ----------------------- | -------------------------- |
| `POST` | `/auth/register`        | Create a new user account  |
| `POST` | `/auth/jwt/login`       | Login and get access token |
| `POST` | `/auth/jwt/logout`      | Logout (invalidate token)  |
| `POST` | `/auth/forgot-password` | Request password reset     |
| `POST` | `/auth/reset-password`  | Reset password with token  |

### User Endpoints

| Method   | Endpoint                          | Description                    |
| -------- | --------------------------------- | ------------------------------ |
| `GET`    | `/users`                          | List all users (paginated)     |
| `GET`    | `/users/{id}`                     | Get user details               |
| `PATCH`  | `/users/{id}`                     | Update user                    |
| `DELETE` | `/users/{id}`                     | Delete user                    |
| `POST`   | `/users/me/change-password`       | Change current user's password |
| `POST`   | `/users/me/service-account-token` | Create service account token   |
| `GET`    | `/users/export-all`               | Export all data (admin only)   |
| `POST`   | `/users/import-all`               | Import all data (admin only)   |

### Environment Endpoints

| Method   | Endpoint                       | Description                       |
| -------- | ------------------------------ | --------------------------------- |
| `GET`    | `/envs/`                       | List all environments (paginated) |
| `POST`   | `/envs/`                       | Create a new environment          |
| `GET`    | `/envs/{id}`                   | Get environment details           |
| `GET`    | `/envs/filter-one?name={name}` | Get environment by name           |
| `PATCH`  | `/envs/{id}`                   | Update environment                |
| `DELETE` | `/envs/{id}`                   | Delete environment                |
| `POST`   | `/envs/check-byok-credentials` | Check BYOK credentials            |

### Team Endpoints

| Method   | Endpoint                        | Description                |
| -------- | ------------------------------- | -------------------------- |
| `GET`    | `/teams/`                       | List all teams (paginated) |
| `POST`   | `/teams/`                       | Create a new team          |
| `GET`    | `/teams/{id}`                   | Get team details           |
| `GET`    | `/teams/filter-one?name={name}` | Get team by name           |
| `PATCH`  | `/teams/{id}`                   | Update team                |
| `DELETE` | `/teams/{id}`                   | Delete team                |

### Team Member Endpoints

| Method   | Endpoint                        | Description          |
| -------- | ------------------------------- | -------------------- |
| `GET`    | `/teams/{id}/members`           | List team members    |
| `POST`   | `/teams/{id}/members/bulk`      | Add multiple members |
| `DELETE` | `/teams/{id}/members/{user_id}` | Remove a member      |

### Pipeline Endpoints

| Method      | Endpoint                               | Description                          |
| ----------- | -------------------------------------- | ------------------------------------ |
| `GET`       | `/pipelines/`                          | List all pipelines (paginated)       |
| `GET`       | `/pipelines/count-status`              | Count pipelines by status            |
| `POST`      | `/pipelines/`                          | Create a new pipeline                |
| `GET`       | `/pipelines/{id}`                      | Get pipeline details                 |
| `GET`       | `/pipelines/filter-one?name={name}`    | Get pipeline by name                 |
| `PATCH`     | `/pipelines/{id}`                      | Update pipeline                      |
| `DELETE`    | `/pipelines/{id}`                      | Delete pipeline                      |
| `POST`      | `/pipelines/{id}/start`                | Start pipeline (202)                 |
| `POST`      | `/pipelines/{id}/pause`                | Pause pipeline (202)                 |
| `WebSocket` | `/pipelines/{id}/logs`                 | Stream pipeline logs                 |
| `GET`       | `/pipelines/{id}/status`               | Get pipeline status (deprecated)     |
| `GET`       | `/pipelines/{id}/configuration`        | Export pipeline config               |
| `POST`      | `/pipelines/{id}/configuration`        | Import pipeline config               |
| `GET`       | `/pipelines/{id}/worker-config`        | Get worker config                    |
| `GET`       | `/pipelines/status/all-non-draft`      | List non-draft pipelines with status |
| `GET`       | `/pipelines/{id}/latency-informations` | Get pipeline latency info            |

### Connector Endpoints

| Method      | Endpoint                                | Description                  |
| ----------- | --------------------------------------- | ---------------------------- |
| `GET`       | `/connectors/source-config`             | List source configurations   |
| `GET`       | `/connectors/target-config`             | List target configurations   |
| `GET`       | `/connectors/filter-one?name={name}`    | Get connector by name        |
| `GET`       | `/connectors/{id}`                      | Get connector by ID          |
| `GET`       | `/connectors/{id}/source-worker-config` | Get source worker config     |
| `GET`       | `/connectors/{id}/target-worker-config` | Get target worker config     |
| `POST`      | `/connectors/`                          | Create connector             |
| `PATCH`     | `/connectors/{id}`                      | Update connector             |
| `DELETE`    | `/connectors/{id}`                      | Delete connector             |
| `POST`      | `/connectors/{id}/start`                | Start connector worker (202) |
| `POST`      | `/connectors/{id}/stop`                 | Stop connector worker (202)  |
| `WebSocket` | `/connectors/{id}/logs`                 | Stream connector logs        |

### Subscription Endpoints

| Method   | Endpoint                       | Description                    |
| -------- | ------------------------------ | ------------------------------ |
| `GET`    | `/subscriptions/`              | List subscriptions (paginated) |
| `GET`    | `/subscriptions/{id}`          | Get subscription details       |
| `POST`   | `/subscriptions/`              | Create subscription            |
| `POST`   | `/subscriptions/subscribe-all` | Bulk create subscriptions      |
| `PATCH`  | `/subscriptions/{id}`          | Update subscription            |
| `DELETE` | `/subscriptions/{id}`          | Delete subscription            |
| `POST`   | `/subscriptions/{id}/start`    | Start subscription (202)       |
| `POST`   | `/subscriptions/{id}/pause`    | Pause subscription (202)       |

### DataModel Endpoints

| Method   | Endpoint                       | Description                 |
| -------- | ------------------------------ | --------------------------- |
| `GET`    | `/datamodels/`                 | List datamodels (paginated) |
| `GET`    | `/datamodels/{id}`             | Get datamodel by ID         |
| `GET`    | `/datamodels/{id}/monitoring`  | Get datamodel monitoring    |
| `GET`    | `/datamodels/{id}/schema`      | Get datamodel schema        |
| `PATCH`  | `/datamodels/{id}/error-table` | Update error table config   |
| `DELETE` | `/datamodels/{id}`             | Delete datamodel            |

### SMT/Transformation Endpoints

| Method | Endpoint              | Description                   |
| ------ | --------------------- | ----------------------------- |
| `POST` | `/smt/process_mapper` | Process mapper transformation |

### Connector Type Specific Endpoints

#### Kafka Source

| Method | Endpoint                                          | Description                    |
| ------ | ------------------------------------------------- | ------------------------------ |
| `POST` | `/connector-types/kafka-source/check-credentials` | Check Kafka source credentials |
| `POST` | `/connector-types/kafka-source/fetch-messages`    | Fetch messages from Kafka      |
| `POST` | `/connector-types/kafka-source/list-topics`       | List Kafka topics              |

#### Oracle Target

| Method | Endpoint                                           | Description                     |
| ------ | -------------------------------------------------- | ------------------------------- |
| `POST` | `/connector-types/oracle-target/check-credentials` | Check Oracle target credentials |

***

## 🔧 Connector Configurations

### KAFKA\_SOURCE Configuration

```json theme={null}
{
  "topic": "user-events",
  "consumer_group_id": "pipeline-consumer",
  "bootstrap_servers": "kafka1:9092,kafka2:9092",
  "security_protocol": "SASL_SSL",
  "sasl_mechanism": "PLAIN",
  "sasl_username": "kafka_user",
  "sasl_password": "kafka_pass",
  "include_metadata": false
}
```

**Field Descriptions:**

| Field               | Type    | Required | Description                                                        |
| ------------------- | ------- | -------- | ------------------------------------------------------------------ |
| `topic`             | string  | ✅        | Kafka topic to consume from                                        |
| `bootstrap_servers` | string  | ✅        | Comma-separated list of Kafka brokers                              |
| `consumer_group_id` | string  | ⬜        | Consumer group ID (auto-generated if not provided)                 |
| `security_protocol` | string  | ⬜        | `PLAINTEXT`, `SASL_PLAINTEXT`, `SASL_SSL`, `SSL`                   |
| `sasl_mechanism`    | string  | ⬜        | `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, `GSSAPI`, `OAUTHBEARER` |
| `sasl_username`     | string  | ⬜        | SASL username (required if using SASL)                             |
| `sasl_password`     | string  | ⬜        | SASL password (required if using SASL)                             |
| `include_metadata`  | boolean | ⬜        | Include Kafka metadata in messages (default: false)                |

***

### KAFKA\_TARGET Configuration

```json theme={null}
{
  "bootstrap_server": "kafka1:9092",
  "security_protocol": "SASL_SSL",
  "sasl_mechanism": "PLAIN",
  "sasl_username": "kafka_user",
  "sasl_password": "kafka_pass",
  "ca_cert": "",
  "group_id": "consumer-group"
}
```

**Field Descriptions:**

| Field               | Type   | Required | Description                                                        |
| ------------------- | ------ | -------- | ------------------------------------------------------------------ |
| `bootstrap_server`  | string | ✅        | Kafka broker address                                               |
| `security_protocol` | string | ⬜        | `PLAINTEXT`, `SASL_PLAINTEXT`, `SASL_SSL`, `SSL`                   |
| `sasl_mechanism`    | string | ⬜        | `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, `GSSAPI`, `OAUTHBEARER` |
| `sasl_username`     | string | ⬜        | SASL username                                                      |
| `sasl_password`     | string | ⬜        | SASL password                                                      |
| `ca_cert`           | string | ⬜        | CA certificate for SSL                                             |
| `group_id`          | string | ⬜        | Consumer group ID                                                  |

***

### ORACLE\_TARGET Configuration

```json theme={null}
{
  "database": "ORCL",
  "user": "etl_user",
  "password": "etl_pass",
  "host": "oracle.company.com",
  "port": "1521",
  "server_name": "XE",
  "server_id": "oraclesrv01"
}
```

**Field Descriptions:**

| Field         | Type   | Required | Description                    |
| ------------- | ------ | -------- | ------------------------------ |
| `database`    | string | ✅        | Oracle database name           |
| `user`        | string | ✅        | Username for Oracle            |
| `password`    | string | ✅        | Password for Oracle            |
| `host`        | string | ✅        | Hostname or IP address         |
| `port`        | string | ✅        | Port number (typically `1521`) |
| `server_name` | string | ✅        | Service name (e.g., `XE`)      |
| `server_id`   | string | ✅        | Server identifier/SID          |

***

**Mapper Column Config :**

| Field         | Type    | Description                                                           |
| ------------- | ------- | --------------------------------------------------------------------- |
| `key`         | string  | Column name in target                                                 |
| `path`        | string  | JSON path to extract value (e.g., `$.field.nested`)                   |
| `static`      | string  | Static value (alternative to path)                                    |
| `cast`        | string  | Cast to: `string`, `int`, `float`, `bool`, `date`, `datetime`, `time` |
| `cast_format` | string  | Format for casting (e.g., date format)                                |
| `primary_key` | boolean | Is this a primary key column                                          |
| `nullable`    | boolean | Can this column be null                                               |

***

## 🎯 Common Patterns

### Pattern 1: List Resources with Filters

```bash theme={null}
# Get all pipelines in a specific state
curl -X GET "https://your-server/api/pipelines/?state=live" \
  -H "Authorization: Bearer $TOKEN" -k

# Get pipelines for a specific team
curl -X GET "https://your-server/api/pipelines/?team_id=$TEAM_ID" \
  -H "Authorization: Bearer $TOKEN" -k

# Search pipelines by name
curl -X GET "https://your-server/api/pipelines/?search=kafka" \
  -H "Authorization: Bearer $TOKEN" -k
```

### Pattern 2: Pagination

```bash theme={null}
# Get first page (default 50 items)
curl -X GET "https://your-server/api/pipelines/?skip=0&limit=50" \
  -H "Authorization: Bearer $TOKEN" -k

# Get second page
curl -X GET "https://your-server/api/pipelines/?skip=50&limit=50" \
  -H "Authorization: Bearer $TOKEN" -k
```

### Pattern 3: Error Handling

```python theme={null}
import requests

try:
    response = requests.get(
        "https://your-server/api/pipelines/",
        headers={"Authorization": f"Bearer {TOKEN}"},
        verify=False
    )
    response.raise_for_status()  # Raises exception for 4xx/5xx
    data = response.json()
except requests.exceptions.HTTPError as e:
    print(f"HTTP Error: {e}")
    print(f"Response: {e.response.text}")
except requests.exceptions.ConnectionError as e:
    print(f"Connection Error: {e}")
except Exception as e:
    print(f"Error: {e}")
```

### Pattern 4: Bulk Operations

```python theme={null}
# Create multiple pipelines
pipeline_configs = [
    {"name": "pipeline-1", "team_id": TEAM_ID, ...},
    {"name": "pipeline-2", "team_id": TEAM_ID, ...},
    {"name": "pipeline-3", "team_id": TEAM_ID, ...}
]

for config in pipeline_configs:
    response = requests.post(
        f"{SERVER}/api/pipelines/",
        headers=headers,
        json=config
    )
    print(f"Created: {response.json()['id']}")
```

### Pattern 5: Test Connector Credentials Before Creating

```bash theme={null}
# Test Kafka source credentials
curl -X POST "https://your-server/api/connector-types/kafka-source/check-credentials" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "bootstrap_servers": "kafka:9092",
    "topic": "test-topic",
    "security_protocol": "PLAINTEXT"
  }' -k

# Test Oracle target credentials
curl -X POST "https://your-server/api/connector-types/oracle-target/check-credentials" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "host": "oracle.company.com",
    "port": "1521",
    "database": "ORCL",
    "user": "etl_user",
    "password": "etl_pass",
    "server_name": "XE",
    "server_id": "oraclesrv01"
  }' -k
```

***

## 🐛 Troubleshooting

### Issue: "401 Unauthorized"

**Cause:** Token is missing, expired, or invalid

**Solution:**

1. Check that you included the `Authorization` header
2. Verify the token format: `Bearer YOUR_TOKEN`
3. Log in again to get a fresh token

```bash theme={null}
curl -X POST "https://your-server/api/auth/jwt/login" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d 'username=YOUR_EMAIL&password=YOUR_PASSWORD' -k
```

***

### Issue: "403 Forbidden"

**Cause:** You don't have permission for this operation

**Solution:**

1. Verify you're a member of the team
2. Check if you have the required role (owner vs member)
3. Contact the team owner to grant permissions

***

### Issue: "422 Validation Error"

**Cause:** Request data doesn't meet validation requirements

**Solution:**

1. Check the error response for specific field errors
2. Verify all required fields are provided
3. Ensure data types match (UUID, string, boolean, etc.)

**Example Error Response:**

```json theme={null}
{
  "detail": [
    {
      "loc": ["body", "source_name"],
      "msg": "field required",
      "type": "value_error.missing"
    }
  ]
}
```

**Fix:**

```json theme={null}
{
  "name": "my-pipeline",
  "team_id": "uuid-here",
  "source_name": "my-source",
  "source_type": "KAFKA_SOURCE",
  "source_config": {...}
}
```

***

### Issue: Invalid pipeline name

**Cause:** Pipeline name contains invalid characters

**Solution:** Pipeline names must:

* Contain only alphanumeric characters, hyphens (`-`), and underscores (`_`)
* Be maximum 255 characters long
* Not be empty

**Valid names:**

* `kafka-to-oracle-pipeline`
* `user_events_pipeline_v2`
* `Pipeline123`

**Invalid names:**

* `pipeline name` (contains space)
* `pipeline.name` (contains dot)
* `pipeline@name` (contains special character)

***

### Issue: Cannot specify both existing and new connector

**Cause:** You provided both `existing_source_id` AND `source_name/type/config`

**Solution:** Choose ONE approach:

**Option A (Existing):**

```json theme={null}
{
  "existing_source_id": "uuid-here"
}
```

**Option B (New):**

```json theme={null}
{
  "source_name": "my-source",
  "source_type": "KAFKA_SOURCE",
  "source_config": {...}
}
```

***

### Issue: Pipeline stuck in "BUILDING" state

**Cause:** Deployment failed or is taking longer than expected

**Solution:**

1. Check pipeline logs via WebSocket
2. Verify connector configurations are correct
3. Check infrastructure resources (CPU, memory)

```bash theme={null}
# Get detailed status
curl -X GET "https://your-server/api/pipelines/$PIPELINE_ID" \
  -H "Authorization: Bearer $TOKEN" -k
```

***

### Issue: SSL Certificate Error

**Cause:** Self-signed certificate or untrusted CA

**Solution for curl:**

```bash theme={null}
# Add -k or --insecure flag
curl -X GET "https://your-server/api/pipelines/" \
  -H "Authorization: Bearer $TOKEN" -k
```

**Solution for Python:**

```python theme={null}
requests.get(url, verify=False)
```

**Solution for HTTPie:**

```bash theme={null}
http --verify=no GET https://your-server/api/pipelines/
```

***

## 📝 Complete Example Script

Here's a complete Python script that performs all steps:

```python theme={null}
#!/usr/bin/env python3
"""
Complete Popsink API example script
"""
import requests
import json
from typing import Optional

# Configuration
SERVER = "https://your-server"
VERIFY_SSL = False  # Set to True in production

class PopsinkClient:
    def __init__(self, server: str, verify_ssl: bool = True):
        self.server = server.rstrip('/')
        self.verify_ssl = verify_ssl
        self.token: Optional[str] = None

    def register(self, email: str, password: str) -> dict:
        """Register a new user"""
        response = requests.post(
            f"{self.server}/api/auth/register",
            json={
                "email": email,
                "password": password,
                "is_active": True
            },
            verify=self.verify_ssl
        )
        response.raise_for_status()
        return response.json()

    def login(self, email: str, password: str) -> str:
        """Login and get token"""
        response = requests.post(
            f"{self.server}/api/auth/jwt/login",
            data={
                "username": email,
                "password": password
            },
            verify=self.verify_ssl
        )
        response.raise_for_status()
        data = response.json()
        self.token = data["access_token"]
        return self.token

    def _headers(self) -> dict:
        """Get headers with auth token"""
        if not self.token:
            raise ValueError("Not logged in. Call login() first.")
        return {
            "Authorization": f"Bearer {self.token}",
            "Content-Type": "application/json"
        }

    def create_environment(self, name: str, description: str = "") -> dict:
        """Create an environment"""
        response = requests.post(
            f"{self.server}/api/envs/",
            headers=self._headers(),
            json={
                "name": name,
                "description": description,
                "use_retention": True
            },
            verify=self.verify_ssl
        )
        response.raise_for_status()
        return response.json()

    def create_team(self, name: str, description: str = "") -> dict:
        """Create a team"""
        response = requests.post(
            f"{self.server}/api/teams/",
            headers=self._headers(),
            json={
                "name": name,
                "description": description
            },
            verify=self.verify_ssl
        )
        response.raise_for_status()
        return response.json()

    def create_pipeline(self, name: str, team_id: str, config: dict) -> dict:
        """Create a pipeline"""
        pipeline_data = {
            "name": name,
            "team_id": team_id,
            **config
        }
        response = requests.post(
            f"{self.server}/api/pipelines/",
            headers=self._headers(),
            json=pipeline_data,
            verify=self.verify_ssl
        )
        response.raise_for_status()
        return response.json()

    def start_pipeline(self, pipeline_id: str) -> str:
        """Start a pipeline"""
        response = requests.post(
            f"{self.server}/api/pipelines/{pipeline_id}/start",
            headers=self._headers(),
            verify=self.verify_ssl
        )
        response.raise_for_status()
        return response.json()


def main():
    # Initialize client
    client = PopsinkClient(SERVER, verify_ssl=VERIFY_SSL)

    # 1. Register
    print("1. Registering user...")
    user = client.register("demo@example.com", "SecurePassword123!")
    print(f"   ✓ User created: {user['id']}")

    # 2. Login
    print("\n2. Logging in...")
    token = client.login("demo@example.com", "SecurePassword123!")
    print(f"   ✓ Logged in (token: {token[:20]}...)")

    # 3. Create environment
    print("\n3. Creating environment...")
    env = client.create_environment("demo-env", "Demo environment")
    print(f"   ✓ Environment created: {env['id']}")

    # 4. Create team
    print("\n4. Creating team...")
    team = client.create_team("Demo Team", "Demo team for testing")
    print(f"   ✓ Team created: {team['id']}")

    # 5. Create pipeline
    print("\n5. Creating pipeline...")
    pipeline_config = {
        "source_name": "demo-kafka-source",
        "source_type": "KAFKA_SOURCE",
        "source_config": {
            "topic": "demo-topic",
            "bootstrap_servers": "kafka:9092",
            "consumer_group_id": "demo-group",
            "security_protocol": "PLAINTEXT"
        },
        "target_name": "demo-kafka-target",
        "target_type": "KAFKA_TARGET",
        "target_config": {
            "bootstrap_server": "kafka:9092",
            "security_protocol": "PLAINTEXT"
        },
        "subscription_target_table_name": "events",
        "subscription_backfill": False
    }
    pipeline = client.create_pipeline("demo-pipeline", team['id'], pipeline_config)
    print(f"   ✓ Pipeline created: {pipeline['id']}")

    # 6. Start pipeline
    print("\n6. Starting pipeline...")
    state = client.start_pipeline(pipeline['id'])
    print(f"   ✓ Pipeline state: {state}")

    print("\n✓ All steps completed successfully!")


if __name__ == "__main__":
    main()
```

***

## 🎓 Next Steps

Now that you understand the basics:

1. **Explore the API:** Try listing resources, filtering, pagination
2. **Monitor Pipelines:** Use logs and status endpoints
3. **Handle Errors:** Implement proper error handling
4. **Test Credentials:** Use credential check endpoints before creating connectors
5. **Optimize:** Reuse connectors, batch operations
6. **Automate:** Create scripts or CI/CD pipelines

***

## 📞 Support

* **Documentation:** This guide
* **API Schema:** `https://your-server/api/docs` (Swagger UI)
* **OpenAPI Spec:** `https://your-server/api/openapi.json`

***

**Happy Data Processing! 🚀**


# Quickstart
Source: https://docs.popsink.com/quickstart

Launch your first Popsink pipeline in minutes with this guided tutorial.

Follow this quickstart guide to create a Popsink pipeline that streams data from a source to a target connector. You will use
the Popsink console, configure alerting, and validate that data flows end-to-end.

<Steps>
  <Step title="Sign in and create a workspace">
    1. Sign in to the [Popsink console](https://console.popsink.com) with your organization account.
    2. Create a new workspace or select an existing one. This workspace hosts the pipelines and credentials you configure.
  </Step>

  <Step title="Configure a source connector">
    1. Open **Connectors → Sources** and click **New Source**.
    2. Select the connector that matches your system. For example, choose **Postgres** to capture database changes.
    3. Provide the required credentials and connection parameters.
    4. Validate the connection and save the source.

    <Callout type="tip">
      Review the connector guides to learn about network prerequisites or advanced settings specific to each technology.
    </Callout>

    <Callout type="warning">
      Ensure you have workspace administrator rights (or equivalent) before creating and validating new connectors.
    </Callout>
  </Step>

  <Step title="Configure a target connector">
    1. Open **Connectors → Targets** and create a new target connector.
    2. Select the destination technology (for example **BigQuery** or **Snowflake**).
    3. Provide write credentials and define the storage location (dataset or schema) Popsink should use.
    4. Test the connection before saving the target.
  </Step>

  <Step title="Build your pipeline">
    1. Go to a **Target Connector** and click **Ingest Data**.
    2. Select the source data models.
    3. Save and start the pipeline.
  </Step>

  <Step title="Validate the data flow">
    * Inspect the dashboard to confirm that events are processed and delivered.
    * Check on the target system that tables or collections are updated as expected.
    * Monitor alerts and metrics to maintain long-term performance.
  </Step>
</Steps>

<AccordionGroup>
  <Accordion title="Connection verification fails">
    Confirm that the required ports are open and that the supplied credentials have the expected read or write privileges. Compare against the recommended configuration in the connector documentation.
  </Accordion>

  <Accordion title="The pipeline stays pending after launch">
    Make sure change data capture is enabled on the source and that quotas or limits on the target side are not exceeded. Review the related alerts to get incident details.
  </Accordion>
</AccordionGroup>

<CardGroup>
  <Card title="Connector guides" icon="plug-circle-bolt" href="/connectors/source/overview">
    Explore all source and target connectors, their prerequisites, and best practices.
  </Card>

  <Card title="Deployment guide" icon="building" href="/deployment">
    Choose the deployment model that best suits you (SaaS, BYOC, or on-premises) and prepare your environment.
  </Card>
</CardGroup>


# Popsink Roadmap
Source: https://docs.popsink.com/roadmap



This document outlines the upcoming features and strategic initiatives in the Popsink product roadmap. Each milestone contributes to enhancing reliability, scalability, and enterprise readiness for our data integration platform.

***

last updated : 23/04/2026

## Advanced Configuration Target

**Goal:** Provide fine-grained control over deployment and runtime configurations.

**Key Highlights:**

* Support for per-target resource tuning (memory, concurrency, retries).
* Environment-specific overrides for staging, QA, and production.
* Custom configuration templates for advanced users.

**Benefits:**

* Improved performance optimization.
* Simplified management of complex deployments.

***

## Terraform Provider

**Goal:** Enable full infrastructure automation for Popsink through Terraform.

**Key Highlights:**

* Manage Popsink resources (connectors, pipelines) as code.
* Integrate with existing CI/CD pipelines.
* Version-controlled infrastructure state.

**Benefits:**

* Reproducible environments.
* Streamlined deployment workflows.
* Reduced manual configuration.

***

## Observability

**Goal:** Improve insight into system operations and data flows.

**Key Highlights:**

* Enhanced logging with contextual metadata.
* Metrics export via OpenTelemetry.
* Real-time health dashboards for pipelines and connectors.

**Benefits:**

* Faster troubleshooting.
* Proactive performance monitoring.
* Transparent operational visibility.

***

## Monitoring

**Goal:** Deliver a unified monitoring experience across the Popsink platform.

**Key Highlights:**

* Built-in alerting and anomaly detection.
* Integration with external tools (Grafana, Prometheus, Datadog).
* Historical trend analysis for pipeline performance.

**Benefits:**

* Early detection of issues.
* Reduced downtime.
* Improved system reliability.

***

## Exactly Once Connectors

**Goal:** Guarantee “exactly-once” message delivery semantics for critical data pipelines.

**Key Highlights:**

* Idempotent processing mechanisms.
* Transactional delivery and checkpointing.
* Support for distributed systems and message queues.

**Benefits:**

* Elimination of duplicates and data loss.
* Strong data integrity guarantees.
* Compliance with financial and regulated data flows.

***

## Hybrid Deployment Self-Serve

**Goal:** Support hybrid and multi-cloud deployments.

**Key Highlights:**

* Ability to run Popsink across cloud and on-premise environments.
* Secure data processing without leaving customer networks.
* Seamless coordination between distributed agents.

**Benefits:**

* Flexibility for enterprise architectures.
* Compliance with data residency requirements.
* Cost optimization across environments.

***

## SSO Self-Serve

**Goal:** Empower organizations to manage Single Sign-On independently.

**Key Highlights:**

* Self-service SSO configuration via admin dashboard.
* Support for SAML, OIDC, and OAuth2 providers.
* Role-based access synchronization with identity providers.

**Benefits:**

* Simplified user management.
* Stronger authentication controls.
* Reduced dependency on support teams.


# Setup a IBMi system
Source: https://docs.popsink.com/snowflake-ibmi/db2-setup



if you're new to IBMi/AS400 or don't currently have a system to use, this section will guide you through setting a test
environment using the [PUB400.com](https://www.pub400.com) service.

## What is IBMi/AS400 ?

IBMi/AS400 (now commonly referred to as IBMi on Power Systems) is a midrange computer platform developed by IBM. Originally introduced in 1988 as the AS/400 (Application System/400), it was designed for business applications, especially in industries like finance, manufacturing, and retail.

The system runs on the IBMi operating system, which is known for its stability, security, and integrated database (DB2). It supports both traditional RPG and COBOL applications as well as modern development using Java, PHP, and SQL.

Although the hardware and name have evolved over the years (from AS/400 to iSeries to System i and now IBM Power Systems running IBMi), many organizations still refer to it as AS/400 due to its long legacy.

It's therefore an entirely different platform than traditional PC computers. It's still used widely to this day in many sectors.

## What is PUB400 ?

[PUB400.com](https://www.pub400.com) is a public, free IBMi (formerly AS/400) server you can use to learn, experiment, and develop.

It’s basically a playground for IBMi: a system running IBMi version 7.5, made accessible to the community so people can try out IBMi tools,
write programs (RPG, COBOL, CL, SQL, etc.), use open‑source/modern tooling, and generally build skills.

## 1. PUB400.com sign-up & connection test

Opening a free account is really easy, head to sign-up page and fill up this form :

<img alt="image" />

You'll receive a first email to confirm your email address, and then, another one to give you more details
on how to use your new test system.

IBMi/AS400 systems often requires you to go through a dedicated tn5250 terminal emulator. For our use cases,
you'll need the terminal to enable [journaling](journaling) on your source tables.

We recommend you to use a JAVA-based multi-platform terminal [tn5250j](http://tn5250j.org/)
Make sure JAVA is installed, simply download the zip distribution and unzip it to your install folder.

Run the command `java -jar ./tn5250j.jar` and configure your connection to pub400.com. The logon window looks like this :

<img alt="image" />

On first login, your password will need to be changed, use \[TAB] to move for one field to another.

On the welcome page, try running the command `DSPLIBL` to display the list of libraries/schemas.

<img alt="image" />

Now the result will look like this

<img alt="image" />

That's great, it means that PUB400 has already created a sample schema called `${YOUR_USERNAME}1`

## 2. SQL connection

IBMi has an embedded DB2 SQL database tightly coupled with the system. Tables are files and vice versa. That's why,
there's no need to install a SQL server.

### Query test

Now, try to connect to our library by using an ordinary JDBC-based SQL client. In this example,
connection string is simply `jdbc:as400://pub400.COM;libraries={YOUR_USERNAME}1`. Here's a screenshot of [DBeaver](https://dbeaver.io/)

<img alt="image" />

### Build schema

Time to build our first test database. This database will be used later on to do replication tests with Snowflake.
Run the following creation script :

```oracle-sql theme={null}
-- 1. Categories Table
CREATE TABLE Categories (
    CategoryID INTEGER GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1) PRIMARY KEY,
    CategoryName VARCHAR(100) NOT NULL,
    Description VARCHAR(500)
);

-- 2. Suppliers Table
CREATE TABLE Suppliers (
    SupplierID INTEGER GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1) PRIMARY KEY,
    SupplierName VARCHAR(150) NOT NULL,
    ContactName VARCHAR(100),
    Address VARCHAR(200),
    Phone VARCHAR(50),
    Email VARCHAR(100)
);

-- 3. Products Table
CREATE TABLE Products (
    ProductID INTEGER GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1) PRIMARY KEY,
    ProductName VARCHAR(150) NOT NULL,
    CategoryID INTEGER,
    SupplierID INTEGER,
    UnitPrice DECIMAL(10, 2),
    ReorderLevel INTEGER DEFAULT 10,
    Discontinued SMALLINT DEFAULT 0,
    FOREIGN KEY (CategoryID) REFERENCES Categories(CategoryID),
    FOREIGN KEY (SupplierID) REFERENCES Suppliers(SupplierID)
);

-- 4. Stock Table
CREATE TABLE Stock (
    StockID INTEGER GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1) PRIMARY KEY,
    ProductID INTEGER NOT NULL,
    QuantityInStock INTEGER NOT NULL,
    LastUpdated TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (ProductID) REFERENCES Products(ProductID)
);

-- 5. Purchases Table
CREATE TABLE Purchases (
    PurchaseID INTEGER GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1) PRIMARY KEY,
    ProductID INTEGER NOT NULL,
    SupplierID INTEGER NOT NULL,
    Quantity INTEGER NOT NULL,
    PurchaseDate DATE NOT NULL,
    UnitCost DECIMAL(10, 2),
    FOREIGN KEY (ProductID) REFERENCES Products(ProductID),
    FOREIGN KEY (SupplierID) REFERENCES Suppliers(SupplierID)
);

-- 6. Sales Table
CREATE TABLE Sales (
    SaleID INTEGER GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1) PRIMARY KEY,
    ProductID INTEGER NOT NULL,
    Quantity INTEGER NOT NULL,
    SaleDate DATE NOT NULL,
    UnitPrice DECIMAL(10, 2),
    FOREIGN KEY (ProductID) REFERENCES Products(ProductID)
);
```

### Fill up the database

Now let's insert simulated data :

```oracle-sql theme={null}
INSERT INTO Categories (CategoryName, Description) VALUES
('Electronics', 'Devices and gadgets'),
('Groceries', 'Daily consumables and food items'),
('Stationery', 'Office and school supplies');


INSERT INTO Suppliers (SupplierName, ContactName, Address, Phone, Email) VALUES
('TechSource Ltd.', 'Alice Johnson', '123 Tech Ave', '555-1234', 'alice@techsource.com'),
('FoodWorld Inc.', 'Bob Smith', '789 Market St', '555-5678', 'bob@foodworld.com'),
('OfficePlus', 'Carol Lee', '456 Office Rd', '555-9012', 'carol@officeplus.com');


INSERT INTO Products (ProductName, CategoryID, SupplierID, UnitPrice, ReorderLevel, Discontinued) VALUES
('USB Flash Drive 32GB', 1, 1, 12.99, 20, 0),
('Organic Apples 1kg', 2, 2, 3.49, 50, 0),
('Notebook A4', 3, 3, 1.25, 100, 0);

INSERT INTO Stock (ProductID, QuantityInStock) VALUES
(1, 150),
(2, 300),
(3, 500);

INSERT INTO Purchases (ProductID, SupplierID, Quantity, PurchaseDate, UnitCost) VALUES
(1, 1, 200, DATE('2025-09-10'), 10.00),
(2, 2, 500, DATE('2025-09-15'), 2.50),
(3, 3, 1000, DATE('2025-09-12'), 0.90);

INSERT INTO Sales (ProductID, Quantity, SaleDate, UnitPrice) VALUES
(1, 50, DATE('2025-09-18'), 14.99),
(2, 100, DATE('2025-09-19'), 3.99),
(3, 250, DATE('2025-09-20'), 1.50);
```

## What's next ?

Well done, you've just created a functional IBMi test environment. Both SQL and command line are functional.
From there, it's straightforward to develop basic queries and applications with JDBC for instance.
A lot of valuable information can be found on [PUB400.COM](https://www.pub400.com) to help you dive deeper into the [command line](https://www.ibm.com/docs/en/i/7.4.0?topic=product-i-commands).

In our next step, we will use the terminal to configure our source tables with [journaling](journaling) enabled.


# DB2 Journaling
Source: https://docs.popsink.com/snowflake-ibmi/journaling



## What is journaling ?

Journaling is a mechanism used in database systems to log all data modification operations (such as INSERT, UPDATE, and DELETE) to a sequential transaction log (journal) separate from the base tables. This log provides a reliable, auditable trail of changes, supports crash recovery, and enables data consistency across system failures.

For mission-critical environments, proper journaling setup and receiver management are essential to ensure data integrity and minimize downtime.

## What is Change Data Capture ? (CDC)

Change data capture is a technique involving listening to a DB journal to extract the changes since a certain point in time and apply these changes to an external system.

This is a particularly efficient way to replicate (i.e. synchronize) one database onto another. No need for a sophisticated setup on the source database, a pre-existing journal can be used as it's often the case on critical databases.

The synchronization doesn't need to periodically scan a list of tables as the journal centralizes the changes.

## How is Journaling implemented on IBMi DB2 ?

DB2 tracks **multiple tables** with a designated **journal** witch itself delegates the storage to a **receiver**.

While it's possible to have multiple journals assigned to tables belonging to the same library (or schema), it's recommended to create a single journal for an entire DB2 instance or a single journal per schema.

A schema often reflects a single unit of work with few transactions overlapping schemas.

<img alt="image" />

## Configure DB2 Journal

On IBMi, everything is an object, that is a file under a library. Consider libraries as the equivalent of folders in recent systems. In the following examples, use your own library name as defined by your DB2 setup.

Use a tn5250 terminal emulator to connect to your IBMi environment and run the following commands.

### 1.  Create a Journal Receiver

This command creates a receiver named MY\_RCVER under the library POPSINK1 :

`CRTJRNRCV POPSINK1/MY_RCVER`

### 2. Create a Journal

This command creates a journal named MY\_JOURNAL attached to MY\_RCVER under the library POPSINK1 :

`CRTJRN POPSINK1/MY_JOURNAL POPSINK1/MY_RCVER`

### 3. Enable Journaling on source tables

For each table to replicate named \$TABLE, run the command :

`STRJRNPF POPSINK1/${TABLE} POPSINK1/MY_JOURNAL`

That is, based on the tables created previously in [DB2 setup](db2-setup) :

```
STRJRNPF POPSINK1/CATEGORIES POPSINK1/MY_JOURNAL
STRJRNPF POPSINK1/SUPPLIERS POPSINK1/MY_JOURNAL
STRJRNPF POPSINK1/PRODUCTS POPSINK1/MY_JOURNAL
STRJRNPF POPSINK1/STOCK POPSINK1/MY_JOURNAL
STRJRNPF POPSINK1/PURCHASES POPSINK1/MY_JOURNAL
STRJRNPF POPSINK1/SALES POPSINK1/MY_JOURNAL
```

### 4. Give relevant rights to the CDC user

Your account (here POPSINK for example) needs to be allowed to read from the journal :

```
GRTOBJAUT OBJ(POPSINK1) OBJTYPE(*LIB) USER(POPSINK) AUT(*EXECUTE)
GRTOBJAUT OBJ(POPSINK1/*ALL) OBJTYPE(*JRNRCV) USER(POPSINK) AUT(*USE)
GRTOBJAUT OBJ(POPSINK1/MY_JOURNAL) OBJTYPE(*JRN) USER(POPSINK) AUT(*USE *OBJEXIST)
GRTOBJAUT OBJ(POPSINK1/MY_JOURNAL) OBJTYPE(*JRN) USER(POPSINK) AUT(*OBJEXIST)

GRTOBJAUT OBJ(POPSINK1/*ALL) OBJTYPE(*FILE) USER(POPSINK) AUT(*USE)
```

### 5. Switch journaling from \*AFTER to \*BOTH

By default, journaling only stores the image after the change, not the one before. This can make certain operations complex, for instance primary key updates - these cannot be handled unless both images are available.

Besides, with both images, Snowflake replication will provide more information on DELETE operations.

For each table to replicate named \$TABLE, run the command :

```
CHGJRNOBJ OBJ((POPSINK1/${TABLE} *FILE)) ATR(*IMAGES) IMAGES(*BOTH)
```

That is, based on the tables created previously in [DB2 setup](db2-setup) :

```
CHGJRNOBJ OBJ((POPSINK1/CATEGORIES *FILE)) ATR(*IMAGES) IMAGES(*BOTH)
CHGJRNOBJ OBJ((POPSINK1/SUPPLIERS *FILE)) ATR(*IMAGES) IMAGES(*BOTH)
CHGJRNOBJ OBJ((POPSINK1/PRODUCTS *FILE)) ATR(*IMAGES) IMAGES(*BOTH)
CHGJRNOBJ OBJ((POPSINK1/STOCK *FILE)) ATR(*IMAGES) IMAGES(*BOTH)
CHGJRNOBJ OBJ((POPSINK1/PURCHASES *FILE)) ATR(*IMAGES) IMAGES(*BOTH)
CHGJRNOBJ OBJ((POPSINK1/SALES *FILE)) ATR(*IMAGES) IMAGES(*BOTH)
```

## Conclusion

Journaling requires additional I/O and storage space. While not covered in this guide, monitoring and periodically deleting old journal entries is essential to save disk space.

Well Done ! you are finally ready to start replicating data from IBMi DB2 using Popsink technology.

Jump straight to our [quickstart](quickstart) if you've installed Popsink's Snowflake IBMi connector.


# Quickstart
Source: https://docs.popsink.com/snowflake-ibmi/quickstart



# Quickstart

You've just downloaded Popsink's connector to IBMi DB2 from Snowflake's Marketplace, and you're now eager to run your
first replication. Then, you've come to right the place !

On first launch, the application will welcome you with a step-by-step configuration wizard that you can pause and resume any
time, your progress will be saved.

If you're entirely new to
IBMi/AS400 and currently have no system to connect to, please head to [this page](./db2_setup.md), follow the instructions
and start experimenting with your new DB2 system.

If not, you've already got a DB2 system in place, and you want to replicate it, please continue.

## 1. Prerequisites

In this important step, the user goes through a check-list of actions that need to be taken before moving on to the next
steps. Not everything can be setup through the user interface, often for security reasons.

<img alt="image" />

### 1.1 Enable Journaling

It's a vital step to capture changes from the source databases with no additional overhead. Please head to [journaling](./journaling.md),
if you need help setting it up on your DB2 environment.

Take notes of the journaled tables you'd like to replicate.

### 1.2 Find an available name for the target database

Snowflake's security prevents an app to list all the existing databases and schemas. That's why your administrator needs to run
`SHOW DATABASES` to confirm that your choice of target database name is valid.

When you're ready you can mark the prerequisites as **'Completed'** and move to the next step.

## 2. Connector's Configuration

On this page, you will configure a few simple but essential settings :

* Grant privileges to create the target database and run synchronization tasks.
* Define what warehouse to connect to. The running costs of the app will be billed to this warehouse.
* Define the target database name. You **must** choose an available name.
* Define the schema name storing the replicated tables.

<img alt="image" />

When you're finished, click on **'Configure'**.

## 3. Firewall & Credentials Configuration

Next step enables your connection to your source IBMi system. You will need 2 things :

* A network access to your hostname.
* A username & password to an account with rights to read the DB2 journal.

Snowflake has an embedded system of [external access integration (EAI)](https://docs.snowflake.com/en/developer-guide/external-network-access/creating-using-external-network-access) that securely declares the port your
connection requires, as well as credential data (here a simple username/password pair, the only mean to connect to IBMi)

<img alt="image" />

When you've entered all the fields, clicking on **'Enable access'** will take you to the next step where you simply need
to refresh your page. During this restart, you'll be asked to allow access to your source host, click on **'Review'**
, then **'Next'**. In the description page of app, please click on **'Launch App'**.

<img alt="image" />

<img alt="image" />

After this simple step your source data will become accessible.

## 4. DB2 Connection's Configuration

Now your connection to DB2 is valid, it's time to define the source data you want to replicate in this simple step :

1. Choose a source schema.
2. Enter a comma-separated list of journaled tables belonging to this schema.

These tables have to be assigned to the same DB2 journal.

Click on **'Connect'** to move to the last step.

<img alt="image" />

## 5. Finalize

To finish with, a few settings can be modified before starting the replication :

1. In case a source table is really large, or you only want to capture changes, it's possible to skip the initial snapshot with the following checkbox.
2. By default, synchronization period is 20 seconds, meaning the changes will be read from DB2 every 20 seconds by a Snowflake's scheduled task. It's possible to change this setting according to your latency and power requirements. A larger period means higher latencies but less compute usage.

<img alt="image" />

Click on **'Finalize Configuration'** to start the replication.

## Monitor the replication

Snowflake will now run 3 different tasks (or 2 if you've decided to skip the initial snapshot) :

* **`AS400_CREATE_TARGET`** - creates the target tables in the target schema. replicated tables will be prefixed by `POPSINK_`.
* **`AS400_SNAPSHOT`** - selects all the source data at startup time and fills the target tables.
* **`AS400_SYNC`** - polls the DB2 journal to load all the changes since last synchronization and update target tables accordingly.

They can be viewed under the Task menu of Snowflake.


# Mapper
Source: https://docs.popsink.com/transform

Transform and reshape your data before it lands in the target system

The **Mapper** is a column-level transformation engine that runs inside a Subscription. It reads a raw CDC message (a JSON object) and produces one or more flat rows according to a mapping you define. Each row is what ultimately gets written to the target table.

## How it works

A mapper configuration is a list of **column rules**. Each rule declares:

* the **output column name** (`key`)
* **where the value comes from** — a path into the message, a static string, or a computed expression
* optional **transformations** applied to the extracted value (cast, hash, encrypt, concatenate)

Popsink evaluates all rules against every incoming message and emits the resulting rows to the target.

```json theme={null}
[
  { "key": "id",         "path": "user_id",        "primary_key": true },
  { "key": "email",      "path": "user|email" },
  { "key": "source",     "static": "crm" },
  { "key": "created_at", "path": "created_at",      "cast": "datetime" }
]
```

The example above maps three fields from the source message and injects a hardcoded `source` column.

***

## Column configuration reference

| Field                | Type    | Required | Description                                                                                       |
| -------------------- | ------- | -------- | ------------------------------------------------------------------------------------------------- |
| `key`                | string  | Yes      | Output column name in the target table                                                            |
| `path`               | string  | No       | Pipe-separated JSON path to the source value                                                      |
| `static`             | string  | No       | Hardcoded value (used when there is no source path)                                               |
| `cast`               | string  | No       | Type conversion: `string`, `int`, `float`, `bool`, `date`, `datetime`, `time`                     |
| `cast_format`        | string  | No       | Format string for `date`, `datetime`, or `time` casts (e.g. `%Y-%m-%d`)                           |
| `primary_key`        | boolean | No       | Marks this column as part of the primary key (required for JDBC targets)                          |
| `nullable`           | boolean | No       | Whether `null` is accepted. Defaults to `true`. If `false`, a missing value produces an error row |
| `concatenate_fields` | array   | No       | Combine multiple paths and static strings into a single value                                     |
| `hash_method`        | string  | No       | Hash the value before writing: `md5`, `sha256`, `sha512`                                          |
| `encrypt_method`     | string  | No       | Encrypt the value: `aes-256-gcm`                                                                  |
| `encrypt_key`        | string  | No       | Encryption passphrase (required when `encrypt_method` is set)                                     |

***

## Path syntax

Paths navigate the JSON structure of a message using `|` as separator.

```
user|address|city        → message.user.address.city
items|0                  → first element of the items array
```

### Nested objects

```json theme={null}
// Message
{ "order": { "shipping": { "country": "FR" } } }

// Rule
{ "key": "country", "path": "order|shipping|country" }

// Result
{ "country": "FR" }
```

### Static values

When `path` is empty and `static` is set, that literal string is written to every output row.

```json theme={null}
{ "key": "source_system", "static": "salesforce" }
```

***

## Wildcards

Use `*` in a path to **expand an array**. Each element of the array produces a separate output row.

### Basic expansion

```json theme={null}
// Message
{ "tags": ["billing", "premium", "active"] }

// Rule
{ "key": "tag", "path": "tags|*" }

// Result (3 rows)
{ "tag": "billing" }
{ "tag": "premium" }
{ "tag": "active" }
```

### Nested expansion

Wildcards can appear at multiple levels. The mapper walks through every combination and produces one row per deepest element.

```json theme={null}
// Message
{
  "orders": [
    {
      "id": "O-1",
      "items": [
        { "sku": "A", "qty": 2 },
        { "sku": "B", "qty": 1 }
      ]
    },
    {
      "id": "O-2",
      "items": [
        { "sku": "C", "qty": 5 }
      ]
    }
  ]
}

// Rules
[
  { "key": "order_id", "path": "orders|*|id" },
  { "key": "sku",      "path": "orders|*|items|*|sku" },
  { "key": "qty",      "path": "orders|*|items|*|qty" }
]

// Result (3 rows)
{ "order_id": "O-1", "sku": "A", "qty": 2 }
{ "order_id": "O-1", "sku": "B", "qty": 1 }
{ "order_id": "O-2", "sku": "C", "qty": 5 }
```

### Parallel lists (zip semantics)

When two columns share the **same root path up to the first `*`**, the mapper treats them as aligned lists and zips them by index — no cartesian product.

```json theme={null}
// Message
{
  "names":  ["Alice", "Bob", "Carol"],
  "scores": [95,      82,    78]
}

// Rules — both share root "names" / "scores" at the same level
[
  { "key": "name",  "path": "names|*" },
  { "key": "score", "path": "scores|*" }
]

// Result (3 rows, zipped)
{ "name": "Alice", "score": 95 }
{ "name": "Bob",   "score": 82 }
{ "name": "Carol", "score": 78 }
```

This also works with deeper paths under different roots:

```json theme={null}
// Message
{
  "orders":    [{ "line": { "sku": "SKU-1" } }, { "line": { "sku": "SKU-2" } }],
  "shipments": [{ "route": { "carrier": "DHL" } }, { "route": { "carrier": "UPS" } }]
}

// Rules
[
  { "key": "sku",     "path": "orders|*|line|sku" },
  { "key": "carrier", "path": "shipments|*|route|carrier" }
]

// Result (2 rows, index-aligned)
{ "sku": "SKU-1", "carrier": "DHL" }
{ "sku": "SKU-2", "carrier": "UPS" }
```

### Wildcard on a non-array field

If the path segment marked `*` points to an object (not an array), the wildcard is silently skipped and the mapper continues navigating the object normally.

### Deduplication

When wildcard expansion produces identical rows, duplicates are automatically removed.

***

## Transformations

Transformations are applied in this order: **concatenate → hash → encrypt → cast**.

### Type casting

```json theme={null}
{ "key": "amount",     "path": "raw_amount",  "cast": "float" }
{ "key": "is_active",  "path": "active_flag", "cast": "bool" }
{ "key": "created_on", "path": "ts",          "cast": "date", "cast_format": "%Y-%m-%d" }
```

Boolean casting recognises `"true"`, `"false"`, `"1"`, `"0"`, `"yes"`, `"no"` (case-insensitive) in addition to native booleans.

For `date` / `datetime` / `time`, use a `cast_format` string in [strftime format](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes). When omitted, ISO 8601 strings are parsed automatically.

### Concatenation

Combine multiple source fields and/or static separators into a single output column.

```json theme={null}
{
  "key": "full_address",
  "concatenate_fields": [
    { "path": "street_number" },
    { "static": " " },
    { "path": "street_name" },
    { "static": ", " },
    { "path": "city" }
  ]
}
// Result: "123 Main St, New York"
```

Each entry in `concatenate_fields` has a `path` and/or a `static` value. They are appended in order.

### Hashing

Hash a value before it lands in the target — useful for pseudonymisation.

```json theme={null}
{ "key": "email_hash", "path": "email", "hash_method": "sha256" }
```

Supported algorithms: `md5`, `sha256`, `sha512`. A `null` input produces a `null` output (no error).

### Encryption

Encrypt a value with AES-256-GCM. A unique random nonce is generated per call, so the same input produces a different ciphertext each time.

```json theme={null}
{
  "key": "ssn_encrypted",
  "path": "ssn",
  "encrypt_method": "aes-256-gcm",
  "encrypt_key": "my-secret-passphrase"
}
```

The encryption key is derived with PBKDF2. Store it somewhere safe — there is no way to decrypt without it.

***

## Null handling

By default, every column is **nullable**: if the path does not exist in the message the output value is `null` and no error is raised.

Set `"nullable": false` to treat a missing field as an error:

```json theme={null}
{ "key": "user_id", "path": "id", "nullable": false, "primary_key": true }
```

When a non-nullable column cannot be resolved, the mapper adds an `"error"` key to the output row describing the failure.

***

## Primary keys

Targets that write to a database table (PostgreSQL, Snowflake, Oracle, …) require at least one column marked `"primary_key": true`. This is what the connector uses to perform upserts.

```json theme={null}
{ "key": "order_id", "path": "id",      "primary_key": true }
{ "key": "item_seq", "path": "seq_num", "primary_key": true }
```

***

## Testing and debugging

### In the UI

The subscription builder includes a **live preview** panel. Paste a raw CDC message and the panel shows the exact rows the mapper will produce before you save the subscription.

### Via the API

You can call the mapper directly without any pipeline or subscription:

```http theme={null}
POST /smt/process_mapper
Content-Type: application/json

{
  "table_name": "orders",
  "config": [
    { "key": "id",     "path": "order_id", "primary_key": true },
    { "key": "amount", "path": "total",    "cast": "float" },
    { "key": "tag",    "path": "tags|*" }
  ],
  "message": {
    "order_id": "abc123",
    "total": "49.99",
    "tags": ["new", "vip"]
  }
}
```

Response:

```json theme={null}
[
  { "id": "abc123", "amount": 49.99, "tag": "new" },
  { "id": "abc123", "amount": 49.99, "tag": "vip" }
]
```

This endpoint is available on any Popsink instance and requires no special permissions. It is the fastest way to iterate on a mapper configuration.

### Reading error rows

When a transformation fails (bad cast, non-nullable field missing, etc.) the mapper does not drop the row — it includes it with an `"error"` key:

```json theme={null}
{ "id": null, "amount": null, "error": "Field 'order_id' is required but was not found in message" }
```

Check for this key in the preview output to diagnose misconfigured paths or type mismatches.

### Common mistakes

| Symptom                                 | Likely cause                                                                                                                            |
| --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Output has fewer rows than expected     | A wildcard path points to an object instead of an array — check the shape of your message                                               |
| Duplicate rows in output                | Multiple wildcards at the same level produce a cartesian product instead of zipping — verify that both paths share the same root prefix |
| `null` where a value was expected       | The path separator is wrong — use `\|` not `.` to separate path segments                                                                |
| Cast failure                            | The source value cannot be converted (e.g. `"N/A"` cast to `int`) — add `"nullable": true` or fix the source value                      |
| JDBC connector rejects the subscription | No column has `"primary_key": true` — required for upsert-capable targets                                                               |