Skip to main content
This guide walks through the full lifecycle of a replication over the Popsink REST API: authenticate, pick an environment and a team, create a source and a target connector, subscribe the source’s data models to the target, then start and monitor the workers.
Every Popsink instance serves its own live OpenAPI schema. It is the source of truth for request and response shapes:
  • Swagger UI: https://<your-instance>/api/docs
  • OpenAPI spec: https://<your-instance>/api/openapi.json

Quick start


Concepts


API overview

Base URL

All endpoints live under /api:
Trailing slashes are part of the path: use /connectors/, not /connectors.

Authentication

Every endpoint except login requires a bearer token:

Pagination

List endpoints are paginated with page (starting at 1) and size (default 50, max 100). They return:

Idempotent creation

POST /envs/, POST /teams/ and POST /connectors/ accept an Idempotency-Key header. A retried call with the same key and body returns the original response instead of creating a duplicate:

HTTP status codes


Step 1: Authenticate

You can get a token in two ways.

Option A: email and password

Response (200 OK):
When the access token expires, exchange the refresh token for a new pair:

Option B: organization API key (self-hosted)

Create an API key in the control plane under Dashboard → API keys → Create. It starts with psk_, is shown once, and grants access to a chosen set of environments.
The returned token lasts 10 minutes and comes with no refresh token: exchange the key again when you get a 401.
API-key login only works on deployments running in self-hosted mode. Otherwise it returns 400 This endpoint is only available in SELF_HOSTED mode. Check GET /api/config/deployment → deployment_mode.

Creating a user

POST /auth/register creates a user with an email and a password. It only exists when the deployment allows self-registration (otherwise it returns 404). The server decides the is_active, is_superuser and is_verified flags itself and ignores them in the body.
From now on, every example sends -H "Authorization: Bearer $TOKEN".

Step 2: Pick an environment

List the environments you can reach:
Endpoints that take no env / env_id parameter use your active environment. Set it explicitly:

Creating an environment

An environment needs the Kafka broker that stores its retention (used for replay and lifecycle management):
The response never returns the broker credentials.

Step 3: Create a team

Response (201 Created):
Add users as owners (admins of the team) or members. Users who already belong to the team are skipped:
Response: 204 No Content. List them with GET /teams/$TEAM_ID/members.

Step 4: Check the source credentials

Each connector type has pre-flight endpoints that test a configuration before you save it. They take the same body as the connector’s json_configuration (see Connector configurations):
Then list the tables you can capture:

Step 5: Create the source connector

Response (201 Created):
Popsink creates one data model per entry of the table list, named {connector name}_{entry}. The list is whitelist for database sources and topic for Kafka (topics_config_key tells you which). A CDC source with an empty table list is refused.
List the data models it produced:
To capture more tables later, PATCH the connector with a longer whitelist: the new entries become new data models.

Step 6: Create the target connector

Same endpoint, with a target type:

Step 7: Subscribe the data models to the target

A subscription delivers one data model to one target connector:
Response (201 Created):
To deliver one data model to several targets at once:

Mapper configuration

smt_config is a list of output columns. Each entry:
Test a mapping on a sample message with POST /smt/process_mapper (config, table_name, message) before you save it.

Step 8: Start and monitor

Start the workers

Connectors are created paused. Start the source, then the target:
Both return 202 Accepted before the worker converges. Poll the connector until status is live (or error):
Stop a connector with POST /connectors/{id}/stop: it reads stopping, then paused once the worker is gone.

Enable or disable a single table

You can disable one data model or one subscription without stopping the connector. Popsink restarts the worker automatically to apply the change:

Monitor


Step 9: Update a connector or a subscription

Connector

PATCH /connectors/{id} accepts name, json_configuration and team_id. json_configuration is replaced as a whole, not merged: send the full object, since any key you omit is dropped.
A PATCH never shrinks the table list: entries you omit from whitelist / topic are kept. To stop capturing a table for good, delete its data model (DELETE /datamodels/{id}). connector_type can’t change.

Subscription

All fields of PATCH /subscriptions/{id} are optional. The target worker restarts to apply the change. The column mapping is called mapper_config here:

Step 10: Backfill

Reload existing rows into the destinations. Preview first with POST /backfills/preview, then trigger:
You can scope a backfill with source_connector_ids, datamodel_ids and target_connector_ids. Follow it with GET /backfills/status?source_connector_id=<SOURCE_ID>, and cancel it with POST /backfills/{run_id}/cancel.
"truncate_history": true empties every destination table before reloading it. It is the only option that deletes data in your warehouse.

Connector configurations

The json_configuration of a connector depends on its type. The examples below cover the most common ones. GET /connector-types/ returns every type with its form fields, and the POST /<type>/check-credentials body in the OpenAPI schema documents each field.

Secrets

Any sensitive field (passwords, private keys, tokens, service-account JSON) accepts either an inline value or a reference to a Kubernetes secret that already exists in the worker namespace:
Popsink never stores the referenced value: the worker pod reads it at runtime. We recommend this pattern for API and Terraform consumers. Inline secrets work too, but every read returns them as "<redacted>". GET /k8s-secrets lists the secrets available in the worker namespace. When you re-test an existing connector whose secrets come back as "<redacted>", add ?connector_id=<CONNECTOR_ID> to its check-credentials call: Popsink uses the stored secrets instead of the redacted values.

SSH tunnel

Database sources (PostgreSQL, MySQL, MariaDB, SQL Server, Oracle, IBM i) can connect through an SSH bastion:
ssh_private_key is the base64-encoded private key.

KAFKA_SOURCE Configuration

Kafka source documentation
Pre-flight endpoints: fetch-messages previews messages. Its body takes exactly one of these shapes: {"datamodelId": "..."}, {"connectorId": "...", "source_topic": "user-events"}, or the inline configuration above.

POSTGRES_SOURCE Configuration

PostgreSQL source documentation
Pre-flight: POST /postgres-source/check-credentials, /list-tables, /fetch-schema.

MYSQL_SOURCE Configuration

MySQL source documentation. MARIADB_SOURCE takes the same fields.
Pre-flight: POST /mysql-source/check-credentials, /list-tables, /fetch-schema (/mariadb-source/... for MariaDB).

MSSQL_SOURCE Configuration

SQL Server source documentation
Pre-flight: POST /mssql-source/check-credentials, /list-tables, /fetch-schema. To read the transaction log without CDC tables, use MSSQL_LOG_SOURCE (/mssql-log-source/...).

ORACLE_SOURCE Configuration

Oracle source documentation
Pre-flight: POST /oracle-source/check-credentials, /list-tables, /fetch-schema.

IBMI_SOURCE Configuration

IBM i source documentation
Pre-flight: POST /ibmi-source/check-credentials, /list-tables, /fetch-schema.

MONGO_SOURCE Configuration

MongoDB source documentation
Pre-flight: POST /mongo-source/check-credentials, /list-collections, /fetch-schema (samples a collection to propose its columns).

SNOWFLAKE_TARGET Configuration

Snowflake target documentation
Pre-flight: POST /snowflake-target/check-credentials, /warehouses.

BIGQUERY_TARGET Configuration

BigQuery target documentation
Pre-flight: POST /bigquery-target/check-credentials.

POSTGRES_TARGET Configuration

PostgreSQL target documentation. MSSQL_TARGET takes the same fields (default port 1433, default schema dbo).
Pre-flight: POST /postgres-target/check-credentials (/mssql-target/... for SQL Server).

KAFKA_TARGET Configuration

Kafka target documentation
For a Kafka target, the subscription’s target_table_name is the output topic.

ELASTICSEARCH_TARGET Configuration

Elasticsearch target documentation
Pre-flight: POST /elasticsearch-target/check-credentials.

WEBHOOK_TARGET Configuration

Webhook target documentation
Pre-flight: POST /webhook-target/check-credentials.

Connector types


API reference

The main endpoints, relative to /api. See /api/docs on your instance for the full list.

Authentication

Users

Environments and teams

Connectors

Data models

Subscriptions

Backfills and transformations

Monitoring and health


Troubleshooting

401 Unauthorized

The token is missing or expired. Log in again, refresh it with POST /auth/jwt/refresh, or, with an API key, exchange the key again (its tokens last 10 minutes).

400 This endpoint is only available in SELF_HOSTED mode

API-key login is only available on self-hosted deployments. Use email and password login, or check GET /config/deployment to confirm which instance you are calling.

403 Forbidden

You are not a member of the team that owns the resource, or your role doesn’t allow writes. Ask a team owner to add you with POST /teams/{id}/members/bulk.

404 on /auth/register

Self-registration is disabled on this deployment. Ask an administrator to invite you.

409 Conflict

A connector with the same name already exists. Pick another name, or send an Idempotency-Key header so that retries return the original resource.

422 Validation Error

The body doesn’t match the schema. The response names the offending field:

Credentials come back as "<redacted>"

That’s expected: inline secrets are redacted on every read. Send the full value (or a valueFrom reference) again when you PATCH the connector, since json_configuration is replaced as a whole.

Connector stuck in building

The worker can’t start. Read GET /connectors/{id}/logs, then check the configuration with the type’s check-credentials endpoint.

Complete example script

This script creates a PostgreSQL source and a Snowflake target, subscribes every table to Snowflake and starts both workers:

Support

  • Swagger UI: https://<your-instance>/api/docs
  • OpenAPI spec: https://<your-instance>/api/openapi.json
  • Claude Code skill: drive the API from Claude Code