Skip to main content
The Postgres Source Connector is a powerful and efficient solution designed to facilitate seamless data integration between your PostgreSQL databases and Popsink. This connector helps you synchronize with PostgreSQL in real-time, ensuring that your target systems remain updated with the latest information in real-time, driving business intelligence, analytics, and operational efficiency.

​
Compatible versions

PostgreSQL 13 and earlier are outside the tested matrix and are not supported. On managed services, use the provider’s own replication role rather than a superuser.

​
Prerequisites

  • Your Postgres should be in logical mode (use SHOW wal_level). On Amazon RDS and Aurora, see Amazon RDS and Aurora PostgreSQL below.
  • A user with the REPLICATION attribute:
The REPLICATION attribute is not inherited through role membership: granting the user a role that has REPLICATION is not enough, the attribute must be set on the user itself. On Amazon RDS and Aurora, use rds_replication instead (see below).
  • The user should be able to list and read the tables you wish to replicate, for the initial load:
  • Are you using PostgreSQL on Heroku? Here’s a great guide to migrating.
  • Each table should have its REPLICA IDENTITY set to FULL
  • You have to create publications for your tables

​
Amazon RDS and Aurora PostgreSQL

On RDS and Aurora, wal_level and the REPLICATION attribute cannot be set directly:
  • Enable logical replication by setting rds.logical_replication = 1 in the DB (or DB cluster, for Aurora) parameter group, then reboot the instance. SHOW wal_level should then return logical.
  • Grant replication rights through the rds_replication role:
  • Connect Popsink to the primary instance. On Aurora, use the cluster (writer) endpoint: reader instances cannot host logical replication slots. On RDS, logical decoding on a read replica is possible from PostgreSQL 16, but the publication must be created on the primary and a replication slot on a replica does not survive a failover.

​
Key Features

  • Real-time Change Data Capture (CDC): The Postgres Source Connector employs a CDC mechanism using the native PostgreSQL logical decoding feature, capturing and streaming changes (inserts, updates, and deletes) as they occur in your database.
  • Schema Evolution and Compatibility: This connector supports schema evolution, automatically detecting changes in the PostgreSQL database schema and adapting the data stream accordingly.
  • Fault-tolerant and Scalable: The Postgres Source Connector is built with fault tolerance and scalability in mind. It is capable of resuming data replication from the last known offset in case of failures, ensuring data consistency and minimal downtime.
  • Initial Load: The connector automatically performs an initial full table load.
  • Advanced Filtering: The Postgres Source Connector provides a range of filtering options, including table and schema filters, allowing you to selectively replicate specific tables and schemas based on your needs.

​
Configuration

​
Advanced Options

​
Connecting through an SSH Tunnel

If the database is only reachable through a bastion host, enable SSH Tunnel and provide: The Host and Port above stay as the database is addressed from the bastion. For key generation, bastion hardening and the other connectivity options (IPsec VPN, Tailscale), see Tunnels and private connectivity.

​
Table Selection

After a successful credential check, pick the tables to replicate from the discovered list. The selection is stored as a comma-separated whitelist in schema.table format.

​
Security

The connector supports SSL/TLS encryption for secure communication between the connector and your PostgreSQL database.