Postgres Source

​

Prerequisites

• Your Postgres should be in logical mode (use SHOW wal_level).
• A user with REPLICATION rights on the tables you wish to connect to. One way to do this is:

CREATE ROLE replication_role REPLICATION;
GRANT replication_role TO <your_user>;

• Are you using PostgreSQL on Heroku? Here’s a great guide to migrating.
• Each table should have its REPLICA IDENTITY set to FULL

ALTER TABLE mytable REPLICA IDENTITY FULL

• You have to create publications for your tables

CREATE PUBLICATION popsink FOR TABLE <your table list here>;

• The user should have the rights to list tables.

​

Key Features

• Real-time Change Data Capture (CDC): The Postgres Source Connector employs a CDC mechanism using the native PostgreSQL logical decoding feature, capturing and streaming changes (inserts, updates, and deletes) as they occur in your database.
• Schema Evolution and Compatibility: This connector supports schema evolution, automatically detecting changes in the PostgreSQL database schema and adapting the data stream accordingly.
• Fault-tolerant and Scalable: The Postgres Source Connector is built with fault tolerance and scalability in mind. It is capable of resuming data replication from the last known offset in case of failures, ensuring data consistency and minimal downtime.
• Initial Load: The connector automatically performs an initial full table load.
• Advanced Filtering: The Postgres Source Connector provides a range of filtering options, including table and schema filters, allowing you to selectively replicate specific tables and schemas based on your needs.

​

Configuration

​

Advanced Options

​

Connecting through an SSH Tunnel

​

Table Selection

​

Security