Skip to main content
The IBM Z Source Connector captures changes (INSERT, UPDATE, DELETE) from Db2 for z/OS by reading the Db2 log. It uses a split architecture, so that almost all processing runs off the mainframe:
  • Stage 1 — zAgent, a small, stateless started task on z/OS that reads the Db2 log and streams it out unchanged.
  • Stage 2 — processing server, which runs off-host on Linux, stores the stream durably and publishes it to Kafka.

Compatible versions

This connector runs an agent inside your mainframe environment, so supported z/OS and Db2 for z/OS levels, including the required Db2 maintenance (APARs), are validated with you during onboarding. Get in touch with your release levels before planning a deployment.

Overview

Design principles

  • Stage 1 does as little as possible. It never decodes, filters or transforms data. This keeps it small, easy to audit and cheap in MIPS.
  • The Db2 log is the buffer. If Stage 2 is unavailable, Stage 1 stops reading and waits. Nothing is spooled on z/OS, and nothing is lost as long as Db2 keeps the log (see Archive log retention).
  • Stage 2 owns the resume point. At every (re)connection, Stage 2 tells Stage 1 where to resume. Restarting the agent is always safe.
  • No change is acknowledged until it is on disk. Stage 2 acknowledges a batch only after it is written and fsync’d to its write-ahead log.

The z/OS agent (Stage 1)

What it is

  • A single-threaded C program (Language Environment), delivered as a load module in a PDSE.
  • The load library must be APF-authorized. Db2 requires IFCID 306 calls to be made in supervisor state and key 0, with the return area in common storage. A small assembler routine switches to that state for the duration of each IFI call only; the rest of the program runs in normal problem state.
  • It runs as a started task under a dedicated user ID (called CDCAGENT on this page).
  • You run one agent per Db2 subsystem, or one agent per data-sharing group. In data sharing, the agent connects to one member and Db2 returns the log records of all members merged in LRSN order.
  • It connects to Db2 with the Call Attachment Facility (CAF, DSNALI) and reads the log with the Instrumentation Facility Interface (DSNWLI, READS request for IFCID 306) — the standard Db2 interface for log-based replication.

What it reads and writes

The agent never writes to Db2 and never stores change data on z/OS.

Footprint

  • CPU: read the log, wrap records in batches, send. No decoding, conversion or compression on z/OS.
  • Memory: a few in-flight batches (window × batch_max_kb, see Configuration).
  • Disk: none apart from SYSOUT.
  • Common storage: one IFI return area of 200 KB to 1 MB in ECSA (key 7).
  • Db2: one CAF thread, and monitor trace class 1 if your Db2 level requires it.

Prerequisites: what to request from your teams

Steps 1 to 5 are required before the first start. Each step can be sent as is to the team concerned.

Step 1. Db2 DBA: privileges and table settings

Impact of DATA CAPTURE CHANGES: it increases log volume for updates on the tables concerned, because Db2 logs the full before-image of the row instead of only the changed bytes. Inserts and deletes are not affected. Plan active and archive log capacity accordingly. It also blocks some ALTER operations while it is on, as documented by IBM.

Step 2. RACF administrator: identity and access

Key ring for AT-TLS — the Stage 1 client certificate, plus the CA that signed the Stage 2 server certificate:
The user also needs READ on IRR.DIGTCERT.LISTRING in the FACILITY class (or an RDATALIB profile) to read its own key ring. What CDCAGENT cannot do: run SQL, change any data, read data sets other than its own load library and configuration, start or stop Db2, or accept inbound connections. Authorized code: the agent is APF-authorized because Db2 requires it for IFCID 306. Supervisor state is used only inside the IFI call routine and to obtain and free the return area. Popsink can provide this routine’s design and source for review by your security team.

Step 3. System programmer: installation

Step 4. DBA and storage: archive log retention

This is the key availability requirement. When Stage 2 is unreachable, the agent stops reading and later resumes from where Stage 2 left off. The log records it still needs must still be available in Db2: on the active log, or on archive logs that Db2 can read without operator action.
Rule: archive log retention ≥ longest planned Stage 2 outage + safety margin. Recommended minimum: 72 hours, so that a weekend outage is covered.
If the resume point is no longer available in Db2, the agent stops with an explicit error. Capture then has to be re-initialized from a new position, with a re-snapshot of the affected tables.

Step 5. Network team

  • Stage 1 is always the TCP client. No inbound connection to z/OS is needed: the agent opens no listening port, and Stage 2 never connects to the mainframe.
  • Any firewall, NAT or load balancer between the two must pass TCP straight through — no TLS termination, no TLS inspection, no HTTP proxy — so that mutual TLS works end to end.

Ready-to-send summary for the DBA

Subject: Db2 prerequisites for Popsink zAgent (CDC) on <SSID> We are installing Popsink zAgent, a log-based change data capture agent for Db2 for z/OS. It reads the Db2 log through the standard IFI interface (IFCID 306), runs no SQL and does not change any data. We need:
  1. GRANT MONITOR2 TO CDCAGENT; and a plan for the agent’s CAF connection (BIND PLAN(ZAGENT) ..., unless CONNECT alone is sufficient on your Db2 level).
  2. ALTER TABLE … DATA CAPTURE CHANGES; on the attached list of tables, at a time you choose (it increases update log volume on these tables).
  3. Confirmation that archive logs are kept for at least 72 hours and can be read by Db2 without operator intervention, plus the peak log rate in MB/s.
  4. The Db2 version, function level and maintenance level, the SSID, and the data-sharing topology.
  5. Monitor trace class 1 active (-START TRACE(MON) CLASS(1), or MON=YES in DSNZPARM), if your Db2 level requires it for IFI READS.
  6. Which of the attached tables are compressed, and how REORG handles their compression dictionaries.
The agent’s load library is APF-authorized, as Db2 requires for IFCID 306. RACF, system programming (APF, started task) and network requests are sent separately to the relevant teams.

Configuration

The configuration is deliberately small. Partitioning, topics and Kafka settings are configured only in Stage 2.
There are no certificate paths: TLS is set up in the AT-TLS policy. Unknown keys are rejected at startup.

Started task

Procedure (illustrative, member CDCAGENT in SYS1.PROCLIB or equivalent):
POPSINK.ZAGENT.LOAD and every other library in STEPLIB must be APF-authorized. If one of them is not, the step runs unauthorized and the IFI call fails.

Operator commands

Return codes

A lost Stage 2 connection or a failed Db2 attach is not an error exit: the agent retries with an increasing delay.

Startup sequence

1

Read the configuration

2

Connect to Db2

CAF CONNECT / OPEN.
3

Connect to Stage 2

Open the TCP connection. AT-TLS performs the mutual TLS handshake.
4

Handshake

Send HELLO (protocol version, Db2 identity) and receive HELLO_ACK with the resume LRSN.
5

Stream

Position the IFI read at that LRSN and start the read → batch → send loop.

How Stage 1 and Stage 2 communicate

Security: AT-TLS and mutual TLS

  • The connection is encrypted with TLS 1.3 and mutually authenticated: Stage 1 checks the Stage 2 server certificate, and Stage 2 checks the Stage 1 client certificate. Stage 2 only accepts client certificates signed by the configured CA.
  • On z/OS, TLS is provided by AT-TLS (Application Transparent TLS) in the TCP/IP stack, set up by a Policy Agent rule and a RACF key ring. The agent itself contains no cryptographic code and no certificate files. Your existing z/OS security controls (RACF, ICSF, Policy Agent) apply as for any other application.

Protocol

A simple, versioned binary protocol over one TCP connection. Every message is a 4-byte big-endian length, a 1-byte type, and a body.

Delivery guarantees

When Stage 2 is unavailable

A reconnection is the same as a fresh start: HELLO, then HELLO_ACK with the resume point, then streaming. There is no separate recovery mode.

Stage 2

Deployment

Durability

  • Each batch received from Stage 1 is checked, written to the write-ahead log (WAL) and fsync’d, then acknowledged. This is the only step in Stage 1’s critical path; publishing to Kafka happens asynchronously from the WAL.
  • The WAL is the only durable buffer in the pipeline. It absorbs Kafka outages and slow consumers without slowing Db2. Once Kafka has confirmed delivery, the WAL is trimmed past the published position.
  • If the WAL fills up (Kafka down for a long time), Stage 2 stops acknowledging. Stage 1 then stops reading, and the Db2 archive logs take over as the buffer. Total protection = WAL capacity + Db2 archive log retention.

Network flows

Monitoring

The agent has no metrics endpoint by design: all capture metrics are exposed by Stage 2 in Prometheus format.
Recommended alerts: Stage 1 disconnected for more than N minutes; capture lag above a threshold; WAL more than 80% full; capture lag approaching the archive log retention period.

Operations and failure handling

Initial load (a snapshot of the rows that already exist in the tables) is outside log capture and is planned with you during onboarding.

Appendix

Pre-installation checklist

Sample AT-TLS policy

Illustrative Policy Agent configuration for the outbound connection from CDCAGENT to Stage 2:

Glossary

IBM references