Can a pod in the Popsink namespace open a TCP socket to the source host and
port?
Choosing a shape
A rule of thumb: IPsec if the source is a data centre and your network team is
willing, SSH if the source is a single database with a bastion already in front
of it, an overlay if no inbound rule is ever going to be approved. They also
compose — an IPsec tunnel to the data centre plus an SSH tunnel to one stubborn
host inside it is a perfectly normal end state.
Whichever you pick, the tunnel only has to carry the database port. Popsink
needs no inbound path, no reverse tunnel and no callback.
IPsec site-to-site VPN
The classic enterprise answer: a permanent encrypted tunnel between the network the Popsink data plane runs in and the network the source lives in, terminated by a gateway on each side. Once it is up, the source is simply routable and connectors are configured with its private address as if it were local.What to agree on before anything is built
IPsec fails at the first mismatched parameter, and the error messages are famously unhelpful. Fill this table in with your counterpart before touching a gateway.
Keep the traffic selectors as narrow as both sides can live with. A
/32 on the
remote side and a single egress address on ours is a much easier security review
than two /16s.
The routing gotcha
This is the step that goes wrong on almost every first attempt. Connector workers are pods, so their source address is a pod IP from the cluster’s CNI range — not a node IP and not something the remote side has ever heard of. Two consequences:- If you declare the pod CIDR as the local traffic selector, the remote firewall team has to accept a range that means nothing to them, and overlapping RFC 1918 space becomes likely.
- If you declare the node or subnet CIDR instead, traffic must be SNAT’d to that address on the way out, or it will leave with a pod IP that the tunnel’s policy drops.
- AWS
- Azure
- GCP
- Self-managed (strongSwan)
Put the node group in private subnets whose route table sends
0.0.0.0/0 at
a NAT gateway with a fixed Elastic IP, and attach the Site-to-Site VPN to
the same VPC. The remote selector is then the NAT gateway’s EIP or the
private subnet CIDR, and traffic is SNAT’d for you.MTU
ESP encapsulation adds overhead, and a database connection that completes its handshake then hangs on the first large result set is almost always a fragmentation problem. Clamp TCP MSS on the gateway:Verifying
From inside the cluster, before you create the connector:succeeded here means every connector in that namespace will be able to
connect. A timeout means routing or firewall; a connection refused means the
tunnel is fine and the database is not listening where you think it is.
SSH tunnel
The only pattern Popsink implements itself. When the source sits behind a bastion (jump host) that you already expose, the connector opens an SSH session to the bastion and forwards the database port through it — no change to your network, no gateway, no ticket to the network team. It is configured per connector, in the connector form, and is supported by:PostgreSQL
MySQL
Oracle
SQL Server
IBM i (DB2)
Configuration
The Host and Port of the connector itself stay as the database sees
them from the bastion — usually a private address such as
10.90.4.12 or
db.internal, not something resolvable from the cluster.
Preparing the key
Generate a key pair dedicated to Popsink — never reuse an operator’s personal key — and leave it without a passphrase, since no one is present to type one when a connector restarts.connectorConfigEncryptionKey, like every other connector credential.
Hardening the bastion account
The Popsink account needs to forward one port and nothing else. Express that inauthorized_keys rather than trusting it to behave — this account cannot then
open a shell, forward anything but the database, or be used to pivot:
restrictturns everything off — PTY, agent forwarding, X11,~/.ssh/rc, and all port forwarding — and keeps doing so for any option OpenSSH adds later;port-forwardingputs back only forwarding, whichrestricthad just removed. Omitting it is the usual mistake:permitopennarrows forwarding, it does not re-enable it, so the tunnel fails withadministratively prohibited;permitopenthen narrows that forwarding to exactly one destination;frompins the cluster’s egress address (see the NAT gateway discussion above — you need a stable egress IP for this to be usable).
/usr/sbin/nologin) so the key is good for
the forward and nothing else.
Limitations
One connector, one tunnel
One connector, one tunnel
Each connector establishes its own SSH session. Ten connectors behind the
same bastion means ten concurrent sessions from the same account — make sure
MaxSessions and MaxStartups in sshd_config are not set low enough to
start refusing them.Single hop only
Single hop only
The connector forwards through one bastion. A chain of two jump hosts is not
expressible in the form; put a
ProxyJump on the bastion side, or use an
IPsec tunnel instead.Passphrase-protected keys do not work
Passphrase-protected keys do not work
Connector workers restart unattended — on upgrade, on node drain, on
failure. A key that needs a human is a connector that stops replicating at
3 a.m.
Key rotation is a connector edit
Key rotation is a connector edit
Rotating the key pair means updating the connector configuration. Plan for a
short pause on that connector; replication resumes from its last offset, so
nothing is lost.
Overlay network — Tailscale
An overlay sidesteps the network team entirely: both ends dial out to a coordination service and the source becomes privately routable on a virtual network. No inbound firewall rule, no gateway appliance, no static IP, and the source is never published on the public internet. That makes it the practical answer when the source sits in a network you cannot re-architect — a customer’s data centre, an acquisition still on its own infrastructure, anywhere an inbound rule would take a quarter to approve. Tailscale is a WireGuard mesh, and the usable pattern is a subnet router: one host inside the source network advertises the database subnet to the tailnet, and the Popsink cluster joins the same tailnet and accepts those routes.1
Advertise the source subnet
On a small Linux host inside the source network (see
Sizing the subnet router):Approve the advertised route in the Tailscale admin console, or set
autoApprovers in the ACL so it does not need a human.2
Join the cluster to the tailnet
Install the Tailscale Kubernetes operator in the cluster and expose the
source as a
Service that connectors resolve normally. An ephemeral,
tagged auth key is the right credential here — not a personal one, which
disappears when its owner leaves.3
Lock it down with ACLs
The default tailnet policy is allow-all. Replace it so the Popsink tag can
reach the database port and nothing else:
4
Point the connector at the private address
10.90.4.12:1521 — the same address the subnet router sees. Verify with the
netshoot pod from the IPsec section before creating the connector.Tailscale keys expire. A node whose key expires drops off the tailnet and every
connector through it stalls. Use tagged nodes (whose keys do not expire by
default) or put the renewal in your calendar.
Sizing the subnet router
The subnet router only forwards encrypted traffic to the database; it stores nothing. Tailscale publishes no numeric sizing, so start from:
Bandwidth is the real constraint, above all during the initial load, and it
tends to scale with the instance’s CPU class: WireGuard encryption is
CPU-bound, while memory matters little. The host must have IP forwarding
enabled to act as a subnet router. See Tailscale’s
performance best practices
and subnet router guide.
Firewall rules, by pattern
Tailscale recommends allowing
*:443 rather than a list of destinations,
because the set of DERP relays changes. The router also needs its usual system
egress: DNS, time sync, and OS and Tailscale package updates
(pkgs.tailscale.com). Full list in Tailscale’s
firewall ports reference.
When wildcard destinations are not allowed
Some security policies refuse any* destination. The Tailscale rules can be
narrowed:
- TCP 443, filtered by name (layer 7). Allow
*.tailscale.com, or at minimumcontrolplane.tailscale.comandlogin.tailscale.com(coordination) plusderp*.tailscale.com(relays). Every relay in Tailscale’s published map is currently named undertailscale.com. - TCP 443, filtered by address (layer 4). The coordination server uses
fixed ranges,
192.200.0.0/24and2606:B740:49::/48. Relay addresses are published with their hostnames in Tailscale’s relay map; the set changes over time, so an address-based rule has to be kept up to date. - UDP. On a Popsink-managed deployment, direct connections from the subnet
router (source port 41641 by default) go to Popsink’s egress addresses,
listed in Egress IP addresses and
published as JSON: allow your region’s address, or the whole list, instead
of
*:*. STUN (UDP 3478) only targets the relay servers. This is the preferred setup: higher throughput, and data flows directly between the two ends instead of through third-party relays. - No UDP at all. The tunnel still works: without a direct connection, traffic goes through the relays over TCP 443, with lower throughput — which matters most during an initial load. Relayed traffic stays end-to-end encrypted with WireGuard; the relays cannot decrypt it (see Tailscale’s DERP servers).
Troubleshooting
Deployments and environments
One deployment per isolated network — why a tunnel is often the alternative to
a second deployment.