> ## Documentation Index
> Fetch the complete documentation index at: https://docs.popsink.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Kafka Source

The Kafka Source Connector integrates with Apache Kafka, a distributed, fault-tolerant, and highly-scalable event streaming platform designed for handling real-time data streams. It enables the efficient publishing, storing, and processing of large volumes of events, making it ideal for building data pipelines, messaging systems, and real-time analytics applications. Kafka is widely used across various industries for its high throughput, low latency, and strong durability guarantees.

## Compatible versions

| | |
| - | - |
| **Apache Kafka** | 2.1 and later, including 3.x and 4.x |
| **Compatible platforms** | Confluent Platform & Confluent Cloud, Amazon MSK, Aiven for Apache Kafka, Redpanda, Azure Event Hubs (Kafka endpoint, Standard tier or higher) |
| **Protocol** | Kafka wire protocol over `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, or `SASL_SSL` |

<Note>Brokers older than 2.1 are not supported — the client negotiates API versions with the cluster and no longer implements the pre-2.1 protocol.</Note>

## Key Features

* **Continuous consumption:** the connector subscribes to your Kafka topics and streams events into Popsink as they are produced.
* **Broad authentication support:** PLAINTEXT, SSL, and SASL (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER, GSSAPI), including mutual TLS.
* **Message metadata:** optionally include Kafka message metadata (topic, partition, offset, timestamp) alongside the payload.

## Prerequisites

* A reachable Kafka cluster.
* Credentials with permission to **read** from the source topics (and to use the configured consumer group).

## Configuration

| Field | Required | Description |
| - | - | - |
| **Bootstrap Servers** | Yes | Comma-separated `host:port` pairs (e.g. `kafka1.example.com:9092,kafka2.example.com:9092`) |
| **Security Protocol** | Yes | `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, or `SASL_SSL` |
| **SASL Mechanism** | With SASL | `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBEARER`, or `GSSAPI` |
| **SASL Username / Password** | With SASL | Credentials for the chosen SASL mechanism |
| **Topic** | Yes | Kafka topic to consume from |
| **Consumer Group ID** | No | Consumer group used to track offsets; auto-generated when omitted |
| **Include Metadata** | No | Include Kafka message metadata (topic, partition, offset, timestamp) in the records (default: `false`) |

## TLS and Mutual TLS

When connecting over `SSL` or `SASL_SSL`, you can additionally provide certificates (PEM or base64-encoded PEM):

| Field | Description |
| - | - |
| **CA Certificate** | CA certificate used to verify the brokers |
| **Client Certificate** | Client certificate for mutual TLS |
| **Client Private Key** | Client private key for mutual TLS |

## Security Protocols

| Protocol | Encryption | Authentication |
| - | - | - |
| `PLAINTEXT` | None | None — development only |
| `SSL` | TLS | TLS client certificates (optional) |
| `SASL_PLAINTEXT` | None | SASL |
| `SASL_SSL` | TLS | SASL — recommended for production |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.