Unity Catalog (Delta Lake) Target

The Unity Catalog Target Connector writes your pipeline data to Delta Lake tables stored in Azure Blob Storage and registers them in a Databricks Unity Catalog catalog and schema. Data is written in Delta format with full ACID compliance and schema evolution, and tables are queryable from any Databricks workspace once registered. The connector is delivered as a native worker — no Spark cluster and no SQL warehouse are required on your side — streaming change events directly into your lakehouse. Because the worker writes Delta files directly to Azure Blob Storage (outside Databricks compute) and registers them as external tables, two requirements are easy to miss: the metastore must allow External Data Access, and the Service Principal needs the external-table privileges. Both are detailed below.

Key Features
Prerequisites
Storage Authentication
Databricks Authentication
External Data Access
Catalog and Schema

Key Features

Delta Lake format: data is written as Delta tables with ACID transactions and schema evolution.
Unity Catalog registration: each subscription’s target table is automatically created or updated under the catalog and schema you provide, making it instantly queryable from Databricks.
Native streaming worker: change events are streamed directly into Azure Blob Storage without a Spark job or SQL API in the write path.
Secure by design: Databricks access uses OAuth Machine-to-Machine with a Service Principal — no Personal Access Tokens.

Prerequisites

An Azure Blob Storage account with an existing container for the Delta data.
A Databricks workspace with Unity Catalog enabled.
An existing Unity Catalog catalog and schema — the connector registers tables but does not create the catalog or schema.
External Data Access enabled on the metastore (per-metastore, admin-only). See External Data Access.
A Databricks Service Principal with an OAuth secret and the required privileges on the target catalog and schema. Because the connector registers external tables, those are USE CATALOG, USE SCHEMA, CREATE EXTERNAL TABLE and EXTERNAL USE SCHEMA.

Storage Authentication

Two authentication methods are supported for the underlying Azure Blob Storage. Pick the one that matches how you administer the storage account.

Option A — Connection String

Authenticate with an Azure Blob Storage connection string, found in the Azure Portal under Storage Account → Access keys.

Field	Description
Connection String	e.g. `DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=...;EndpointSuffix=core.windows.net`
Container Name	Existing container where Delta table data will be written (e.g. `my-lakehouse`)

Option B — Service Principal (SPN)

Authenticate with an Azure Active Directory Service Principal. The Service Principal must have the Storage Blob Data Contributor role on the storage account.

Field	Description
Storage Account Name	Name of the Azure Storage account (e.g. `mystorageaccount`)
Tenant ID	Azure AD tenant ID (Azure Portal → Microsoft Entra ID → Overview)
Client ID	Application (client) ID of the registered app
Client Secret	A client secret generated for the registered app
Container Name	Existing container where Delta table data will be written

Databricks Authentication

The connector authenticates to Databricks via OAuth Machine-to-Machine (M2M) using a Databricks Service Principal: it exchanges the Service Principal’s client ID and secret for a short-lived access token at the workspace token endpoint, then registers each target table under your catalog and schema.

Personal Access Tokens (PATs) are not supported by this connector. Databricks deprecates PATs for new integrations — use a Service Principal with an OAuth secret.

Setting up the Service Principal

In your Databricks workspace, open Settings → Identity and access → Service principals and create a new Service Principal (or pick an existing one).
Under the Service Principal, generate an OAuth secret. Copy the displayed client ID and secret — the secret is shown only once.
Grant the Service Principal the Unity Catalog privileges needed to register external tables on the target catalog and schema:

GRANT USE CATALOG ON CATALOG <catalog_name> TO `<service_principal>`;
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal>`;
GRANT CREATE EXTERNAL TABLE ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal>`;
GRANT EXTERNAL USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal>`;

USE CATALOG / USE SCHEMA make the catalog and schema visible; CREATE EXTERNAL TABLE allows registering externally-written Delta files; EXTERNAL USE SCHEMA lets a non-Databricks engine operate on the schema’s tables. Popsink’s credential check verifies these at configuration time when the Service Principal can read its own effective permissions.

Field	Description
Workspace URL	Base URL of your Databricks workspace (e.g. `https://adb-1234567890123456.7.azuredatabricks.net`)
Databricks Client ID	The Service Principal’s application ID (UUID)
Databricks Client Secret	The OAuth secret generated for the Service Principal

External Data Access

The worker writes Delta files directly to your storage container and registers them as external tables via the Unity Catalog REST API. Because the write happens outside Databricks compute, the metastore must have the External Data Access flag enabled. It is a per-metastore setting that only a metastore admin can change. When it is disabled, table registration fails with a 403 PERMISSION_DENIED (EXTERNAL_ACCESS_DISABLED_ON_METASTORE). Enable it one of two ways:

UI: Catalog → Metastore details → External data access → enable.
API:

PATCH /api/2.1/unity-catalog/metastores/{metastore_id}
{ "external_access_enabled": true }

Popsink’s credential check fails fast at configuration time if External Data Access is disabled or the external-table grants are missing, so you can fix these before the connector goes live rather than discovering them at runtime.

Catalog and Schema

Field	Description
Catalog Name	Unity Catalog catalog where tables will be registered (e.g. `main`)
Schema Name	Schema (database) under that catalog (e.g. `bronze`)

Both must exist before the connector starts. Tables are created or updated using each subscription’s target table name within this catalog and schema.

Welcome

Platform Overview

Deployment

Source Connectors

Target Connectors

On-Premises & Integrations

Transformation

Changelog

Table of Contents

Key Features

Prerequisites

Storage Authentication

Option A — Connection String

Option B — Service Principal (SPN)

Databricks Authentication

Setting up the Service Principal

External Data Access

Catalog and Schema

​Table of Contents

​Key Features

​Prerequisites

​Storage Authentication

​Option A — Connection String

​Option B — Service Principal (SPN)

​Databricks Authentication

​Setting up the Service Principal

​External Data Access

​Catalog and Schema

Table of Contents

Key Features

Prerequisites

Storage Authentication

Option A — Connection String

Option B — Service Principal (SPN)

Databricks Authentication

Setting up the Service Principal

External Data Access

Catalog and Schema