The Unity Catalog Target Connector writes your pipeline data to Delta Lake tables stored in Azure Blob Storage and registers them in a Databricks Unity Catalog catalog and schema. Data is written in Delta format with full ACID compliance and schema evolution, and tables are queryable from any Databricks workspace once registered. The connector is delivered as a native worker — no Spark cluster and no SQL warehouse are required on your side — streaming change events directly into your lakehouse.
Because the worker writes Delta files directly to Azure Blob Storage (outside Databricks compute) and registers them as external tables, two requirements are easy to miss: the metastore must allow External Data Access, and the Service Principal needs the external-table privileges. Both are detailed below.
Table of Contents
- Key Features
- Prerequisites
- Storage Authentication
- Databricks Authentication
- External Data Access
- Catalog and Schema
Key Features
- Delta Lake format: data is written as Delta tables with ACID transactions and schema evolution.
- Unity Catalog registration: each subscription’s target table is automatically created or updated under the catalog and schema you provide, making it instantly queryable from Databricks.
- Native streaming worker: change events are streamed directly into Azure Blob Storage without a Spark job or SQL API in the write path.
- Secure by design: Databricks access uses OAuth Machine-to-Machine with a Service Principal — no Personal Access Tokens.
Prerequisites
- An Azure Blob Storage account with an existing container for the Delta data.
- A Databricks workspace with Unity Catalog enabled.
- An existing Unity Catalog catalog and schema — the connector registers tables but does not create the catalog or schema.
- External Data Access enabled on the metastore (per-metastore, admin-only). See External Data Access.
- A Databricks Service Principal with an OAuth secret and the required privileges on the target catalog and schema. Because the connector registers external tables, those are
USE CATALOG, USE SCHEMA, CREATE EXTERNAL TABLE and EXTERNAL USE SCHEMA.
Storage Authentication
Two authentication methods are supported for the underlying Azure Blob Storage. Pick the one that matches how you administer the storage account.
Option A — Connection String
Authenticate with an Azure Blob Storage connection string, found in the Azure Portal under Storage Account → Access keys.
| Field | Description |
|---|
| Connection String | e.g. DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=...;EndpointSuffix=core.windows.net |
| Container Name | Existing container where Delta table data will be written (e.g. my-lakehouse) |
Option B — Service Principal (SPN)
Authenticate with an Azure Active Directory Service Principal. The Service Principal must have the Storage Blob Data Contributor role on the storage account.
| Field | Description |
|---|
| Storage Account Name | Name of the Azure Storage account (e.g. mystorageaccount) |
| Tenant ID | Azure AD tenant ID (Azure Portal → Microsoft Entra ID → Overview) |
| Client ID | Application (client) ID of the registered app |
| Client Secret | A client secret generated for the registered app |
| Container Name | Existing container where Delta table data will be written |
Databricks Authentication
The connector authenticates to Databricks via OAuth Machine-to-Machine (M2M) using a Databricks Service Principal: it exchanges the Service Principal’s client ID and secret for a short-lived access token at the workspace token endpoint, then registers each target table under your catalog and schema.
Personal Access Tokens (PATs) are not supported by this connector. Databricks deprecates PATs for new integrations — use a Service Principal with an OAuth secret.
Setting up the Service Principal
- In your Databricks workspace, open Settings → Identity and access → Service principals and create a new Service Principal (or pick an existing one).
- Under the Service Principal, generate an OAuth secret. Copy the displayed client ID and secret — the secret is shown only once.
- Grant the Service Principal the Unity Catalog privileges needed to register external tables on the target catalog and schema:
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<service_principal>`;
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal>`;
GRANT CREATE EXTERNAL TABLE ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal>`;
GRANT EXTERNAL USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal>`;
USE CATALOG / USE SCHEMA make the catalog and schema visible; CREATE EXTERNAL TABLE allows registering externally-written Delta files; EXTERNAL USE SCHEMA lets a non-Databricks engine operate on the schema’s tables. Popsink’s credential check verifies these at configuration time when the Service Principal can read its own effective permissions.
| Field | Description |
|---|
| Workspace URL | Base URL of your Databricks workspace (e.g. https://adb-1234567890123456.7.azuredatabricks.net) |
| Databricks Client ID | The Service Principal’s application ID (UUID) |
| Databricks Client Secret | The OAuth secret generated for the Service Principal |
External Data Access
The worker writes Delta files directly to your storage container and registers them as external tables via the Unity Catalog REST API. Because the write happens outside Databricks compute, the metastore must have the External Data Access flag enabled. It is a per-metastore setting that only a metastore admin can change. When it is disabled, table registration fails with a 403 PERMISSION_DENIED (EXTERNAL_ACCESS_DISABLED_ON_METASTORE).
Enable it one of two ways:
- UI: Catalog → Metastore details → External data access → enable.
- API:
PATCH /api/2.1/unity-catalog/metastores/{metastore_id}
{ "external_access_enabled": true }
Popsink’s credential check fails fast at configuration time if External Data Access is disabled or the external-table grants are missing, so you can fix these before the connector goes live rather than discovering them at runtime.
Catalog and Schema
| Field | Description |
|---|
| Catalog Name | Unity Catalog catalog where tables will be registered (e.g. main) |
| Schema Name | Schema (database) under that catalog (e.g. bronze) |
