Skip to main content
The REST API Source Connector lets you ingest data from any REST API into Popsink pipelines by providing a declarative JSON configuration. It is the generic counterpart to the dedicated API connectors (Zendesk, Shopify, Pipedrive, …): if a SaaS platform doesn’t have its own connector yet, you can usually integrate it in minutes with this one. The configuration follows the dlt rest_api_source schema, describing the API client (base URL, authentication, pagination) and the resources (endpoints) to ingest.

​
Key Features

  • Connect to any REST API: base URL, authentication, pagination, and endpoints are all declared in a single JSON configuration.
  • Multiple auth styles: bearer tokens, API keys (header or query), HTTP basic, and OAuth2 client credentials.
  • Incremental synchronization: declare a cursor field per resource and only new or updated records are delivered on each run.
  • Scheduled ingestion: flexible scheduling by minutes, hours, or days, with an immediate first run on creation.

​
Configuration

FieldRequiredDescription
ConfigurationYesA JSON object following the dlt rest_api_source schema — see below
ScheduleYesSync frequency — see Scheduling

​
Configuration structure

The configuration JSON has two main blocks:
  • client — how to reach and authenticate against the API: base_url, auth, optional paginator and headers.
  • resources — an array of endpoints to ingest. Each resource becomes a stream/table, with its path, a data_selector pointing at the records in the response, optional query params, and an optional incremental cursor.

​
Example

Ingesting two resources from an API authenticated with a bearer token, with cursor-based incremental sync on updated_at: 
{
  "client": {
    "base_url": "https://api.example.com/v1",
    "auth": {
      "type": "bearer",
      "token": "YOUR_API_TOKEN"
    },
    "paginator": {
      "type": "json_link",
      "next_url_path": "paging.next"
    }
  },
  "resources": [
    {
      "name": "customers",
      "endpoint": {
        "path": "customers",
        "data_selector": "data",
        "params": { "per_page": 100 },
        "incremental": {
          "cursor_path": "updated_at",
          "initial_value": "1970-01-01T00:00:00Z"
        }
      }
    },
    {
      "name": "invoices",
      "endpoint": {
        "path": "invoices",
        "data_selector": "data"
      }
    }
  ]
}
Refer to the dlt REST API source documentation for the full schema, including all supported authentication types and paginators.

​
Scheduling

API source connectors run on a schedule rather than streaming continuously. A first run is always triggered immediately after the connector is created; subsequent runs depend on the selected mode:
ModeBehaviorExample
daysRuns every N days at 00:00 UTC + offset (in minutes)Offset 30 → every day at 00:30 UTC
hoursRuns every N hours at :00 + offset (in minutes)Offset 15 → 00:15, 01:15, 02:15, …
minutesRuns every N minutes, anchored on the connector creation timeCreated 10:03, interval 10 → 10:13, 10:23, …