REST API Source

​

Key Features

• Connect to any REST API: base URL, authentication, pagination, and endpoints are all declared in a single JSON configuration.
• Multiple auth styles: bearer tokens, API keys (header or query), HTTP basic, and OAuth2 client credentials.
• Incremental synchronization: declare a cursor field per resource and only new or updated records are delivered on each run.
• Scheduled ingestion: flexible scheduling by minutes, hours, or days, with an immediate first run on creation.

​

Configuration

​

Configuration structure

• client — how to reach and authenticate against the API: base_url, auth, optional paginator and headers.
• resources — an array of endpoints to ingest. Each resource becomes a stream/table, with its path, a data_selector pointing at the records in the response, optional query params, and an optional incremental cursor.

​

Example

{
  "client": {
    "base_url": "https://api.example.com/v1",
    "auth": {
      "type": "bearer",
      "token": "YOUR_API_TOKEN"
    },
    "paginator": {
      "type": "json_link",
      "next_url_path": "paging.next"
    }
  },
  "resources": [
    {
      "name": "customers",
      "endpoint": {
        "path": "customers",
        "data_selector": "data",
        "params": { "per_page": 100 },
        "incremental": {
          "cursor_path": "updated_at",
          "initial_value": "1970-01-01T00:00:00Z"
        }
      }
    },
    {
      "name": "invoices",
      "endpoint": {
        "path": "invoices",
        "data_selector": "data"
      }
    }
  ]
}

​

Scheduling