Data Pipeline API

This will be useful for setting up ETL workflow and controlling data pipeline on your side.

API Key

Please see guide to setup your API key.

Data Import

Execute Job

To execute a data import job, you will need to use the following HTTP endpoint:

POST /data_imports/<import_id>/execute.json

If successful, this endpoint will return the job ID of the job created. To get the result, you will need to poll for the job data with:

GET /data_imports/job_result/<job_id>.json

Data Transform

Execute Job

The same to Data Import, you can execute a data transform job with:

POST /data_transforms/<transform_id>/execute.json

Job Logs

You can view information about the running job using this endpoint:

GET jobs/<job_id>/logs.json

Check Last Run of Imports/Transforms

Sometimes you want to run a custom processing job daily at a particular time, but only if the ETL jobs ran successfully. And you need a way to check the status of the ETL jobs.

In that case, you can use the jobs/get_last_jobs.json endpoint.

Parameters:

• source_type: DataImport or DataTransform
• ids: array of your transform/import IDs

Sample Request:

GET /jobs/last_run_jobs.json?source_type=DataTransform&ids[]=123&ids=[]456

Sample Responses:

{
  "123": {
    "id":11106256,
    "status":"running",
    "start_time":"2017-12-12T04:30:31Z",
    "end_time":null,
    "created_at":"2017-12-12T04:30:31Z"
  },
  "456": {
      "id":11106257,
      "status":"success",
      "start_time":"2017-12-12T04:30:31Z",
      "end_time":"2017-12-12T05:30:31Z",
      "created_at":"2017-12-12T04:30:31Z"
  }
}