Datasource API Overview

API reference for Data Sources

Description

The APIs below can be used for Data Sources operations such as

  1. Create a Data Source
  2. Delete a Data Source
  3. Change or update Data Source details
  4. Run configuration checks
  5. Setup and run metadata extraction

Data Source API's is currently certified for mysql, oracle, postgres, sqlserver, redshift, teradata, and snowflake.

NOTE: Make sure that to input these data sources in lower case for the JSON body.

Open API 3.0 Specification

Datasource APIs are described using the Open API 3.0 Specification (OAS). OAS is a broadly adopted industry standard for describing APIs.

To see the specification, replace {AlationInstanceURL} below with your Alation instance's URL and visit the link:

{AlationInstanceURL}/openapi/datasources/

NOTE

  1. These APIs are available in Alation version 2020.4 and later.
  2. The Swagger UI is not enabled by default on an Alation instance. Please set the flag alation.feature_flags.enable_swagger to True using alation_conf.

Data Source Retrieval (Bulk)

This API can be used to retrieve multiple existing data sources in bulk.

URL

GET /integration/v1/datasource/?**<params>**

Replace <params> with your list of parameters.

URL Parameters

NameRequiredDescription
include_undeployedNoSpecifies if undeployed1 data sources should be included in retrieved list. (By default, it is false)
Example: /integration/v1/datasource/?include_undeployed=true
include_hiddenNoSpecifies if hidden2 data sources should be included in retrieved list. Only supported for users with Server Admin role. (By default, it is false)
Example: /integration/v1/datasource/?include_hidden=true
  1. Undeployed data sources are the ones whose configurations and deployment setup is not complete. Admins cannot trigger metadata extraction on undeployed data sources.

  2. Hidden data sources are not visible in the UI. These are the data sources created via the API with the is_hidden property set to True. There is no UI for the Settings page of such sources and they can only be accessed through the API.

Headers

HTTP HeaderValue
TOKEN<your_token>

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

[
  {
    "host": "10.11.21.125",
    "port": 3306,
    "deployment_setup_complete": true,
    "db_username": "sample_username",
    "dbname": "sample_dbname",
    "supports_explain": true,
    "has_aws_glue_metastore": false,
    "supports_qli_daterange": true,
    "icon": "db",
    "qli_aws_access_key_id": "",
    "private": false,
    "data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
    "obfuscate_literals": null,
    "is_virtual": false,
    "metastore_type": 0,
    "latest_extraction_successful": false,
    "is_hidden": false,
    "id": 372,
    "owner_ids": [
        1
    ],
    "exclude_additional_columns_in_qli": false,
    "disable_auto_extraction": true,
    "hive_logs_source_type": 0,
    "metastore_uri": null,
    "qualified_name": "",
    "is_hive": false,
    "title": "sample_title",
    "dbtype": "mysql",
    "supports_default_schema_extraction": false,
    "supports_profiling_v2": false,
    "qli_aws_region": "",
    "is_gone": false,
    "favorited_by_list": null,
    "webhdfs_server": null,
    "enabled_in_compose": true,
    "supports_profiling": true,
    "supports_qli_diagnostics": true,
    "is_presto_hive": false,
    "nosql_mde_sample_size": 100,
    "qli_hive_connection_source": 0,
    "cron_extraction": "20 2 * * 1",
    "negative_filter_words": null,
    "supports_compose": true,
    "hive_logs_source": null,
    "has_hdfs_based_qli": false,
    "has_aws_s3_based_qli": false,
    "can_data_upload": true,
    "description": "<p>sample_description</p>",
    "enable_designated_credential": false,
    "all_schemas": null,
    "deleted": false,
    "supports_md_diagnostics": true,
    "aws_region": null,
    "limit_schemas": null,
    "has_previewable_qli": false,
    "hive_tez_logs_source": null,
    "remove_filtered_schemas": false,
    "profiling_tip": null,
    "aws_access_key_id": null,
    "has_metastore_uri": false,
    "webhdfs_username": "sample_username",
    "webhdfs_port": 50070,
    "latest_extraction_time": null,
    "uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
    "url": "/data/372/",
    "enable_default_schema_extraction": false,
    "jdbc_driver": null,
    "unresolved_mention_fingerprint_method": 0,
    "builtin_datasource": null,
    "otype": "data",
    "can_toggle_ds_privacy": true,
    "exclude_schemas": null,
    "compose_oauth_enabled": false
  }
]

Data Source Retrieval (Point)

This API can be used to retrieve a specific data source.

URL

GET /integration/v1/datasource/**<ds_id>**/

Replace <ds_id> with your data source id.

Headers

HTTP HeaderValue
TOKEN<your_token>

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

  {
    "host": "10.11.21.125",
    "port": 3306,
    "deployment_setup_complete": true,
    "db_username": "sample_username",
    "dbname": "sample_dbname",
    "supports_explain": true,
    "has_aws_glue_metastore": false,
    "supports_qli_daterange": true,
    "icon": "db",
    "qli_aws_access_key_id": "",
    "private": false,
    "data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
    "obfuscate_literals": null,
    "is_virtual": false,
    "metastore_type": 0,
    "latest_extraction_successful": false,
    "is_hidden": false,
    "id": 372,
    "owner_ids": [
        1
    ],
    "exclude_additional_columns_in_qli": false,
    "disable_auto_extraction": true,
    "hive_logs_source_type": 0,
    "metastore_uri": null,
    "qualified_name": "",
    "is_hive": false,
    "title": "sample_title",
    "dbtype": "mysql",
    "supports_default_schema_extraction": false,
    "supports_profiling_v2": false,
    "qli_aws_region": "",
    "is_gone": false,
    "favorited_by_list": null,
    "webhdfs_server": null,
    "enabled_in_compose": true,
    "supports_profiling": true,
    "supports_qli_diagnostics": true,
    "is_presto_hive": false,
    "nosql_mde_sample_size": 100,
    "qli_hive_connection_source": 0,
    "cron_extraction": "20 2 * * 1",
    "negative_filter_words": null,
    "supports_compose": true,
    "hive_logs_source": null,
    "has_hdfs_based_qli": false,
    "has_aws_s3_based_qli": false,
    "can_data_upload": true,
    "description": "<p>sample_description</p>",
    "enable_designated_credential": false,
    "all_schemas": null,
    "deleted": false,
    "supports_md_diagnostics": true,
    "aws_region": null,
    "limit_schemas": null,
    "has_previewable_qli": false,
    "hive_tez_logs_source": null,
    "remove_filtered_schemas": false,
    "profiling_tip": null,
    "aws_access_key_id": null,
    "has_metastore_uri": false,
    "webhdfs_username": "sample_username",
    "webhdfs_port": 50070,
    "latest_extraction_time": null,
    "uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
    "url": "/data/372/",
    "enable_default_schema_extraction": false,
    "jdbc_driver": null,
    "unresolved_mention_fingerprint_method": 0,
    "builtin_datasource": null,
    "otype": "data",
    "can_toggle_ds_privacy": true,
    "exclude_schemas": null,
    "compose_oauth_enabled": false
  }

Data Source Creation

This API can be used to create a new data source.

URL

POST /integration/v1/datasource/

Data Parameters

NameRequiredDescription
dbtypeYesThe database type. Currently the certified types are mysql, oracle, postgresql, sqlserver, redshift, teradata, and snowflake.
hostYes1The host of the data source.
portYes1The port of the data source.
uriNo2Jdbc uri for connecting to datasources.
dbnameNo3The database name of the data source.
db_usernameYes4The service account username.
db_passwordNoThe service account password.
titleYes5The title of the data source.
descriptionNoThe description of the data source.
deployment_setup_completeNoIf the deployment setup is complete. When set to true, complete data source information is required, else, only partial information is required. Defaults to True.
privateNoIf the data source is private. Defaults to false.
is_virtualNoIf the data source is virtual. Defaults to false.
is_hiddenNoIf the data source is hidden. Defaults to false.
instance_or_warehouse_nameNoRequired for Snowflake (when used with host/port) and optional for SQL Server.
  1. Not required if datasource is virtual (is_virtual == True) or if the uri parameter is provided.

  2. Ensure you either give host/port or uri parameter. Please do not enter both the parameters.

  3. Required only for oracle (service name is dbname), postgresql, and redshift.

  4. Not required if data source is undeployed (deployment_setup_complete == False) or data source is virtual (is_virtual == True).

  5. Not required if data source is undeployed (deployment_setup_complete == False).

Sample Request Body with Host and Port

{
  "host": "10.11.21.125",
  "port": 3306,
  "dbtype": "mysql",
  "dbname": "sample_dbname",
  "db_username": "sample_username",
  "db_password": "sample_password",
  "title": "sample_title",
  "description": "sample_description"
}

Sample Request Body with URI

{
  "uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
  "dbtype": "mysql",
  "db_username": "sample_username",
  "db_password": "sample_password",
  "title": "sample_title",
  "description": "sample_description"
}

Headers

HTTP HeaderValue
TOKEN<your_token>
Content-Typeapplication/json

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 201 CREATED

Body

{
    "host": "10.11.21.125",
    "port": 3306,
    "deployment_setup_complete": true,
    "db_username": "sample_username",
    "dbname": "sample_dbname",
    "supports_explain": true,
    "has_aws_glue_metastore": false,
    "supports_qli_daterange": true,
    "icon": "db",
    "qli_aws_access_key_id": "",
    "private": false,
    "data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
    "obfuscate_literals": null,
    "is_virtual": false,
    "metastore_type": 0,
    "latest_extraction_successful": false,
    "is_hidden": false,
    "id": 372,
    "owner_ids": [
        1
    ],
    "exclude_additional_columns_in_qli": false,
    "disable_auto_extraction": true,
    "hive_logs_source_type": 0,
    "metastore_uri": null,
    "qualified_name": "",
    "is_hive": false,
    "title": "sample_title",
    "dbtype": "mysql",
    "supports_default_schema_extraction": false,
    "supports_profiling_v2": false,
    "qli_aws_region": "",
    "is_gone": false,
    "favorited_by_list": null,
    "webhdfs_server": null,
    "enabled_in_compose": true,
    "supports_profiling": true,
    "supports_qli_diagnostics": true,
    "is_presto_hive": false,
    "nosql_mde_sample_size": 100,
    "qli_hive_connection_source": 0,
    "cron_extraction": "20 2 * * 1",
    "negative_filter_words": null,
    "supports_compose": true,
    "hive_logs_source": null,
    "has_hdfs_based_qli": false,
    "has_aws_s3_based_qli": false,
    "can_data_upload": true,
    "description": "<p>sample_description</p>",
    "enable_designated_credential": false,
    "all_schemas": null,
    "deleted": false,
    "supports_md_diagnostics": true,
    "aws_region": null,
    "limit_schemas": null,
    "has_previewable_qli": false,
    "hive_tez_logs_source": null,
    "remove_filtered_schemas": false,
    "profiling_tip": null,
    "aws_access_key_id": null,
    "has_metastore_uri": false,
    "webhdfs_username": "sample_username",
    "webhdfs_port": 50070,
    "latest_extraction_time": null,
    "uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
    "url": "/data/372/",
    "enable_default_schema_extraction": false,
    "jdbc_driver": null,
    "unresolved_mention_fingerprint_method": 0,
    "builtin_datasource": null,
    "otype": "data",
    "can_toggle_ds_privacy": true,
    "exclude_schemas": null,
    "compose_oauth_enabled": false
}

Data Source Update

This API can be used to update an existing data source.

URL

PUT /integration/v1/datasource/**<ds_id>**/

Replace <ds_id> with your data source id.

Data Parameters

NameRequiredDescription
dbtype1NoThe database type.
hostNo2The host of the data source.
portNo2The port of the data source.
uriNoThe uri of the data source. Before deployment, data source host and port information can be updated. After deployment, a uri is constructed from the host and port, and instead of the host and port, the uri should be updated.
dbnameNo3The database name of the data source.
db_usernameNoThe service account username.
db_passwordNoThe service account password.
titleNo4The title of the data source.
descriptionNoThe description of the data source.
deployment_setup_completeNoBoolean flag determining if the deployment setup is complete. When set to true, complete data source information is required, otherwise, only partial information is required. Defaults to True. After this field is updated to True and data source is deployed, it cannot be updated to False again.
privateNoBoolean flag determining if the datasource is private. Defaults to False.
  1. Cannot update the dbtype if the datasource has been deployed (deployment_setup_complete == True). Certified types are mysql, oracle, postgresql, sqlserver, redshift, teradata and snowflake.

  2. Host/Port update only possible if data source is undeployed that is deployment_setup_complete == False.

  3. dbname cannot be updated if data source is deployed i.e. deployment_setup_complete == True. Please update the dbname directly in the jdbc uri.

  4. title required only if the value was not previously provided when the data source was undeployed i.e. (deployment_setup_complete == False).

Sample Request Body

{
    "uri": "mysql://10.11.21.130:3306/?allowMultiQueries=true",
    "db_username": "updated_username",
    "db_password":"updated_password",
    "title": "updated_title",
    "description": "updated_description"
}

Headers

HTTP HeaderValue
TOKEN<your_token>
Content-Typeapplication/json

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

{
    "host": "10.11.21.125",
    "port": 3306,
    "deployment_setup_complete": false,
    "db_username": "username",
    "supports_explain": true,
    "private": false,
    "data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
    "is_hidden": false,
    "hive_logs_source_type": 0,
    "metastore_uri": null,
    "is_hive": false,
    "title": "title",
    "is_gone": false,
    "is_virtual": false,
    "webhdfs_server": null,
    "supports_qli_diagnostics": false,
    "is_presto_hive": false,
    "latest_extraction_time": null,
    "negative_filter_words": null,
    "has_hdfs_based_qli": false,
    "can_data_upload": true,
    "qualified_name": "",
    "all_schemas": null,
    "has_previewable_qli": true,
    "hive_tez_logs_source": null,
    "has_metastore_uri": false,
    "webhdfs_port": 50070,
    "supports_qli_daterange": true,
    "latest_extraction_successful": false,
    "id": 172,
    "owner_ids": [
        2
    ],
    "dbtype": "mysql",
    "favorited_by_list": null,
    "supports_compose": true,
    "hive_logs_source": null,
    "description": "",
    "enable_designated_credential": false,
    "deleted": false,
    "limit_schemas": null,
    "obfuscate_literals": null,
    "remove_filtered_schemas": false,
    "profiling_tip": null,
    "supports_profiling": true,
    "webhdfs_username": "username",
    "icon": "db",
    "url": "/add_data_source/172/",
    "uri": "mysql://10.11.21.125:3306/?allowMultiQueries=true",
    "otype": "data",
    "exclude_schemas": null
}

Data Source Deletion

This API can be used to delete a data source.

URL

DELETE /integration/v1/datasource/**<ds_id>**/

Replace <ds_id> with your data source id.

Headers

HTTP HeaderValue
TOKEN<your_token>
Content-Typeapplication/json

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Data Source Configuration Checks

These APIs can be used to run data source configuration checks.

URL

GET /integration/v1/datasource/**<ds_id>**/configuration_check/network_connection/ GET /integration/v1/datasource/**<ds_id>**/configuration_check/service_account_authentication/ GET /integration/v1/datasource/**<ds_id>**/configuration_check/service_account_privileges/ GET /integration/v1/datasource/**<ds_id>**/configuration_check/

Replace <ds_id> with your data source id.

Headers

HTTP HeaderValue
TOKEN<your_token>
Content-Typeapplication/json

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

{
    "status": "ok"
}

Data Source Metadata Extraction

These APIs can be used to get available schemas, trigger metadata extraction and get or update metadata extraction configurations

URL

GET /integration/v1/datasource/**<ds_id>**/available_schemas/?**<params>**

Replace <ds_id> with your data source id.

Replace <params> with your list of parameters.

This API can be used to return the list of available schemas.

Note: Run with force_refresh = true for the first time triggering this API and it is not supported for virtual datasources.

URL Parameters

NameRequiredDescription
force_refreshNoSpecifies if schema list retrieved should be refreshed.

Headers

HTTP HeaderValue
TOKEN<your_token>
Content-Typeapplication/json

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

{
    "schemas": [
      "schema_01",
      "schema_02"
    ]
}

URL

GET /integration/v1/datasource/**<ds_id>**/sync_configuration/metadata_extraction/ PUT /integration/v1/datasource/**<ds_id>**/sync_configuration/metadata_extraction/ PATCH /integration/v1/datasource/**<ds_id>**/sync_configuration/metadata_extraction/

Replace <ds_id> with your data source id.

This API can also be used to get and update the existing configurations for metadata extraction.

Data Parameters

NameRequiredDescription
cron_extractionNoThe extraction schedule in crontab format (minute, hour, day of month, month of year, day of week)
disable_auto_extractionNoTrue if the extraction schedule should not be executed, false to run extraction according to cron_extraction
limit_schemasYesSchemas to include in metadata extraction. If there are no schemas to include, then set to an empty array. Both parameters - limit_schemas and exclude_schemas - cannot have values filled at the same time.
exclude_schemasYesSchemas to exclude from metadata extraction. If there are no schemas to exclude, then set to an empty array. Both parameters - exclude_schemas and limit_schemas - cannot have values filled at the same time.
remove_filtered_schemasNoBoolean flag determining if existing schemas should be removed. If set to True, the existing schemas which are not present in the current metadata extraction scope, are marked as deleted. Defaults to False.

Sample Request Body

{
    "cron_extraction": "0 4 4 * 2",
    "disable_auto_extraction": false,
    "limit_schemas": [],
    "exclude_schemas": ["abc"],
    "remove_filtered_schemas": false
}

Headers

HTTP HeaderValue
TOKEN<your_token>
Content-Typeapplication/json

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

{
    "cron_extraction": "0 4 4 * 2",
    "remove_filtered_schemas": false,
    "limit_schemas": [],
    "exclude_schemas": [
        "abc"
    ],
    "disable_auto_extraction": false
}

URL

POST /integration/v1/datasource/**<ds_id>**/metadata_extraction_job/

Replace <ds_id> with your data source id.

This API allows admins to trigger metadata extraction with or without parameters. Ensure that you run available_schemas API before you run this API for the first time.

Data Parameters

NameRequiredDescription
limit_schemasYesSchemas to include in metadata extraction. If there are no schemas to include, then set to an empty array. Both parameters - limit_schemas and exclude_schemas - cannot have values filled at the same time.
exclude_schemasYesSchemas to exclude from metadata extraction. If there are no schemas to exclude, then set to an empty array. Both parameters - exclude_schemas and limit_schemas - cannot have values filled at the same time.
remove_filtered_schemasYesBoolean flag determining if existing schemas should be removed. If set to True, the existing schemas which are not present in the current metadata extraction scope, are marked as deleted. Defaults to False.

Sample Request Body

{
    "limit_schemas":[],
    "exclude_schemas": ["abc"],
    "remove_filtered_schemas": false
}

Headers

HTTP HeaderValue
TOKEN<your_token>
Content-Typeapplication/json

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

{"job_id": 165}

This API allows admins to trigger metadata extraction with or without parameters. It requires admin privilege. It only works for data sources which are deployed. If parameters are passed then it overrides default parameters.