Datasource API Overview

API reference for Data Sources

Description

The APIs below can be used for Data Sources operations such as

Create a Data Source
Delete a Data Source
Change or update Data Source details
Run configuration checks
Setup and run metadata extraction

Data Source API's is currently certified for mysql, oracle, postgresql, sqlserver, redshift, teradata, and snowflake.

NOTE: Make sure that to input these data sources in lower case for the JSON body.

Open API 3.0 Specification

Datasource APIs are described using the Open API 3.0 Specification (OAS). OAS is a broadly adopted industry standard for describing APIs.

To see the specification, replace {AlationInstanceURL} below with your Alation instance's URL and visit the link:

{AlationInstanceURL}/openapi/datasources/

NOTE

These APIs are available in Alation version 2020.4 and later.
The Swagger UI is not enabled by default on an Alation instance. Please set the flag alation.feature_flags.enable_swagger to True using alation_conf.

Data Source Retrieval (Bulk)

This API can be used to retrieve multiple existing data sources in bulk.

URL

GET /integration/v1/datasource/?**<params>**

Replace <params> with your list of parameters.

URL Parameters

Name	Required	Description
include_undeployed	No	Specifies if undeployed¹ data sources should be included in retrieved list. (By default, it is false) Example: `/integration/v1/datasource/?include_undeployed=true`
include_hidden	No	Specifies if hidden² data sources should be included in retrieved list. Only supported for users with Server Admin role. (By default, it is false) Example: `/integration/v1/datasource/?include_hidden=true`

Undeployed data sources are the ones whose configurations and deployment setup is not complete. Admins cannot trigger metadata extraction on undeployed data sources.
Hidden data sources are not visible in the UI. These are the data sources created via the API with the is_hidden property set to True. There is no UI for the Settings page of such sources and they can only be accessed through the API.

Headers

HTTP Header	Value
TOKEN	<your_token>

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

[
  {
    "host": "10.11.21.125",
    "port": 3306,
    "deployment_setup_complete": true,
    "db_username": "sample_username",
    "dbname": "sample_dbname",
    "supports_explain": true,
    "has_aws_glue_metastore": false,
    "supports_qli_daterange": true,
    "icon": "db",
    "qli_aws_access_key_id": "",
    "private": false,
    "data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
    "obfuscate_literals": null,
    "is_virtual": false,
    "metastore_type": 0,
    "latest_extraction_successful": false,
    "is_hidden": false,
    "id": 372,
    "owner_ids": [
        1
    ],
    "exclude_additional_columns_in_qli": false,
    "disable_auto_extraction": true,
    "hive_logs_source_type": 0,
    "metastore_uri": null,
    "qualified_name": "",
    "is_hive": false,
    "title": "sample_title",
    "dbtype": "mysql",
    "supports_default_schema_extraction": false,
    "supports_profiling_v2": false,
    "qli_aws_region": "",
    "is_gone": false,
    "favorited_by_list": null,
    "webhdfs_server": null,
    "enabled_in_compose": true,
    "supports_profiling": true,
    "supports_qli_diagnostics": true,
    "is_presto_hive": false,
    "nosql_mde_sample_size": 100,
    "qli_hive_connection_source": 0,
    "cron_extraction": "20 2 * * 1",
    "negative_filter_words": null,
    "supports_compose": true,
    "hive_logs_source": null,
    "has_hdfs_based_qli": false,
    "has_aws_s3_based_qli": false,
    "can_data_upload": true,
    "description": "<p>sample_description</p>",
    "enable_designated_credential": false,
    "all_schemas": null,
    "deleted": false,
    "supports_md_diagnostics": true,
    "aws_region": null,
    "limit_schemas": null,
    "has_previewable_qli": false,
    "hive_tez_logs_source": null,
    "remove_filtered_schemas": false,
    "profiling_tip": null,
    "aws_access_key_id": null,
    "has_metastore_uri": false,
    "webhdfs_username": "sample_username",
    "webhdfs_port": 50070,
    "latest_extraction_time": null,
    "uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
    "url": "/data/372/",
    "enable_default_schema_extraction": false,
    "jdbc_driver": null,
    "unresolved_mention_fingerprint_method": 0,
    "builtin_datasource": null,
    "otype": "data",
    "can_toggle_ds_privacy": true,
    "exclude_schemas": null,
    "compose_oauth_enabled": false
  }
]

Data Source Retrieval (Point)

This API can be used to retrieve a specific data source.

URL

GET /integration/v1/datasource/**<ds_id>**/

Replace <ds_id> with your data source id.

Headers

HTTP Header	Value
TOKEN	<your_token>

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

  {
    "host": "10.11.21.125",
    "port": 3306,
    "deployment_setup_complete": true,
    "db_username": "sample_username",
    "dbname": "sample_dbname",
    "supports_explain": true,
    "has_aws_glue_metastore": false,
    "supports_qli_daterange": true,
    "icon": "db",
    "qli_aws_access_key_id": "",
    "private": false,
    "data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
    "obfuscate_literals": null,
    "is_virtual": false,
    "metastore_type": 0,
    "latest_extraction_successful": false,
    "is_hidden": false,
    "id": 372,
    "owner_ids": [
        1
    ],
    "exclude_additional_columns_in_qli": false,
    "disable_auto_extraction": true,
    "hive_logs_source_type": 0,
    "metastore_uri": null,
    "qualified_name": "",
    "is_hive": false,
    "title": "sample_title",
    "dbtype": "mysql",
    "supports_default_schema_extraction": false,
    "supports_profiling_v2": false,
    "qli_aws_region": "",
    "is_gone": false,
    "favorited_by_list": null,
    "webhdfs_server": null,
    "enabled_in_compose": true,
    "supports_profiling": true,
    "supports_qli_diagnostics": true,
    "is_presto_hive": false,
    "nosql_mde_sample_size": 100,
    "qli_hive_connection_source": 0,
    "cron_extraction": "20 2 * * 1",
    "negative_filter_words": null,
    "supports_compose": true,
    "hive_logs_source": null,
    "has_hdfs_based_qli": false,
    "has_aws_s3_based_qli": false,
    "can_data_upload": true,
    "description": "<p>sample_description</p>",
    "enable_designated_credential": false,
    "all_schemas": null,
    "deleted": false,
    "supports_md_diagnostics": true,
    "aws_region": null,
    "limit_schemas": null,
    "has_previewable_qli": false,
    "hive_tez_logs_source": null,
    "remove_filtered_schemas": false,
    "profiling_tip": null,
    "aws_access_key_id": null,
    "has_metastore_uri": false,
    "webhdfs_username": "sample_username",
    "webhdfs_port": 50070,
    "latest_extraction_time": null,
    "uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
    "url": "/data/372/",
    "enable_default_schema_extraction": false,
    "jdbc_driver": null,
    "unresolved_mention_fingerprint_method": 0,
    "builtin_datasource": null,
    "otype": "data",
    "can_toggle_ds_privacy": true,
    "exclude_schemas": null,
    "compose_oauth_enabled": false
  }

Data Source Creation

This API can be used to create a new data source.

URL

POST /integration/v1/datasource/

Data Parameters

Name	Required	Description
dbtype	Yes	The database type. Currently the certified types are `mysql`, `oracle`, `postgresql`, `sqlserver`, `redshift`, `teradata`, and `snowflake`.
host	Yes¹	The host of the data source.
port	Yes¹	The port of the data source.
uri	No²	Jdbc uri for connecting to datasources.
dbname	No³	The database name of the data source.
db_username	Yes⁴	The service account username.
db_password	No	The service account password.
title	Yes⁵	The title of the data source.
description	No	The description of the data source.
deployment_setup_complete	No	If the deployment setup is complete. When set to true, complete data source information is required, else, only partial information is required. Defaults to True.
private	No	If the data source is private. Defaults to false.
is_virtual	No	If the data source is virtual. Defaults to false.
is_hidden	No	If the data source is hidden. Defaults to false.
instance_or_warehouse_name	No	Required for Snowflake (when used with host/port) and optional for SQL Server.

Not required if datasource is virtual (is_virtual == True) or if the uri parameter is provided.
Ensure you either give host/port or uri parameter. Please do not enter both the parameters.
Required only for oracle (service name is dbname), postgresql, and redshift.
Not required if data source is undeployed (deployment_setup_complete == False) or data source is virtual (is_virtual == True).
Not required if data source is undeployed (deployment_setup_complete == False).

Sample Request Body with Host and Port

{
  "host": "10.11.21.125",
  "port": 3306,
  "dbtype": "mysql",
  "dbname": "sample_dbname",
  "db_username": "sample_username",
  "db_password": "sample_password",
  "title": "sample_title",
  "description": "sample_description"
}

Sample Request Body with URI

{
  "uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
  "dbtype": "mysql",
  "db_username": "sample_username",
  "db_password": "sample_password",
  "title": "sample_title",
  "description": "sample_description"
}

Headers

HTTP Header	Value
TOKEN	<your_token>
Content-Type	application/json

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 201 CREATED

Body

{
    "host": "10.11.21.125",
    "port": 3306,
    "deployment_setup_complete": true,
    "db_username": "sample_username",
    "dbname": "sample_dbname",
    "supports_explain": true,
    "has_aws_glue_metastore": false,
    "supports_qli_daterange": true,
    "icon": "db",
    "qli_aws_access_key_id": "",
    "private": false,
    "data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
    "obfuscate_literals": null,
    "is_virtual": false,
    "metastore_type": 0,
    "latest_extraction_successful": false,
    "is_hidden": false,
    "id": 372,
    "owner_ids": [
        1
    ],
    "exclude_additional_columns_in_qli": false,
    "disable_auto_extraction": true,
    "hive_logs_source_type": 0,
    "metastore_uri": null,
    "qualified_name": "",
    "is_hive": false,
    "title": "sample_title",
    "dbtype": "mysql",
    "supports_default_schema_extraction": false,
    "supports_profiling_v2": false,
    "qli_aws_region": "",
    "is_gone": false,
    "favorited_by_list": null,
    "webhdfs_server": null,
    "enabled_in_compose": true,
    "supports_profiling": true,
    "supports_qli_diagnostics": true,
    "is_presto_hive": false,
    "nosql_mde_sample_size": 100,
    "qli_hive_connection_source": 0,
    "cron_extraction": "20 2 * * 1",
    "negative_filter_words": null,
    "supports_compose": true,
    "hive_logs_source": null,
    "has_hdfs_based_qli": false,
    "has_aws_s3_based_qli": false,
    "can_data_upload": true,
    "description": "<p>sample_description</p>",
    "enable_designated_credential": false,
    "all_schemas": null,
    "deleted": false,
    "supports_md_diagnostics": true,
    "aws_region": null,
    "limit_schemas": null,
    "has_previewable_qli": false,
    "hive_tez_logs_source": null,
    "remove_filtered_schemas": false,
    "profiling_tip": null,
    "aws_access_key_id": null,
    "has_metastore_uri": false,
    "webhdfs_username": "sample_username",
    "webhdfs_port": 50070,
    "latest_extraction_time": null,
    "uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
    "url": "/data/372/",
    "enable_default_schema_extraction": false,
    "jdbc_driver": null,
    "unresolved_mention_fingerprint_method": 0,
    "builtin_datasource": null,
    "otype": "data",
    "can_toggle_ds_privacy": true,
    "exclude_schemas": null,
    "compose_oauth_enabled": false
}

Data Source Update

This API can be used to update an existing data source.

URL

PUT /integration/v1/datasource/**<ds_id>**/

Replace <ds_id> with your data source id.

Data Parameters

Name	Required	Description
dbtype¹	No	The database type.
host	No²	The host of the data source.
port	No²	The port of the data source.
uri	No	The uri of the data source. Before deployment, data source host and port information can be updated. After deployment, a uri is constructed from the host and port, and instead of the host and port, the uri should be updated.
dbname	No³	The database name of the data source.
db_username	No	The service account username.
db_password	No	The service account password.
title	No⁴	The title of the data source.
description	No	The description of the data source.
deployment_setup_complete	No	Boolean flag determining if the deployment setup is complete. When set to true, complete data source information is required, otherwise, only partial information is required. Defaults to True. After this field is updated to True and data source is deployed, it cannot be updated to False again.
private	No	Boolean flag determining if the datasource is private. Defaults to False.

Cannot update the dbtype if the datasource has been deployed (deployment_setup_complete == True). Certified types are mysql, oracle, postgresql, sqlserver, redshift, teradata and snowflake.
Host/Port update only possible if data source is undeployed that is deployment_setup_complete == False.
dbname cannot be updated if data source is deployed i.e. deployment_setup_complete == True. Please update the dbname directly in the jdbc uri.
title required only if the value was not previously provided when the data source was undeployed i.e. (deployment_setup_complete == False).

Sample Request Body

{
    "uri": "mysql://10.11.21.130:3306/?allowMultiQueries=true",
    "db_username": "updated_username",
    "db_password":"updated_password",
    "title": "updated_title",
    "description": "updated_description"
}

Headers

HTTP Header	Value
TOKEN	<your_token>
Content-Type	application/json

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

{
    "host": "10.11.21.125",
    "port": 3306,
    "deployment_setup_complete": false,
    "db_username": "username",
    "supports_explain": true,
    "private": false,
    "data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
    "is_hidden": false,
    "hive_logs_source_type": 0,
    "metastore_uri": null,
    "is_hive": false,
    "title": "title",
    "is_gone": false,
    "is_virtual": false,
    "webhdfs_server": null,
    "supports_qli_diagnostics": false,
    "is_presto_hive": false,
    "latest_extraction_time": null,
    "negative_filter_words": null,
    "has_hdfs_based_qli": false,
    "can_data_upload": true,
    "qualified_name": "",
    "all_schemas": null,
    "has_previewable_qli": true,
    "hive_tez_logs_source": null,
    "has_metastore_uri": false,
    "webhdfs_port": 50070,
    "supports_qli_daterange": true,
    "latest_extraction_successful": false,
    "id": 172,
    "owner_ids": [
        2
    ],
    "dbtype": "mysql",
    "favorited_by_list": null,
    "supports_compose": true,
    "hive_logs_source": null,
    "description": "",
    "enable_designated_credential": false,
    "deleted": false,
    "limit_schemas": null,
    "obfuscate_literals": null,
    "remove_filtered_schemas": false,
    "profiling_tip": null,
    "supports_profiling": true,
    "webhdfs_username": "username",
    "icon": "db",
    "url": "/add_data_source/172/",
    "uri": "mysql://10.11.21.125:3306/?allowMultiQueries=true",
    "otype": "data",
    "exclude_schemas": null
}

Data Source Deletion

This API can be used to delete a data source.

URL

DELETE /integration/v1/datasource/**<ds_id>**/

Replace <ds_id> with your data source id.

Headers

HTTP Header	Value
TOKEN	<your_token>
Content-Type	application/json

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Data Source Configuration Checks

These APIs can be used to run data source configuration checks.

URL

GET /integration/v1/datasource/**<ds_id>**/configuration_check/network_connection/ GET /integration/v1/datasource/**<ds_id>**/configuration_check/service_account_authentication/ GET /integration/v1/datasource/**<ds_id>**/configuration_check/service_account_privileges/ GET /integration/v1/datasource/**<ds_id>**/configuration_check/

Replace <ds_id> with your data source id.

Headers

HTTP Header	Value
TOKEN	<your_token>
Content-Type	application/json

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

{
    "status": "ok"
}

Data Source Metadata Extraction

These APIs can be used to get available schemas, trigger metadata extraction and get or update metadata extraction configurations

URL

GET /integration/v1/datasource/**<ds_id>**/available_schemas/?**<params>**

Replace <ds_id> with your data source id.

Replace <params> with your list of parameters.

This API can be used to return the list of available schemas.

Note: Run with force_refresh = true for the first time triggering this API and it is not supported for virtual datasources.

URL Parameters

Name	Required	Description
force_refresh	No	Specifies if schema list retrieved should be refreshed.

Headers

HTTP Header	Value
TOKEN	<your_token>
Content-Type	application/json

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

{
    "schemas": [
      "schema_01",
      "schema_02"
    ]
}

URL

GET /integration/v1/datasource/**<ds_id>**/sync_configuration/metadata_extraction/ PUT /integration/v1/datasource/**<ds_id>**/sync_configuration/metadata_extraction/ PATCH /integration/v1/datasource/**<ds_id>**/sync_configuration/metadata_extraction/

Replace <ds_id> with your data source id.

This API can also be used to get and update the existing configurations for metadata extraction.

Data Parameters

Name	Required	Description
cron_extraction	No	The extraction schedule in crontab format (minute, hour, day of month, month of year, day of week)
disable_auto_extraction	No	True if the extraction schedule should not be executed, false to run extraction according to `cron_extraction`
limit_schemas	Yes	Schemas to include in metadata extraction. If there are no schemas to include, then set to an empty array. Both parameters - `limit_schemas` and `exclude_schemas` - cannot have values filled at the same time.
exclude_schemas	Yes	Schemas to exclude from metadata extraction. If there are no schemas to exclude, then set to an empty array. Both parameters - `exclude_schemas` and `limit_schemas` - cannot have values filled at the same time.
remove_filtered_schemas	No	Boolean flag determining if existing schemas should be removed. If set to `True`, the existing schemas which are not present in the current metadata extraction scope, are marked as deleted. Defaults to `False`.

Sample Request Body

{
    "cron_extraction": "0 4 4 * 2",
    "disable_auto_extraction": false,
    "limit_schemas": [],
    "exclude_schemas": ["abc"],
    "remove_filtered_schemas": false
}

Headers

HTTP Header	Value
TOKEN	<your_token>
Content-Type	application/json

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

{
    "cron_extraction": "0 4 4 * 2",
    "remove_filtered_schemas": false,
    "limit_schemas": [],
    "exclude_schemas": [
        "abc"
    ],
    "disable_auto_extraction": false
}

URL

POST /integration/v1/datasource/**<ds_id>**/metadata_extraction_job/

Replace <ds_id> with your data source id.

This API allows admins to trigger metadata extraction with or without parameters. Ensure that you run available_schemas API before you run this API for the first time.

Data Parameters

Name	Required	Description
limit_schemas	Yes	Schemas to include in metadata extraction. If there are no schemas to include, then set to an empty array. Both parameters - `limit_schemas` and `exclude_schemas` - cannot have values filled at the same time.
exclude_schemas	Yes	Schemas to exclude from metadata extraction. If there are no schemas to exclude, then set to an empty array. Both parameters - `exclude_schemas` and `limit_schemas` - cannot have values filled at the same time.
remove_filtered_schemas	Yes	Boolean flag determining if existing schemas should be removed. If set to `True`, the existing schemas which are not present in the current metadata extraction scope, are marked as deleted. Defaults to `False`.

Sample Request Body

{
    "limit_schemas":[],
    "exclude_schemas": ["abc"],
    "remove_filtered_schemas": false
}

Headers

HTTP Header	Value
TOKEN	<your_token>
Content-Type	application/json

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

{"job_id": 165}

This API allows admins to trigger metadata extraction with or without parameters. It requires admin privilege. It only works for data sources which are deployed. If parameters are passed then it overrides default parameters.