API reference for Data Sources
Description
The APIs below can be used for Data Sources operations such as
- Create a Data Source
- Delete a Data Source
- Change or update Data Source details
- Run configuration checks
- Setup and run metadata extraction
Data Source API's is currently certified for mysql
, oracle
, postgresql
, sqlserver
, redshift
, teradata
, and snowflake
.
NOTE: Make sure that to input these data sources in lower case for the JSON body.
Open API 3.0 Specification
Datasource APIs are described using the Open API 3.0 Specification (OAS). OAS is a broadly adopted industry standard for describing APIs.
To see the specification, replace {AlationInstanceURL}
below with your Alation instance's URL and visit the link:
{AlationInstanceURL}/openapi/datasources/
NOTE
- These APIs are available in Alation version 2020.4 and later.
- The Swagger UI is not enabled by default on an Alation instance. Please set the flag
alation.feature_flags.enable_swagger
toTrue
usingalation_conf
.
Data Source Retrieval (Bulk)
This API can be used to retrieve multiple existing data sources in bulk.
URL
GET
/integration/v1/datasource/?**<params>**
Replace <params> with your list of parameters.
URL Parameters
Name | Required | Description |
---|---|---|
include_undeployed | No | Specifies if undeployed1 data sources should be included in retrieved list. (By default, it is false) Example: /integration/v1/datasource/?include_undeployed=true |
include_hidden | No | Specifies if hidden2 data sources should be included in retrieved list. Only supported for users with Server Admin role. (By default, it is false) Example: /integration/v1/datasource/?include_hidden=true |
-
Undeployed data sources are the ones whose configurations and deployment setup is not complete. Admins cannot trigger metadata extraction on undeployed data sources.
-
Hidden data sources are not visible in the UI. These are the data sources created via the API with the
is_hidden
property set toTrue
. There is no UI for the Settings page of such sources and they can only be accessed through the API.
Headers
HTTP Header | Value |
---|---|
TOKEN | <your_token> |
Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).
Success Response
Content-Type: application/json
Status: 200 OK
Body
[
{
"host": "10.11.21.125",
"port": 3306,
"deployment_setup_complete": true,
"db_username": "sample_username",
"dbname": "sample_dbname",
"supports_explain": true,
"has_aws_glue_metastore": false,
"supports_qli_daterange": true,
"icon": "db",
"qli_aws_access_key_id": "",
"private": false,
"data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
"obfuscate_literals": null,
"is_virtual": false,
"metastore_type": 0,
"latest_extraction_successful": false,
"is_hidden": false,
"id": 372,
"owner_ids": [
1
],
"exclude_additional_columns_in_qli": false,
"disable_auto_extraction": true,
"hive_logs_source_type": 0,
"metastore_uri": null,
"qualified_name": "",
"is_hive": false,
"title": "sample_title",
"dbtype": "mysql",
"supports_default_schema_extraction": false,
"supports_profiling_v2": false,
"qli_aws_region": "",
"is_gone": false,
"favorited_by_list": null,
"webhdfs_server": null,
"enabled_in_compose": true,
"supports_profiling": true,
"supports_qli_diagnostics": true,
"is_presto_hive": false,
"nosql_mde_sample_size": 100,
"qli_hive_connection_source": 0,
"cron_extraction": "20 2 * * 1",
"negative_filter_words": null,
"supports_compose": true,
"hive_logs_source": null,
"has_hdfs_based_qli": false,
"has_aws_s3_based_qli": false,
"can_data_upload": true,
"description": "<p>sample_description</p>",
"enable_designated_credential": false,
"all_schemas": null,
"deleted": false,
"supports_md_diagnostics": true,
"aws_region": null,
"limit_schemas": null,
"has_previewable_qli": false,
"hive_tez_logs_source": null,
"remove_filtered_schemas": false,
"profiling_tip": null,
"aws_access_key_id": null,
"has_metastore_uri": false,
"webhdfs_username": "sample_username",
"webhdfs_port": 50070,
"latest_extraction_time": null,
"uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
"url": "/data/372/",
"enable_default_schema_extraction": false,
"jdbc_driver": null,
"unresolved_mention_fingerprint_method": 0,
"builtin_datasource": null,
"otype": "data",
"can_toggle_ds_privacy": true,
"exclude_schemas": null,
"compose_oauth_enabled": false
}
]
Data Source Retrieval (Point)
This API can be used to retrieve a specific data source.
URL
GET
/integration/v1/datasource/**<ds_id>**/
Replace <ds_id> with your data source id.
Headers
HTTP Header | Value |
---|---|
TOKEN | <your_token> |
Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).
Success Response
Content-Type: application/json
Status: 200 OK
Body
{
"host": "10.11.21.125",
"port": 3306,
"deployment_setup_complete": true,
"db_username": "sample_username",
"dbname": "sample_dbname",
"supports_explain": true,
"has_aws_glue_metastore": false,
"supports_qli_daterange": true,
"icon": "db",
"qli_aws_access_key_id": "",
"private": false,
"data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
"obfuscate_literals": null,
"is_virtual": false,
"metastore_type": 0,
"latest_extraction_successful": false,
"is_hidden": false,
"id": 372,
"owner_ids": [
1
],
"exclude_additional_columns_in_qli": false,
"disable_auto_extraction": true,
"hive_logs_source_type": 0,
"metastore_uri": null,
"qualified_name": "",
"is_hive": false,
"title": "sample_title",
"dbtype": "mysql",
"supports_default_schema_extraction": false,
"supports_profiling_v2": false,
"qli_aws_region": "",
"is_gone": false,
"favorited_by_list": null,
"webhdfs_server": null,
"enabled_in_compose": true,
"supports_profiling": true,
"supports_qli_diagnostics": true,
"is_presto_hive": false,
"nosql_mde_sample_size": 100,
"qli_hive_connection_source": 0,
"cron_extraction": "20 2 * * 1",
"negative_filter_words": null,
"supports_compose": true,
"hive_logs_source": null,
"has_hdfs_based_qli": false,
"has_aws_s3_based_qli": false,
"can_data_upload": true,
"description": "<p>sample_description</p>",
"enable_designated_credential": false,
"all_schemas": null,
"deleted": false,
"supports_md_diagnostics": true,
"aws_region": null,
"limit_schemas": null,
"has_previewable_qli": false,
"hive_tez_logs_source": null,
"remove_filtered_schemas": false,
"profiling_tip": null,
"aws_access_key_id": null,
"has_metastore_uri": false,
"webhdfs_username": "sample_username",
"webhdfs_port": 50070,
"latest_extraction_time": null,
"uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
"url": "/data/372/",
"enable_default_schema_extraction": false,
"jdbc_driver": null,
"unresolved_mention_fingerprint_method": 0,
"builtin_datasource": null,
"otype": "data",
"can_toggle_ds_privacy": true,
"exclude_schemas": null,
"compose_oauth_enabled": false
}
Data Source Creation
This API can be used to create a new data source.
URL
POST
/integration/v1/datasource/
Data Parameters
Name | Required | Description |
---|---|---|
dbtype | Yes | The database type. Currently the certified types are mysql , oracle , postgresql , sqlserver , redshift , teradata , and snowflake . |
host | Yes1 | The host of the data source. |
port | Yes1 | The port of the data source. |
uri | No2 | Jdbc uri for connecting to datasources. |
dbname | No3 | The database name of the data source. |
db_username | Yes4 | The service account username. |
db_password | No | The service account password. |
title | Yes5 | The title of the data source. |
description | No | The description of the data source. |
deployment_setup_complete | No | If the deployment setup is complete. When set to true, complete data source information is required, else, only partial information is required. Defaults to True. |
private | No | If the data source is private. Defaults to false. |
is_virtual | No | If the data source is virtual. Defaults to false. |
is_hidden | No | If the data source is hidden. Defaults to false. |
instance_or_warehouse_name | No | Required for Snowflake (when used with host/port) and optional for SQL Server. |
-
Not required if datasource is virtual (is_virtual == True) or if the
uri
parameter is provided. -
Ensure you either give
host/port
oruri
parameter. Please do not enter both the parameters. -
Required only for
oracle
(service name is dbname),postgresql
, andredshift
. -
Not required if data source is undeployed (deployment_setup_complete == False) or data source is virtual (is_virtual == True).
-
Not required if data source is undeployed (deployment_setup_complete == False).
Sample Request Body with Host and Port
{
"host": "10.11.21.125",
"port": 3306,
"dbtype": "mysql",
"dbname": "sample_dbname",
"db_username": "sample_username",
"db_password": "sample_password",
"title": "sample_title",
"description": "sample_description"
}
Sample Request Body with URI
{
"uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
"dbtype": "mysql",
"db_username": "sample_username",
"db_password": "sample_password",
"title": "sample_title",
"description": "sample_description"
}
Headers
HTTP Header | Value |
---|---|
TOKEN | <your_token> |
Content-Type | application/json |
Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).
Success Response
Content-Type: application/json
Status: 201 CREATED
Body
{
"host": "10.11.21.125",
"port": 3306,
"deployment_setup_complete": true,
"db_username": "sample_username",
"dbname": "sample_dbname",
"supports_explain": true,
"has_aws_glue_metastore": false,
"supports_qli_daterange": true,
"icon": "db",
"qli_aws_access_key_id": "",
"private": false,
"data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
"obfuscate_literals": null,
"is_virtual": false,
"metastore_type": 0,
"latest_extraction_successful": false,
"is_hidden": false,
"id": 372,
"owner_ids": [
1
],
"exclude_additional_columns_in_qli": false,
"disable_auto_extraction": true,
"hive_logs_source_type": 0,
"metastore_uri": null,
"qualified_name": "",
"is_hive": false,
"title": "sample_title",
"dbtype": "mysql",
"supports_default_schema_extraction": false,
"supports_profiling_v2": false,
"qli_aws_region": "",
"is_gone": false,
"favorited_by_list": null,
"webhdfs_server": null,
"enabled_in_compose": true,
"supports_profiling": true,
"supports_qli_diagnostics": true,
"is_presto_hive": false,
"nosql_mde_sample_size": 100,
"qli_hive_connection_source": 0,
"cron_extraction": "20 2 * * 1",
"negative_filter_words": null,
"supports_compose": true,
"hive_logs_source": null,
"has_hdfs_based_qli": false,
"has_aws_s3_based_qli": false,
"can_data_upload": true,
"description": "<p>sample_description</p>",
"enable_designated_credential": false,
"all_schemas": null,
"deleted": false,
"supports_md_diagnostics": true,
"aws_region": null,
"limit_schemas": null,
"has_previewable_qli": false,
"hive_tez_logs_source": null,
"remove_filtered_schemas": false,
"profiling_tip": null,
"aws_access_key_id": null,
"has_metastore_uri": false,
"webhdfs_username": "sample_username",
"webhdfs_port": 50070,
"latest_extraction_time": null,
"uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
"url": "/data/372/",
"enable_default_schema_extraction": false,
"jdbc_driver": null,
"unresolved_mention_fingerprint_method": 0,
"builtin_datasource": null,
"otype": "data",
"can_toggle_ds_privacy": true,
"exclude_schemas": null,
"compose_oauth_enabled": false
}
Data Source Update
This API can be used to update an existing data source.
URL
PUT
/integration/v1/datasource/**<ds_id>**/
Replace <ds_id> with your data source id.
Data Parameters
Name | Required | Description |
---|---|---|
dbtype1 | No | The database type. |
host | No2 | The host of the data source. |
port | No2 | The port of the data source. |
uri | No | The uri of the data source. Before deployment, data source host and port information can be updated. After deployment, a uri is constructed from the host and port, and instead of the host and port, the uri should be updated. |
dbname | No3 | The database name of the data source. |
db_username | No | The service account username. |
db_password | No | The service account password. |
title | No4 | The title of the data source. |
description | No | The description of the data source. |
deployment_setup_complete | No | Boolean flag determining if the deployment setup is complete. When set to true, complete data source information is required, otherwise, only partial information is required. Defaults to True. After this field is updated to True and data source is deployed, it cannot be updated to False again. |
private | No | Boolean flag determining if the datasource is private. Defaults to False. |
-
Cannot update the
dbtype
if the datasource has been deployed (deployment_setup_complete == True). Certified types aremysql
,oracle
,postgresql
,sqlserver
,redshift
,teradata
andsnowflake
. -
Host/Port update only possible if data source is undeployed that is deployment_setup_complete == False.
-
dbname
cannot be updated if data source is deployed i.e. deployment_setup_complete == True. Please update thedbname
directly in the jdbc uri. -
title
required only if the value was not previously provided when the data source was undeployed i.e. (deployment_setup_complete == False).
Sample Request Body
{
"uri": "mysql://10.11.21.130:3306/?allowMultiQueries=true",
"db_username": "updated_username",
"db_password":"updated_password",
"title": "updated_title",
"description": "updated_description"
}
Headers
HTTP Header | Value |
---|---|
TOKEN | <your_token> |
Content-Type | application/json |
Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).
Success Response
Content-Type: application/json
Status: 200 OK
Body
{
"host": "10.11.21.125",
"port": 3306,
"deployment_setup_complete": false,
"db_username": "username",
"supports_explain": true,
"private": false,
"data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
"is_hidden": false,
"hive_logs_source_type": 0,
"metastore_uri": null,
"is_hive": false,
"title": "title",
"is_gone": false,
"is_virtual": false,
"webhdfs_server": null,
"supports_qli_diagnostics": false,
"is_presto_hive": false,
"latest_extraction_time": null,
"negative_filter_words": null,
"has_hdfs_based_qli": false,
"can_data_upload": true,
"qualified_name": "",
"all_schemas": null,
"has_previewable_qli": true,
"hive_tez_logs_source": null,
"has_metastore_uri": false,
"webhdfs_port": 50070,
"supports_qli_daterange": true,
"latest_extraction_successful": false,
"id": 172,
"owner_ids": [
2
],
"dbtype": "mysql",
"favorited_by_list": null,
"supports_compose": true,
"hive_logs_source": null,
"description": "",
"enable_designated_credential": false,
"deleted": false,
"limit_schemas": null,
"obfuscate_literals": null,
"remove_filtered_schemas": false,
"profiling_tip": null,
"supports_profiling": true,
"webhdfs_username": "username",
"icon": "db",
"url": "/add_data_source/172/",
"uri": "mysql://10.11.21.125:3306/?allowMultiQueries=true",
"otype": "data",
"exclude_schemas": null
}
Data Source Deletion
This API can be used to delete a data source.
URL
DELETE
/integration/v1/datasource/**<ds_id>**/
Replace <ds_id> with your data source id.
Headers
HTTP Header | Value |
---|---|
TOKEN | <your_token> |
Content-Type | application/json |
Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).
Success Response
Content-Type: application/json
Status: 200 OK
Data Source Configuration Checks
These APIs can be used to run data source configuration checks.
URL
GET
/integration/v1/datasource/**<ds_id>**/configuration_check/network_connection/
GET
/integration/v1/datasource/**<ds_id>**/configuration_check/service_account_authentication/
GET
/integration/v1/datasource/**<ds_id>**/configuration_check/service_account_privileges/
GET
/integration/v1/datasource/**<ds_id>**/configuration_check/
Replace <ds_id> with your data source id.
Headers
HTTP Header | Value |
---|---|
TOKEN | <your_token> |
Content-Type | application/json |
Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).
Success Response
Content-Type: application/json
Status: 200 OK
Body
{
"status": "ok"
}
Data Source Metadata Extraction
These APIs can be used to get available schemas, trigger metadata extraction and get or update metadata extraction configurations
URL
GET
/integration/v1/datasource/**<ds_id>**/available_schemas/?**<params>**
Replace <ds_id> with your data source id.
Replace <params> with your list of parameters.
This API can be used to return the list of available schemas.
Note: Run with force_refresh = true for the first time triggering this API and it is not supported for virtual datasources.
URL Parameters
Name | Required | Description |
---|---|---|
force_refresh | No | Specifies if schema list retrieved should be refreshed. |
Headers
HTTP Header | Value |
---|---|
TOKEN | <your_token> |
Content-Type | application/json |
Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).
Success Response
Content-Type: application/json
Status: 200 OK
Body
{
"schemas": [
"schema_01",
"schema_02"
]
}
URL
GET
/integration/v1/datasource/**<ds_id>**/sync_configuration/metadata_extraction/
PUT
/integration/v1/datasource/**<ds_id>**/sync_configuration/metadata_extraction/
PATCH
/integration/v1/datasource/**<ds_id>**/sync_configuration/metadata_extraction/
Replace <ds_id> with your data source id.
This API can also be used to get and update the existing configurations for metadata extraction.
Data Parameters
Name | Required | Description |
---|---|---|
cron_extraction | No | The extraction schedule in crontab format (minute, hour, day of month, month of year, day of week) |
disable_auto_extraction | No | True if the extraction schedule should not be executed, false to run extraction according to cron_extraction |
limit_schemas | Yes | Schemas to include in metadata extraction. If there are no schemas to include, then set to an empty array. Both parameters - limit_schemas and exclude_schemas - cannot have values filled at the same time. |
exclude_schemas | Yes | Schemas to exclude from metadata extraction. If there are no schemas to exclude, then set to an empty array. Both parameters - exclude_schemas and limit_schemas - cannot have values filled at the same time. |
remove_filtered_schemas | No | Boolean flag determining if existing schemas should be removed. If set to True , the existing schemas which are not present in the current metadata extraction scope, are marked as deleted. Defaults to False . |
Sample Request Body
{
"cron_extraction": "0 4 4 * 2",
"disable_auto_extraction": false,
"limit_schemas": [],
"exclude_schemas": ["abc"],
"remove_filtered_schemas": false
}
Headers
HTTP Header | Value |
---|---|
TOKEN | <your_token> |
Content-Type | application/json |
Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).
Success Response
Content-Type: application/json
Status: 200 OK
Body
{
"cron_extraction": "0 4 4 * 2",
"remove_filtered_schemas": false,
"limit_schemas": [],
"exclude_schemas": [
"abc"
],
"disable_auto_extraction": false
}
URL
POST
/integration/v1/datasource/**<ds_id>**/metadata_extraction_job/
Replace <ds_id> with your data source id.
This API allows admins to trigger metadata extraction with or without parameters. Ensure that you run available_schemas
API before you run this API for the first time.
Data Parameters
Name | Required | Description |
---|---|---|
limit_schemas | Yes | Schemas to include in metadata extraction. If there are no schemas to include, then set to an empty array. Both parameters - limit_schemas and exclude_schemas - cannot have values filled at the same time. |
exclude_schemas | Yes | Schemas to exclude from metadata extraction. If there are no schemas to exclude, then set to an empty array. Both parameters - exclude_schemas and limit_schemas - cannot have values filled at the same time. |
remove_filtered_schemas | Yes | Boolean flag determining if existing schemas should be removed. If set to True , the existing schemas which are not present in the current metadata extraction scope, are marked as deleted. Defaults to False . |
Sample Request Body
{
"limit_schemas":[],
"exclude_schemas": ["abc"],
"remove_filtered_schemas": false
}
Headers
HTTP Header | Value |
---|---|
TOKEN | <your_token> |
Content-Type | application/json |
Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).
Success Response
Content-Type: application/json
Status: 200 OK
Body
{"job_id": 165}
This API allows admins to trigger metadata extraction with or without parameters. It requires admin privilege. It only works for data sources which are deployed. If parameters are passed then it overrides default parameters.