API reference for Data Sources

Description

The APIs below can be used for Data Sources operations such as

1. Create a Data Source
   
2. Delete a Data Source
   
3. Change or update Data Source details
   
4. Run configuration checks
   
5. Setup and run metadata extraction
   

Data Source API's is currently certified for mysql, oracle, postgres, sqlserver, redshift, teradata, and snowflake.

NOTE: Make sure that to input these data sources in lower case for the JSON body.

Open API 3.0 Specification

Datasource APIs are described using the Open API 3.0 Specification (OAS). OAS is a broadly adopted industry standard for describing APIs.

To see the specification, replace {AlationInstanceURL} below with your Alation instance's URL and visit the link:

{AlationInstanceURL}/openapi/datasources/

NOTE

1. These APIs are available in Alation version 2020.4 and later.
   
2. The Swagger UI is not enabled by default on an Alation instance. Please set the flag alation.feature_flags.enable_swagger to True using alation_conf.
   

Data Source Retrieval (Bulk)

This API can be used to retrieve multiple existing data sources in bulk.

URL

Replace <params> with your list of parameters.

URL Parameters

1. Undeployed data sources are the ones whose configurations and deployment setup is not complete. Admins cannot trigger metadata extraction on undeployed data sources.

2. Hidden data sources are not visible in the UI. These are the data sources created via the API with the is_hidden property set to True. There is no UI for the Settings page of such sources and they can only be accessed through the API.

Headers

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

[
  {
    "host": "10.11.21.125",
    "port": 3306,
    "deployment_setup_complete": true,
    "db_username": "sample_username",
    "dbname": "sample_dbname",
    "supports_explain": true,
    "has_aws_glue_metastore": false,
    "supports_qli_daterange": true,
    "icon": "db",
    "qli_aws_access_key_id": "",
    "private": false,
    "data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
    "obfuscate_literals": null,
    "is_virtual": false,
    "metastore_type": 0,
    "latest_extraction_successful": false,
    "is_hidden": false,
    "id": 372,
    "owner_ids": [
        1
    ],
    "exclude_additional_columns_in_qli": false,
    "disable_auto_extraction": true,
    "hive_logs_source_type": 0,
    "metastore_uri": null,
    "qualified_name": "",
    "is_hive": false,
    "title": "sample_title",
    "dbtype": "mysql",
    "supports_default_schema_extraction": false,
    "supports_profiling_v2": false,
    "qli_aws_region": "",
    "is_gone": false,
    "favorited_by_list": null,
    "webhdfs_server": null,
    "enabled_in_compose": true,
    "supports_profiling": true,
    "supports_qli_diagnostics": true,
    "is_presto_hive": false,
    "nosql_mde_sample_size": 100,
    "qli_hive_connection_source": 0,
    "cron_extraction": "20 2 * * 1",
    "negative_filter_words": null,
    "supports_compose": true,
    "hive_logs_source": null,
    "has_hdfs_based_qli": false,
    "has_aws_s3_based_qli": false,
    "can_data_upload": true,
    "description": "<p>sample_description</p>",
    "enable_designated_credential": false,
    "all_schemas": null,
    "deleted": false,
    "supports_md_diagnostics": true,
    "aws_region": null,
    "limit_schemas": null,
    "has_previewable_qli": false,
    "hive_tez_logs_source": null,
    "remove_filtered_schemas": false,
    "profiling_tip": null,
    "aws_access_key_id": null,
    "has_metastore_uri": false,
    "webhdfs_username": "sample_username",
    "webhdfs_port": 50070,
    "latest_extraction_time": null,
    "uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
    "url": "/data/372/",
    "enable_default_schema_extraction": false,
    "jdbc_driver": null,
    "unresolved_mention_fingerprint_method": 0,
    "builtin_datasource": null,
    "otype": "data",
    "can_toggle_ds_privacy": true,
    "exclude_schemas": null,
    "compose_oauth_enabled": false
  }
]

Data Source Retrieval (Point)

This API can be used to retrieve a specific data source.

URL

Replace <ds_id> with your data source id.

Headers

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

  {
    "host": "10.11.21.125",
    "port": 3306,
    "deployment_setup_complete": true,
    "db_username": "sample_username",
    "dbname": "sample_dbname",
    "supports_explain": true,
    "has_aws_glue_metastore": false,
    "supports_qli_daterange": true,
    "icon": "db",
    "qli_aws_access_key_id": "",
    "private": false,
    "data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
    "obfuscate_literals": null,
    "is_virtual": false,
    "metastore_type": 0,
    "latest_extraction_successful": false,
    "is_hidden": false,
    "id": 372,
    "owner_ids": [
        1
    ],
    "exclude_additional_columns_in_qli": false,
    "disable_auto_extraction": true,
    "hive_logs_source_type": 0,
    "metastore_uri": null,
    "qualified_name": "",
    "is_hive": false,
    "title": "sample_title",
    "dbtype": "mysql",
    "supports_default_schema_extraction": false,
    "supports_profiling_v2": false,
    "qli_aws_region": "",
    "is_gone": false,
    "favorited_by_list": null,
    "webhdfs_server": null,
    "enabled_in_compose": true,
    "supports_profiling": true,
    "supports_qli_diagnostics": true,
    "is_presto_hive": false,
    "nosql_mde_sample_size": 100,
    "qli_hive_connection_source": 0,
    "cron_extraction": "20 2 * * 1",
    "negative_filter_words": null,
    "supports_compose": true,
    "hive_logs_source": null,
    "has_hdfs_based_qli": false,
    "has_aws_s3_based_qli": false,
    "can_data_upload": true,
    "description": "<p>sample_description</p>",
    "enable_designated_credential": false,
    "all_schemas": null,
    "deleted": false,
    "supports_md_diagnostics": true,
    "aws_region": null,
    "limit_schemas": null,
    "has_previewable_qli": false,
    "hive_tez_logs_source": null,
    "remove_filtered_schemas": false,
    "profiling_tip": null,
    "aws_access_key_id": null,
    "has_metastore_uri": false,
    "webhdfs_username": "sample_username",
    "webhdfs_port": 50070,
    "latest_extraction_time": null,
    "uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
    "url": "/data/372/",
    "enable_default_schema_extraction": false,
    "jdbc_driver": null,
    "unresolved_mention_fingerprint_method": 0,
    "builtin_datasource": null,
    "otype": "data",
    "can_toggle_ds_privacy": true,
    "exclude_schemas": null,
    "compose_oauth_enabled": false
  }

Data Source Creation

This API can be used to create a new data source.

URL

Data Parameters

1. Not required if datasource is virtual (is_virtual == True) or if the uri parameter is provided.

2. Ensure you either give host/port or uri parameter. Please do not enter both the parameters.

3. Required only for oracle (service name is dbname), postgresql, and redshift.

4. Not required if data source is undeployed (deployment_setup_complete == False) or data source is virtual (is_virtual == True).

5. Not required if data source is undeployed (deployment_setup_complete == False).

Sample Request Body with Host and Port

{
  "host": "10.11.21.125",
  "port": 3306,
  "dbtype": "mysql",
  "dbname": "sample_dbname",
  "db_username": "sample_username",
  "db_password": "sample_password",
  "title": "sample_title",
  "description": "sample_description"
}

Sample Request Body with URI

{
  "uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
  "dbtype": "mysql",
  "db_username": "sample_username",
  "db_password": "sample_password",
  "title": "sample_title",
  "description": "sample_description"
}

Headers

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 201 CREATED

Body

{
    "host": "10.11.21.125",
    "port": 3306,
    "deployment_setup_complete": true,
    "db_username": "sample_username",
    "dbname": "sample_dbname",
    "supports_explain": true,
    "has_aws_glue_metastore": false,
    "supports_qli_daterange": true,
    "icon": "db",
    "qli_aws_access_key_id": "",
    "private": false,
    "data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
    "obfuscate_literals": null,
    "is_virtual": false,
    "metastore_type": 0,
    "latest_extraction_successful": false,
    "is_hidden": false,
    "id": 372,
    "owner_ids": [
        1
    ],
    "exclude_additional_columns_in_qli": false,
    "disable_auto_extraction": true,
    "hive_logs_source_type": 0,
    "metastore_uri": null,
    "qualified_name": "",
    "is_hive": false,
    "title": "sample_title",
    "dbtype": "mysql",
    "supports_default_schema_extraction": false,
    "supports_profiling_v2": false,
    "qli_aws_region": "",
    "is_gone": false,
    "favorited_by_list": null,
    "webhdfs_server": null,
    "enabled_in_compose": true,
    "supports_profiling": true,
    "supports_qli_diagnostics": true,
    "is_presto_hive": false,
    "nosql_mde_sample_size": 100,
    "qli_hive_connection_source": 0,
    "cron_extraction": "20 2 * * 1",
    "negative_filter_words": null,
    "supports_compose": true,
    "hive_logs_source": null,
    "has_hdfs_based_qli": false,
    "has_aws_s3_based_qli": false,
    "can_data_upload": true,
    "description": "<p>sample_description</p>",
    "enable_designated_credential": false,
    "all_schemas": null,
    "deleted": false,
    "supports_md_diagnostics": true,
    "aws_region": null,
    "limit_schemas": null,
    "has_previewable_qli": false,
    "hive_tez_logs_source": null,
    "remove_filtered_schemas": false,
    "profiling_tip": null,
    "aws_access_key_id": null,
    "has_metastore_uri": false,
    "webhdfs_username": "sample_username",
    "webhdfs_port": 50070,
    "latest_extraction_time": null,
    "uri": "mysql://10.11.21.125:3306/sample_dbname?allowMultiQueries=true",
    "url": "/data/372/",
    "enable_default_schema_extraction": false,
    "jdbc_driver": null,
    "unresolved_mention_fingerprint_method": 0,
    "builtin_datasource": null,
    "otype": "data",
    "can_toggle_ds_privacy": true,
    "exclude_schemas": null,
    "compose_oauth_enabled": false
}

Data Source Update

This API can be used to update an existing data source.

URL

Replace <ds_id> with your data source id.

Data Parameters

1. Cannot update the dbtype if the datasource has been deployed (deployment_setup_complete == True). Certified types are mysql, oracle, postgresql, sqlserver, redshift, teradata and snowflake.

2. Host/Port update only possible if data source is undeployed that is deployment_setup_complete == False.

3. dbname cannot be updated if data source is deployed i.e. deployment_setup_complete == True. Please update the dbname directly in the jdbc uri.

4. title required only if the value was not previously provided when the data source was undeployed i.e. (deployment_setup_complete == False).

Sample Request Body

{
    "uri": "mysql://10.11.21.130:3306/?allowMultiQueries=true",
    "db_username": "updated_username",
    "db_password":"updated_password",
    "title": "updated_title",
    "description": "updated_description"
}

Headers

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

{
    "host": "10.11.21.125",
    "port": 3306,
    "deployment_setup_complete": false,
    "db_username": "username",
    "supports_explain": true,
    "private": false,
    "data_upload_disabled_message": "Your administrator has disabled the Data Uploader for this Data Source.",
    "is_hidden": false,
    "hive_logs_source_type": 0,
    "metastore_uri": null,
    "is_hive": false,
    "title": "title",
    "is_gone": false,
    "is_virtual": false,
    "webhdfs_server": null,
    "supports_qli_diagnostics": false,
    "is_presto_hive": false,
    "latest_extraction_time": null,
    "negative_filter_words": null,
    "has_hdfs_based_qli": false,
    "can_data_upload": true,
    "qualified_name": "",
    "all_schemas": null,
    "has_previewable_qli": true,
    "hive_tez_logs_source": null,
    "has_metastore_uri": false,
    "webhdfs_port": 50070,
    "supports_qli_daterange": true,
    "latest_extraction_successful": false,
    "id": 172,
    "owner_ids": [
        2
    ],
    "dbtype": "mysql",
    "favorited_by_list": null,
    "supports_compose": true,
    "hive_logs_source": null,
    "description": "",
    "enable_designated_credential": false,
    "deleted": false,
    "limit_schemas": null,
    "obfuscate_literals": null,
    "remove_filtered_schemas": false,
    "profiling_tip": null,
    "supports_profiling": true,
    "webhdfs_username": "username",
    "icon": "db",
    "url": "/add_data_source/172/",
    "uri": "mysql://10.11.21.125:3306/?allowMultiQueries=true",
    "otype": "data",
    "exclude_schemas": null
}

Data Source Deletion

This API can be used to delete a data source.

URL

Replace <ds_id> with your data source id.

Headers

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Data Source Configuration Checks

These APIs can be used to run data source configuration checks.

URL

Replace <ds_id> with your data source id.

Headers

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

{
    "status": "ok"
}

Data Source Metadata Extraction

These APIs can be used to get available schemas, trigger metadata extraction and get or update metadata extraction configurations

URL

{
    "schemas": [
      "schema_01",
      "schema_02"
    ]
}

{
    "cron_extraction": "0 4 4 * 2",
    "disable_auto_extraction": false,
    "limit_schemas": [],
    "exclude_schemas": ["abc"],
    "remove_filtered_schemas": false
}

{
    "cron_extraction": "0 4 4 * 2",
    "remove_filtered_schemas": false,
    "limit_schemas": [],
    "exclude_schemas": [
        "abc"
    ],
    "disable_auto_extraction": false
}

{
    "limit_schemas":[],
    "exclude_schemas": ["abc"],
    "remove_filtered_schemas": false
}

{"job_id": 165}