Alation Open Data Quality Initiative Guide
This is an overview of the Open Data Quality Initiative for integration of data quality/data health data into Alation.
Design Guide
Consumer of the information
- Identify who the primary users of the data catalog are.
- Who would be the primary users of the information that this integration would surface?
- Most of the time, these two audiences will be the same. Sometimes, however, there may be use cases where only a subset of the users actually need to see this information.
Keep the Content Simple
Mind the white space. It is really easy to create walls of text that users of any system will ignore. Things to consider:
- Summarize the data so that only the pertinent information shows.
- Most users want to understand what the data means and quickly understand if they can trust the data.
Points of Integration
Data Health Tab (New)
API based Data Health tab for tables in the Alation Catalog that can surface and display data quality rules from integration partners.
Benefits:
- Public API to allow customers and partners to integrate Data Quality tools with the Data Catalog.
- Establishes a framework for deeper and more robust integration with the catalog in future phases.
![Screen Shot 2022-10-04 at 5.01.05 PM.png 1129](https://files.readme.io/426a0ad-Screen_Shot_2022-10-04_at_5.01.05_PM.png)
Data Quality Overview Using Custom Fields
Custom fields provide space for integration partners to load summary information for each object (schema, tables, columns). These fields are Rich Text, so data can be set up to display the summary with visualizations. Additionally, hyperlinks can be added to allow users to easily access the details found in the 3rd party tools.
![Screen Shot 2022-10-04 at 5.01.27 PM.png 1153](https://files.readme.io/2391917-Screen_Shot_2022-10-04_at_5.01.27_PM.png)
API Details
Data Health API Definition
The latest API Definition can be found here: https://developer.alation.com/dev/reference/data_health_ingestion
Example Code
Example Code for Data Health API
Python Code
import requests
import json
url = 'https://{base-url}/integration/v1/data_quality/'
headers = {'Token': '<Access Token>', 'Content-Type': 'application/json'}
myobj = {
"fields": [
{
"field_key": "my.example.rule.key",
"name": "The Example Rule",
"description": "Description",
"type": "STRING"
}
],
"values": [
{
"field_key": "my.example.rule.key",
"object_key": "<ds_id>.<schema>.<table>.<column>",
"object_type": "ATTRIBUTE",
"status": "GOOD",
"value": 25,
"url": "https://www.alation.com"
} ]
}
x = requests.post(url, data = json.dumps(myobj), headers=headers)
print(x.text)
Resulting UI
![Screen Shot 2022-10-04 at 5.01.36 PM.png 1613](https://files.readme.io/afe13ba-Screen_Shot_2022-10-04_at_5.01.36_PM.png)
Example Code for Custom Field
Please refer to the Systems Integrations to Alation with Custom Templates and Fields guide.
Updated 11 months ago