Query Log Upload (Push QLI) API

Description

This API allows admin users to send query events for ingestion in Alation.

URL

POST /api/v1/bulk_metadata/query_log_ingestion/**<ds_id>**/

Replace <ds_id> with the id of the data source for which the query events are to be ingested.

Data Parameters

Name	Type	Required	Description
defaultDatabases	string	Yes	Default database used in the query. If no database is found, use an empty string ("").
sessionId	string	Yes	ID of the session.
startTime	string	Yes	The starting time of the query. The format for this parameter is `yyyy-MM-DD HH:MM:SS.SSS` For Example: {"startTime":"2017-10-11 10:21:22.23445"}
text	string	Yes	Text of the query. For Example: {"text":"select * from emp_schema.emp_table"}
userName	string	Yes	Name of the database account user.
appId	string	No	ID of the application.
clientId	string	No	ID of the client.
clientAddr	string	No	IP address of the client. For Example: {"clientAddr":"10.0.0.1:8000"}
delayTime	float	No	Number of seconds the query was delayed. NOTE: This parameter is available in Teradata.
destinationTable	string	No	This parameter represents the target table to move the results of SELECT query. NOTE: Currently this parameter can only be used for Google BigQuery, and only for customers on Alation's cloud-native architecture who have upgraded to QLI 2.0.
numResultRows	long	No	Number of rows in the result.
secondsTaken	long	No	Total time elapsed (time taken to execute the query in seconds).
sessionStartTime	string	No	The starting time of the session. The format for this parameter is `yyyy-MM-DD HH:MM:SS.SSS` For Example: {"sessionStartTime":"2017-10-11 10:21:22.23445"}
totalIoCount	int	No	The count of IO operation.
totalCpuTime	float	No	This parameter represents the AMPCPUTIME in seconds.

Sample Request Body

{"clientAddr": "10.11.0.75:8000", "startTime": "2017-05-28 05:22:24.457416", "defaultDatabases": "test_db", "text":  "SELECT * from schema_1.table_1", "sessionId": "sess1234", "userName": "test_user", "secondsTaken": 1}
{"clientAddr": "10.11.0.75:8000", "startTime": "2017-05-29 05:22:37.457316", "defaultDatabases": "test_db", "text": "SELECT * from schema_2.table_2", "sessionId": "sess432", "userName": "test_user", "secondsTaken": 2}
{"clientAddr": "10.11.0.75:8000", "startTime": "2017-05-29 05:22:43.457116", "defaultDatabases": "test_db", "text": "SELECT * from schema_3.table_3", "sessionId": "sess5678", "userName": "test_user", "secondsTaken": 5}

NOTE: The Request body can contain one or more JSON objects, each on a new line. Each line represents a query event.

Headers

HTTP Header	Value
TOKEN	<your_token>

Replace <your_token> with API Token which can be obtained from getToken API call (Get API Token).

Success Response

Content-Type: application/json

Status: 200 OK

Body

{
    "num_statements_input": 3,
    "job_id": 1
}

Name	Description
num_statements_input	Represents the num of statements in the request body.
job_id	Unique identifier of the job that is triggered after a success call to the API. This job is responsible to ingest the query events mentioned in the request body. To know the status of a job, please refer Job Status API. The job status can also be found under the 'Query Log Ingestion' tab (https://<your_host>/data/<ds_id>/settings/import_queries/) of the data source.

Error Response

Invalid Token

Status: 401 UNAUTHORIZED

Body

{
   "detail": "Authentication failed"
}

Missing Token Header

Status: 401 UNAUTHORIZED

Body

{
    "detail": "Authentication credentials were not provided."
}

Non-admin user access

Status: 403 FORBIDDEN

Body

{
    "detail": "Permission denied"
}

Code Samples

cURL

#!/bin/bash

# This is an example token. Please replace this with your token.
API_TOKEN="2abcd-4c04-4c21-8692-eda27a877f90"

BASE_URL="https://alation.yourcompany.com/api/v1/bulk_metadata/query_log_ingestion"

# Ingest queries into a data source with unique identifier, 1
curl -X POST "${BASE_URL}/1/" -H 'content-type: application/json' -H "TOKEN: ${API_TOKEN}" --data-binary @body.txt

# contents of body.txt
: '
{"clientAddr": "10.11.0.75:8000", "startTime": "2017-05-28 05:22:24.457416", "defaultDatabases": "test_db","text":  "SELECT * from schema_1.table_1", "sessionId": "sess1234", "userName": "test_user", "secondsTaken": 1}
{"clientAddr": "10.11.0.75:8000", "startTime": "2017-05-29 05:22:37.457316", "defaultDatabases": "test_db", "text": "SELECT * from schema_2.table_2", "sessionId": "sess432", "userName": "test_user", "secondsTaken": 2}
{"clientAddr": "10.11.0.75:8000", "startTime": "2017-05-29 05:22:43.457116", "defaultDatabases": "test_db", "text": "SELECT * from schema_3.table_3", "sessionId": "sess5678", "userName": "test_user", "secondsTaken": 5}
'

Python

import requests
import json

# This is an example token. Please replace this with your token.
headers = {'Token': '2abcd-4c04-4c21-8692-eda27a877f90'}

data = '{"clientAddr": "10.11.0.75:8000", "startTime": "2017-05-28 05:22:24.457416", "defaultDatabases": "test_db","text":  "SELECT * from schema_1.table_1", "sessionId": "sess1234", "userName": "test_user", "secondsTaken": 1}\n{"clientAddr": "10.11.0.75:8000", "startTime": "2017-05-29 05:22:37.457316", "defaultDatabases": "test_db", "text": "SELECT * from schema_2.table_2", "sessionId": "sess432", "userName": "test_user", "secondsTaken": 2}\n{"clientAddr": "10.11.0.75:8000", "startTime": "2017-05-29 05:22:43.457116", "defaultDatabases": "test_db", "text": "SELECT * from schema_3.table_3", "sessionId": "sess5678", "userName": "test_user", "secondsTaken": 5}\n'

# ingest query events
response = requests.post('https://alation.yourcompany.com/api/v1/bulk_metadata/query_log_ingestion/1/', data=data, headers=headers)
job_details = json.loads(response.text)
print ("Job ID: %s" % (job_details['job_id']))

# check the job status
job_id = job_details['job_id']
response = requests.get('https://alation.yourcompany.com/api/v1/bulk_metadata/job?id=' + str(job_id), headers=headers)
job = json.loads(response.text)
print ("Job Status: %s" % (job['status']))