Throttling
Best Practices
When you make Create, Read, Update, or Delete API calls to Alation, we recommend that:
- Payloads contain 1000 objects or less per call
- You do not use multi-threading or multi-processing API calls
It is possible to rate-limit requests made by users to some of Alation's public APIs. The APIs that can be throttled are:
- BI Source (GBMv2) API
- Data Dictionary API
- Documents API
- Folders API
- Lineage API
- Relational Integration API
- SCIM API
- Search API
Throttling is disabled by default. Instructions for enabling and configuring throttling can be found below.
Enable API Throttling
To enable API Throttling for an Alation instance, set the alation_conf parameter alation.api.throttling.enabled
to True
. No restart is required.
alation_conf alation.api.throttling.enabled -s True
For help with alation_conf, see Using alation_conf.
Server Configuration for Alation Cloud Service
Alation Cloud Service customers can request server configuration changes through Alation Support.
After enabling the feature, configure the throttling scope parameters as explained below.
Configure API Throttling
To view all currently available configuration options for API throttling:
-
On the Alation host, enter the Alation shell:
sudo /etc/init.d/alation shell
-
Type the following command and press Enter:
alation_conf throttling
This command prints all parameters relevant to API throttling and their current values:
Name | Default | Description |
---|---|---|
alation.api.throttling.enabled | false | Enables the throttling feature. No restart required. |
alation.api.throttling.global.read | null | Sets the global API rate limit for READ request types. Global settings apply to those APIs which respect throttling, not all available Alation APIs. null means unlimited. |
alation.api.throttling.global.write | null | Sets the global API rate limit for WRITE request types. Global settings apply to those APIs which respect throttling, not all available Alation APIs. null means unlimited. |
alation.api.throttling.bi.read | 20/sec | Sets the rate limit for READ request types specifically for the BI Server (GBMv2) API. |
alation.api.throttling.bi.write | 20/sec | Sets the rate limit for WRITE request types specifically for the BI Server (GBMv2) API. |
alation.api.throttling.data_dictionary.read | 20/sec | Sets the rate limit for READ request types specifically for the Data Dictionary API. |
alation.api.throttling.data_dictionary.read | 20/sec | Sets the rate limit for WRITE request types specifically for the Data Dictionary API. |
alation.api.throttling.document_public_apis.read | null | Sets the rate limit for READ request types specifically for the Documents API. |
alation.api.throttling.document_public_apis.read | 100/minute | Sets the rate limit for WRITE request types specifically for the Documents API. |
alation.api.throttling.folder_public_apis.read | null | Sets the rate limit for READ request types specifically for the Folders API. |
alation.api.throttling.folder_public_apis.read | 100/minute | Sets the rate limit for WRITE request types specifically for the Folders API. |
alation.api.throttling.lineage.read | 20/sec | Sets the rate limit for READ request types specifically for the Lineage v2 API. |
alation.api.throttling.lineage.write | 20/sec | Sets the rate limit for WRITE request types specifically for the Lineage v2 API |
alation.api.throttling.rdbms_integration_apis.read | 20/sec | Sets the rate limit for READ request types specifically for the Relational Integration API. |
alation.api.throttling.rdbms_integration_apis.write | 20/sec | Sets the rate limit for WRITE request types specifically for the Relational Integration API. |
alation.api.throttling.scim_v2.read | 20/sec | Sets the rate limit for READ request types specifically for the SCIM API. |
alation.api.throttling.scim_v2.write | 20/sec | Sets the rate limit for WRITE request types specifically for the SCIM API. |
alation.api.throttling.search.read | 20/sec | Sets the rate limit for READ request types specifically for the search API. |
Set Throttling Scope Parameters
There are two major throttle scope types:
Global (Sustained)
- Applies to all APIs which respect throttling.
- It is recommended to set the rate limit to a larger value. For example: 1,000 requests per day.
- Set using the parameters:
alation.api.throttling.global.read
alation.api.throttling.global.write
API Family-Specific
- Applies to the particular API scope only (for example, Lineage V2 or GBM V2).
- It is recommended to set the rate limit to a small duration to limit expensive operations or throttle DOS attacks. For example, 20 requests per second.
Each scope type comes with two variations:
- READ: applies to GET requests. For example:
alation.api.throttling.lineage.read = 20/sec
- WRITE: applies to POST, PUT, PATCH, and DELETE requests. For example:
alation.api.throttling.lineage.write = 20/sec
Precedence
Throttle configurations are applied based on scope and request type. The API family scope takes precedence over the Global settings when both are set.
Value Format
Rate limit can be set in the following format:
- Number of requests per second: <number>/sec.
- Example:
20/sec
- Example:
- Number of requests per minute: <number>/min.
- Example:
20/min
- Example:
- Number of requests per hour: <number>/hour.
- Example:
20/hour
- Example:
- Number of requests per day: <number>/day.
- Example:
20/day
- Example:
null
: means unlimited, or throttling disabled. You can disable each rate limit by setting it tonull
. The global rate limit is disabled by default.
To set a value, use the alation_conf command:
alation_conf alation.api.throttling.lineage.read -s <value>
Server Configuration for Alation Cloud Service
Alation Cloud Service customers can request server configuration changes through Alation Support.
Throttle Response
If a rate limit for a scope is set to <number>/<time unit>, then an API request within this scope is throttled on <number> + 1 attempt after the time = <time unit>. When a request is throttled, the API responds with status 429 and the time in seconds till the next request in this scope is allowed.
Example 1
[
{
"code": "429000",
"detail": "Rate limit for [read] requests for API scope [lineage] reached. Expected available in 29 seconds."
}
]
Example 2
[
{
"code": "429000",
"detail": "Global rate limit for [read] requests reached. Expected available in 86279 seconds."
}
]
Updated 5 months ago