AWS Rest API for Glue
This is a small script to help access AWS Glue using Rest API. AWS Glue is a serverless data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development.
Using AWS API through Rest is very powerful but at the same time, it's hard to do due to a lack of detailed examples. This is why I am posting here how I managed to get the Glue Get Databases command in Postman.
Here is the official AWS documentation for this call: https://docs.aws.amazon.com/glue/latest/webapi/API_GetDatabases.html
After a lot of trial and error, I have found out that the following Headers are important
POST / HTTP/1.1
Host: glue.us-west-2.amazonaws.com
X-Amz-Target: AWSGlue.GetDatabases
Content-Type: application/x-amz-json-1.1
X-Amz-Date: 20180425T052803Z
Authorization: AWS4-HMAC-SHA256 Credential=AWS_KEY/20180425/us-west-2/glue/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=f6dfb78d7ee837dc57ce6e13d9fd6ec0631d0c5546f0142ce997bc7e9203c7b7
Also, something important is to pass an empty body ({}) if you do not want to specify any of the parameters.
curl — location — request POST ‘https://glue.us-west-2.amazonaws.com/databases?CatalogId=XXXXXXXXXXXX' \
— header ‘X-Amz-Target: AWSGlue.GetDatabases’ \
— header ‘Content-Type: application/x-amz-json-1.1’ \
— header ‘X-Amz-Content-Sha256: mplamplamplampla’ \
— header ‘X-Amz-Date: 20230218T084937Z’ \
— header ‘Authorization: AWS4-HMAC-SHA256 Credential=AWS_KEY/20230218/us-west-2/glue/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-target, Signature=123123123123123123123123123’ \
— data-raw ‘{}’Details to setup Postman
1. In the Params Section add your CatalogId value
2. In the Auth Section Select Type: AWS Signature and fill in your Access Key, Secret Key, and Region
3. In the Headers Section set up X-Amz-Target, Content-Type and X-Amz-Date as above and in the
4. In the Body Section select raw and put empty curly braces ({}) in the body
Similarly, we can use the above procedure to access all the other AWS APIs.