I'm having difficulty finding out why there's no changes/actions happening on my redshift cluster even I'm getting a response like this in Cloudwatch.
{'ClusterIdentifier': 'redshift-chan-dev', 'CreatedAt': datetime.datetime(2022, 12, 28, 8, 57, 9, 208000, tzinfo=tzlocal()),'Database': 'dev', 'Id': '5d5c7b4c-31e6-41c3-991e-f76383d46775','SecretArn': 'REDSHIFT/dev/chan', 'ResponseMetadata': {'RequestId':'5d5c7b4c-31e6-41c3-991e-f76383d46775', 'HTTPStatusCode': 200,'HTTPHeaders': {'x-amzn-requestid': '5d5c7b4c-31e6-41c3-991e-f76383d46775', 'content-type':'application/x-amz-json-1.1', 'content-length': '163', 'date': 'Wed, 28 Dec 202208:57:09 GMT'}, 'RetryAttempts': 0}}
boto3 'redshift-data' api execution code:
response = REDSHIFT_CLIENT.batch_execute_statement( \
Database = "dev", \
SecretArn = 'REDSHIFT/dev/chan', \
ClusterIdentifier = 'redshift-chan-dev', \
Sqls = [
drop_tmp_table,
create_tmp_table,
copy_to_tmp_table,
load_tmp_to_winloss,
]
)
QUERIES
create_tmp_table
create_tmp_table = """CREATE TABLE dev.schema.table( \
id INT, \
transaction_code VARCHAR, \
"type" VARCHAR, \
org_id SMALLINT, \
region VARCHAR, \
..
copy_to_tmp_table
copy_to_tmp_table = """COPY dev.schema.table \
FROM 's3://{bucket}/{TMP_PREFIX}/{file_name}.csv' \
iam_role 'arn:aws:iam::560762403195:role/redshift_copyS3' \
FORMAT AS CSV \
IGNOREHEADER 1"""
Solved the issue by adding describe_statement() call with my lambda function. It shows the "Error" happening when trying to process the sql statements in redshift-data api.
Apparently, it's with the Secret ARN I am using. Should make sure that the credentials of ARN is same with the user I have in the redshift cluster
Thanks for the help @Bill Weiner