Search code examples
pythonaws-lambdaamazon-redshiftboto3

Lambda querying Redshift: Nothing Happening


I'm having difficulty finding out why there's no changes/actions happening on my redshift cluster even I'm getting a response like this in Cloudwatch.

{'ClusterIdentifier': 'redshift-chan-dev', 'CreatedAt': datetime.datetime(2022, 12, 28, 8, 57, 9, 208000, tzinfo=tzlocal()),'Database': 'dev', 'Id': '5d5c7b4c-31e6-41c3-991e-f76383d46775','SecretArn': 'REDSHIFT/dev/chan', 'ResponseMetadata': {'RequestId':'5d5c7b4c-31e6-41c3-991e-f76383d46775', 'HTTPStatusCode': 200,'HTTPHeaders': {'x-amzn-requestid': '5d5c7b4c-31e6-41c3-991e-f76383d46775', 'content-type':'application/x-amz-json-1.1', 'content-length': '163', 'date': 'Wed, 28 Dec 202208:57:09 GMT'}, 'RetryAttempts': 0}}

boto3 'redshift-data' api execution code:

response = REDSHIFT_CLIENT.batch_execute_statement( \
        Database = "dev", \
        SecretArn = 'REDSHIFT/dev/chan', \
        ClusterIdentifier = 'redshift-chan-dev', \
        Sqls = [
            drop_tmp_table,
            create_tmp_table,
            copy_to_tmp_table,
            load_tmp_to_winloss,
        ]
    )

QUERIES

create_tmp_table

create_tmp_table = """CREATE TABLE dev.schema.table( \
id INT, \
transaction_code VARCHAR, \
"type" VARCHAR, \
org_id SMALLINT, \
region VARCHAR, \
..

copy_to_tmp_table

copy_to_tmp_table = """COPY dev.schema.table \
FROM 's3://{bucket}/{TMP_PREFIX}/{file_name}.csv' \
iam_role 'arn:aws:iam::560762403195:role/redshift_copyS3' \
FORMAT AS CSV \
IGNOREHEADER 1"""

Solution

  • Solved the issue by adding describe_statement() call with my lambda function. It shows the "Error" happening when trying to process the sql statements in redshift-data api.

    Apparently, it's with the Secret ARN I am using. Should make sure that the credentials of ARN is same with the user I have in the redshift cluster

    Thanks for the help @Bill Weiner