access denied on CopyObject

0

I have written a simple script to move some redshift logs to a different location as they are written to the default location. When I test the script by copying the logs into the directory manually it works. However, when redshift dumps the logs into the directory my script fails with this:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

So I know there must be something different between how redshift writes the logs and how I write them but I can't for the life of me figure out what that difference is. Any help?

Here is my code. I've set this up with a role that has full s3 permissions. It's pretty straightforward but I can't figure out why it's failing outside of "it's permissions related" but I can't figure out why or what to change.

import boto3

def lambda_handler(event, context):
    s3 = boto3.resource('s3')

    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']
        log_file = key.split('/')[-1]
        log_type = log_file.split('_')[-2]
        log_env = log_file.split('_')[-3]
        new_key = 'redshift/{}/{}/{}'.format(log_type, log_env, log_file)
        print(key)
        print(new_key)
        
        s3.Object(bucket, new_key).copy_from(CopySource={'Bucket': bucket, 'Key': key})
        s3.Object(bucket, key).delete()
neufpas
asked 7 years ago4723 views
2 Answers
0

I'm completely stuck here. I just gave my bucket full public permissions and it's still failing with Access Denied.

The policy attached to my job grants "s3:*" permissions and I know it works because if I drop a file in the bucket it copies correctly.

This problem only exists with the files created by redshift.

Any help?

neufpas
answered 7 years ago
0

I figured out the problem. It ended up being related to a bad key. It turns out that the key string passed in by the s3 event trigger is urlencoded. These urlencoded strings appear as normal strings when the error message appears in the lambda window but adding debug logging to my script let me see the actual string.

I'm not entirely sure why this was causing an Access Denied error, but adding a call to urlib.parse.unquote fixed the issue.

neufpas
answered 7 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions