Search code examples
pythonamazon-web-servicesamazon-s3boto3smart-open

use boto3 session when opening s3 url


I see a lot of code that uses an S3 bucket url to open a file. I would like to use smart open to open a compressed file

session = boto3.Session(ID, pass)
file = open("s3://bucket/file.txt.gz", transport_params=dict(session=session), encoding="utf-8")

However, all the examples I see about smart open and other pulls from boto3 using a url never specify how to use the session when pulling the data from the url only when pushing data to a new bucket. Is there a way to use the url and the session without needing to create a client from my session and access the bucket and key?


Solution

  • As mentioned you just need to replace "wb" with "rb". I was mistaken that this didn't work

    from smart_open import open
    import boto3
    
    url = 's3://bucket/your/keyz'
    session = boto3.Session(aws_access_key_id,
                            aws_secret_access_key,
                            region_name)
    
    with open(url, 'rb', transport_params={'client': session.client('s3')}) as fin:
        file = fin.read()
    
    print(file)