Using Python, I want to copy files that match a pattern sample1
from AWS S3 to FTP server directly without any downloads to local temporary location.
I attempted the following:
import s3fs
from ftplib import FTP_TLS
s3 = s3fs.S3FileSystem(anon=False)
pattern = 'sample1'
rest = [i for i in list if pattern in i]
rest
ftp = FTP_TLS("ftp.test.com")
ftp.login(user ='myUser', passwd = 'PassWrd')
ftp.cwd("box_dest")
for f in rest:
print(f)
with open(f, 'r') as fu:
ftp.storbinary('STOR ' + f, fu)
I'm getting:
[u'test-bucket/abc/test/sample1.csv']
test-bucket/abc/test/sample1.csv
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
IOError: [Errno 2] No such file or directory: u'test-bucket/abc/test/sample1.csv'
Any suggestions on how I can achieve this? Thanks!
To read the file from S3, you need to use S3FileSystem.open
, not os.open
.
And you need to extract just a filename from the original S3 path, when specifying the target FTP path. posixpath.basename
should do.
for f in rest:
print(f)
with s3.open(f, 'r') as fu:
ftp.storbinary('STOR ' + posixpath.basename(f), fu)