I want to read AVRO file in python , when i do that on my local machine its work great (FastAvro) :
with open('/home/user/file.avro', 'rb') as fo:
avro_reader = reader(fo)
for record in avro_reader:
avro_records.append(record)
my problem start when i try to make exactly the some but the AVRO file is store on remote server . i dont want to copy the file to the local server. it's possible to read/open the AVRO file on the remote server?
I don't know if this is the right direction but I made a connection to the remote server using paramiko, I manage to find the relevant file but I lack the connection between reading the file on the remote server and reading the AVRO file.
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect (hostname=myHostname,username=myUsername, password=myPassword)
sftp = ssh.open_sftp()
sftp.listdir('/remotedir/avro/')
thanks
You should be able to do the following:
with sftp.file(path, mode='rb') as fo:
avro_reader = reader(fo)
for record in avro_reader:
avro_records.append(record)