Search code examples
pythonsshavro

How can i read AVRO remotely server with Python


I want to read AVRO file in python , when i do that on my local machine its work great (FastAvro) :

with open('/home/user/file.avro', 'rb') as fo:
avro_reader = reader(fo)
for record in avro_reader:
    avro_records.append(record)

my problem start when i try to make exactly the some but the AVRO file is store on remote server . i dont want to copy the file to the local server. it's possible to read/open the AVRO file on the remote server?

I don't know if this is the right direction but I made a connection to the remote server using paramiko, I manage to find the relevant file but I lack the connection between reading the file on the remote server and reading the AVRO file.

    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect (hostname=myHostname,username=myUsername, password=myPassword)
    sftp = ssh.open_sftp()
    sftp.listdir('/remotedir/avro/')

thanks


Solution

  • You should be able to do the following:

    with sftp.file(path, mode='rb') as fo:
        avro_reader = reader(fo)
        for record in avro_reader:
            avro_records.append(record)