How to transfer multiple files using go

Question

I am trying to write a program in go which has two parts. One part is the client who tries to upload multiple pictures to the other part the server.

The server side should do the following:

1. Get the number of files which will be send
2. Loop for every file
3. Get filename
4. Get the file and save it
5. Go to 3

So far the server side is doing the following:

func getFileFromClient(connection net.Conn) {
    var numberOfPics int
    var err error
    var receivedBytes int64
    var fileName string
    r := bufio.NewReader(connection)
    strNumberOfPics, err := r.ReadString('\n')
    if err != nil {
        fmt.Printf("Error reading: %s\n", err)
        return
    }
    fmt.Printf("Read: %s\n", strNumberOfPics)
    strNumberOfPics = strings.Trim(strNumberOfPics, "\n")
    numberOfPics, err = strconv.Atoi(strNumberOfPics)
    if err != nil {
        fmt.Printf("Error Atoi: %s\n", err)
        panic("Atoi")
    }
    fmt.Printf("Receiving %d pics:\n", numberOfPics)
    for i := 0; i < numberOfPics; i++ {
        // Getting the file name:
        fileName, err = r.ReadString('\n')
        if err != nil {
            fmt.Printf("Error receiving: %s\n", err)
        }
        fmt.Printf("Filename: %s\n", fileName)
        fileName = strings.Trim(fileName, "\n")
        f, err := os.Create(fileName)
        defer f.Close()
        if err != nil {
            fmt.Println("Error creating file")
        }
        receivedBytes, err = io.Copy(f, connection)
        if err != nil {
            panic("Transmission error")
        }

        fmt.Printf("Transmission finished. Received: %d \n", receivedBytes)
    }
}

io.Copy is working for just one file and nothing additional (because it does not empty the queue I think). I do not want to reconnect every time for every file if I do not have too. But I am not sure what I actually can do about that.

Has anyone any suggestions of an existing package or method which could help? Or example code? Or am I just plain wrong and it is a bad idea to even try this with go?

I think it might be enough if the server is able to flush the connection buffer after every read so no additional info is read and/or copied.

Really looking forward for help, thanks in advance

EDIT: Updated Code still not working. I think it might be the bufio.reader

func getFileFromClient(connection net.Conn) {
    var numberOfPics int
    var err error
    var receivedBytes int64
    var fileName string
    r := bufio.NewReader(connection)
    strNumberOfPics, err := r.ReadString('\n')
    if err != nil {
        fmt.Printf("Error reading: %s\n", err)
        return
    }
    strNumberOfPics = strings.Trim(strNumberOfPics, "\n")
    numberOfPics, err = strconv.Atoi(strNumberOfPics)
    if err != nil {
        fmt.Printf("Error Atoi: %s\n", err)
        panic("Atoi")
    }
    fmt.Printf("Receiving %d pics:\n", numberOfPics)
    for i := 0; i < numberOfPics; i++ {
        // Getting the file name:
        fileName, err = r.ReadString('\n')
        if err != nil {
            fmt.Printf("Error receiving: %s\n", err)
        }
        fileName = strings.Trim(fileName, "\n")
        fmt.Printf("Filename: %s\n", fileName)
        f, err := os.Create(fileName)
        defer f.Close()
        if err != nil {
            fmt.Println("Error creating file")
        }
        // Get the file size
        strFileSize, err := r.ReadString('\n')
        if err != nil {
            fmt.Printf("Read size error %s\n", err)
            panic("Read size")
        }
        strFileSize = strings.Trim(strFileSize, "\n")
        fileSize, err := strconv.Atoi(strFileSize)
        if err != nil {
            fmt.Printf("Error size Atoi: %s\n", err)
            panic("size Atoi")
        }
        fmt.Printf("Size of pic: %d\n", fileSize)

        receivedBytes, err = io.CopyN(f, connection, int64(fileSize))
        if err != nil {
            fmt.Printf("Transmission error: %s\n", err)
            panic("Transmission error")
        }
        fmt.Printf("Transmission finished. Received: %d \n", receivedBytes)
    }
}

EDIT 2: I did not get this solution to work. I am pretty sure it is because I used bufio. I did however get it to work by transmitting a single zip file with io.copy. Another solution which worked was to transmit a zip file by using http. If you are stuck trying something similar and need help feel free to send me a message. Thanks to all of you for your help

Answer 1

Keeping your implementation so far, the thing you're missing is that io.Copy() reads from source until it finds an EOF, so it will read all the remaining images in one go.

Also, the client must send, for each image, its size in bytes (you could do that after sending the name). In the server, just read the size and then use io.CopyN() to read that exact number of bytes.

EDIT: as a matter of fact, you could also do things like you were doing and send images in parallel instead of serially, that would mean you open a new connection for each file transfer and then read all of the file withouth needing to send the amount of images or their size.

In case you want an alternative, a good option would be using good 'ol HTTP and multipart requests. There's the built-in module mime/multipart that allows you to do file transfers over HTTP. Of course, that would mean you'd have to rewrite your program.