I have this go code which walks a directory file tree and produces MD5 hashes of every file in it and writes the result in an output file.
package main
import (
"crypto/md5"
"encoding/hex"
"fmt"
"io"
"os"
"path/filepath"
"sync"
)
func main() {
filePath := os.Args[1]
output := os.Args[2]
wg := &sync.WaitGroup{}
err := filepath.Walk(filePath, func(path string, info os.FileInfo, err error) error {
if !info.IsDir() {
wg.Add(1)
go func(path string) {
md5Sum, _ := md5File(path)
if err := writeToFile(path, md5Sum, output); err != nil {
panic(err)
}
wg.Done()
}(path)
}
return nil
})
if err != nil {
panic(err)
}
wg.Wait()
}
func md5File(filePath string) (string, error) {
file, err := os.Open(filePath)
if err != nil {
return "", err
}
defer file.Close()
hash := md5.New()
if _, err := io.Copy(hash, file); err != nil {
return "", err
}
checksum := hash.Sum(nil)
return string(hex.EncodeToString(checksum)), nil
}
func writeToFile(filePath, md5sum, output string) error {
file, err := os.OpenFile(output, os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0755)
if err != nil {
return err
}
defer file.Close()
file.WriteString(fmt.Sprintf("%s %s\n", md5sum, filePath))
return file.Sync()
}
From my understanding it is bound to run into a race condition sometime upon writing to the output file but it never does. I mean I have executed this code countless times never facing any issues. It even produces the same result every time.
Why is that? Am I missing something?
UPDATE: When I say it is bound to face a race condition I mean when running multiple goroutines it is possible for more than one goroutine to want to write to file at the same time.
When I say it is bound to face a race condition I mean when running multiple goroutines it is possible for more than one goroutine to want to write to file at the same time.
It is not a problem to open a file multiple times. And since you explicitly use O_APPEND
the writes don't interact badly with each other. To cite
from man open:
O_APPEND ...
Before each write(2), the file offset is positioned at the end of the file, as if with lseek(2). The modification of the file offset and the write operation are performed as a single atomic step.