"http: wrote more than the declared Content-Length" error in Go

Question

I'm trying out Go and wrote an application that manages a queue of HTTP requests to be processed by worker goroutines.

The concurrency stuff seem to work fine, but I'm getting this error when sending a response back http: wrote more than the declared Content-Length.

Here's the full code:

package main

import (
    "log"
    "net/http"
    "sync"
)

// Job represents a unit of work to be processed by a worker.
type Job struct {
    r *http.Request       // HTTP request to be processed
    w http.ResponseWriter // Response writer to send the result
}

// Queue manages a list of jobs to be processed by workers.
type Queue struct {
    jobs []*Job     // List of jobs in the queue
    mu   sync.Mutex // Mutex to synchronize access to the queue
    cond *sync.Cond // Condition variable for signaling
}

var q Queue // Global instance of the queue
var WORKER_POOL_SIZE = 4 // Number of workers

// Push adds a job to the queue.
func (q *Queue) Push(j *Job) {
    q.mu.Lock()
    defer q.mu.Unlock()
    q.jobs = append(q.jobs, j)
    q.cond.Signal() // Signal a waiting worker that a job is available
    log.Println("Job added to queue")
}

// Pop retrieves and removes a job from the queue.
func (q *Queue) Pop() (*Job, bool) {
    q.mu.Lock()
    defer q.mu.Unlock()
    if len(q.jobs) == 0 {
        q.cond.Wait() // If the queue is empty, wait for a signal
    }
    job := q.jobs[0]
    q.jobs = q.jobs[1:]
    log.Println("Job removed from queue")
    return job, true
}

// handler adds a job to the queue.
func handler(w http.ResponseWriter, r *http.Request) {
    // Create a job with the request and response.
    job := &Job{r, w}

    // Push the job onto the queue.
    q.Push(job)
    log.Println("Received request and added job to queue")
}

// init initializes the condition variable and starts worker goroutines.
func init() {
    q.cond = sync.NewCond(&q.mu)
    for i := 0; i < WORKER_POOL_SIZE; i++ {
        go worker()
    }
}

// worker processes jobs from the queue.
func worker() {
    for {
        job, ok := q.Pop()
        if ok {
            log.Println("Worker processing job")
            doWork(job)
        }
    }
}

// doWork simulates processing a job and sends a response.
func doWork(job *Job) {
    // Extract the "Name" parameter from the request query.
    name := job.r.URL.Query().Get("Name")

    // Check if the name is not empty.
    if name != "" {
        // Send the name as the response.
        _, err := job.w.Write([]byte("Hello, " + name))
        if err != nil {
            log.Println("Error writing response:", err)
        }
        log.Println("Response sent: Hello,", name)
    } else {
        // If the "Name" parameter is missing or empty, send an error response.
        http.Error(job.w, "Name parameter is missing or empty", http.StatusBadRequest)
        log.Println("Error: Name parameter is missing or empty")
    }
}

func main() {
    http.HandleFunc("/addJob", handler)
    log.Println("Server started and listening on :8080")
    err := http.ListenAndServe(":8080", nil)
    if err != nil {
        log.Fatal("Error starting server:", err)
    }
}

Any idea on how to solve this issue? Also, since I'm new to the concurrency stuff in go, do you think it could be improved? Thanks!

Answer 1

As per the http.Handler docs:

Returning signals that the request is finished; it is not valid to use the ResponseWriter or read from the Request.Body after or concurrently with the completion of the ServeHTTP call.

Your handler is pushing the request/writer onto a queue and then returning. This means that you are attempting to write to the ResponseWriter after the handler returned in violation of the above (as there is no synchronisation, it's also possible that the write actually occurs before, or concurrently to, the return).

There are many ways you could resolve this; one technique being:

// Job represents a unit of work to be processed by a worker.
type Job struct {
    r    *http.Request       // HTTP request to be processed
    w    http.ResponseWriter // Response writer to send the result
    done chan struct{}       // Closed when write competed
}

...

func handler(w http.ResponseWriter, r *http.Request) {
    // Create a job with the request and response.
    job := &Job{r, w, make(chan struct{})}

    // Push the job onto the queue.
    q.Push(job)
    log.Println("Received request and added job to queue")
    <-job.done // Wait until job has been processed
}

...

// worker processes jobs from the queue.
func worker() {
    for {
        job, ok := q.Pop()
        if ok {
            log.Println("Worker processing job")
            doWork(job)
            close(job.done) // Notify requester that we are done
        }
    }
}

do you think it could be improved?

That really depends upon the requirements. A common solution would be to just use a channel (the handler sends the request to a channel, and multiple workers receive from same channel).