Deno file server hangs on client refresh

I've implemented a deno file server like this:

import { WebServerSettings } from "../types.ts";

import {
    StatusCodes,
    ReasonPhrases
} from "https://deno.land/x/https_status_codes@v1.2.0/mod.ts";

import { mime } from "https://deno.land/x/mimetypes@v1.0.0/mod.ts";

export class WebServer {
    settings:WebServerSettings;

    constructor(settings:WebServerSettings){
        this.settings = settings;
    }

    async start(){
        const server = Deno.listen(this.settings);
        console.log(`Web Server up and running on port ${this.settings.port}`);

        for await (const connection of server){
            this.handle(connection).catch(err => console.error(err));
        }
    }

    private async handle(connection: Deno.Conn){
        const httpConnection = Deno.serveHttp(connection);
        for await (const requestEvent of httpConnection){
            const url = new URL(requestEvent.request.url);
            let filepath = decodeURIComponent(url.pathname);
            
            const root:string = (filepath.match(/\/[^\/]*/) || [""])[0];
            const local = new Map(Object.entries(this.settings.dirs)).get(root) || 
                this.settings.dirs.default + root;

            filepath = local + 
                filepath.replace(/^\/?$/, "/index.html")
                .replace(root, "");

            let file;
            try {
                file = await Deno.open(filepath, { read: true });
            } catch {
                const response = new Response(
                    ReasonPhrases.NOT_FOUND, { status: StatusCodes.NOT_FOUND });
    
                await requestEvent.respondWith(response);
                return;
            }

            const contentType = mime.getType(filepath) || "application/octet-stream";

            await requestEvent.respondWith(
                new Response(file.readable, { 
                    headers: {
                        "Content-Type": contentType
                    }}));
        }
    }
}

It's a slight modification of the deno file server available on deno examples. Essentially allows statically mapping some folders and adds Content-Type header to response. On my browser I enter the following URL http://localhost:8080 and index.html is served correctly the first time:

<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf8">
        <meta lang="en">
        <title>Client</title>
        <script src="js/protobuf.min.js"></script>
    </head>
    <body>
        Hello World
    </body>
</html>

But if I hit the refresh button on browser, the page loading hangs forever and the server doesn't receive any request. But if then I hit stop and refresh again, the page is then loaded. This is ALWAYS reproducible on any browser. The issue is related to this line

<script src="js/protobuf.min.js"></script>

If I comment it out or if I change the src to //cdn.jsdelivr.net/npm/protobufjs@7.X.X/dist/protobuf.js, everything works fine. It seems that the server has troubles serving "bigger" files.

Solution

Short answer

The core problem is that you are "blocking" the AsyncIterable httpConnection by awaiting inside its associated loop:

for await (const requestEvent of httpConnection) {
  // Don't await in this code block
}

Suggestion: Just use the serve function from https://deno.land/std@0.177.0/http/server.ts instead of the low level HTTP APIs — it will help you avoid this and other similarly subtle issues (your code doesn't contain any logic that uses the low level API capabilities anyway, so you don't need it).

Refs:

Reproducible problem and fix

You haven't provided a minimal, reproducible example, but I'll use the code you did show to demonstrate the issue and show how to fix the specific issue that you asked about — in this way, you and everyone else reading this will have everything needed to follow along and reproduce it. Here are the files in the reproduction:

I'll put the files-to-be-served in a ./public directory:

./public/index.html:

<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf8">
        <meta lang="en">
        <title>Client</title>
        <script src="js/protobuf.min.js"></script>
    </head>
    <body>
        Hello World
    </body>
</html>

cdn.jsdelivr.net/npm/protobufjs@7.X.X/dist/protobuf.js

You mentioned the above URL pattern in the question, but didn't show the local file, so I'll download a copy of this file to ./public/js/protobuf.min.js and show its byte length:

% curl -fsSL 'https://cdn.jsdelivr.net/npm/protobufjs@7.2.2/dist/protobuf.min.js' > public/js/protobuf.min.js

% wc -c < public/js/protobuf.min.js
   74208

Here's the code in the module from your question:

./server.ts:

// You didn't show this file in your question,
// so I've created a substitute for it below on line 12:
// import { WebServerSettings } from "../types.ts";

import {
  ReasonPhrases,
  StatusCodes,
} from "https://deno.land/x/https_status_codes@v1.2.0/mod.ts";

import { mime } from "https://deno.land/x/mimetypes@v1.0.0/mod.ts";

export type WebServerSettings = Parameters<typeof Deno.listen>[0] & {
  dirs: Partial<Record<string, string>> & { default: string };
};

export class WebServer {
  settings: WebServerSettings;

  constructor(settings: WebServerSettings) {
    this.settings = settings;
  }

  async start() {
    const server = Deno.listen(this.settings);
    console.log(`Web Server up and running on port ${this.settings.port}`);

    for await (const connection of server) {
      this.handle(connection).catch((err) => console.error(err));
    }
  }

  private async handle(connection: Deno.Conn) {
    const httpConnection = Deno.serveHttp(connection);
    for await (const requestEvent of httpConnection) {
      const url = new URL(requestEvent.request.url);
      let filepath = decodeURIComponent(url.pathname);

      const root: string = (filepath.match(/\/[^\/]*/) || [""])[0];
      const local = new Map(Object.entries(this.settings.dirs)).get(root) ||
        this.settings.dirs.default + root;

      filepath = local +
        filepath.replace(/^\/?$/, "/index.html")
          .replace(root, "");

      let file;
      try {
        file = await Deno.open(filepath, { read: true });
      } catch {
        const response = new Response(
          ReasonPhrases.NOT_FOUND,
          { status: StatusCodes.NOT_FOUND },
        );

        await requestEvent.respondWith(response);
        return;
      }

      const contentType = mime.getType(filepath) || "application/octet-stream";

      await requestEvent.respondWith(
        new Response(file.readable, {
          headers: {
            "Content-Type": contentType,
          },
        }),
      );
    }
  }
}

And here's a module that imports it and starts a server at the details in your question:

http://localhost:8080

./main.ts:

import * as path from "https://deno.land/std@0.177.0/path/mod.ts";
import { WebServer, type WebServerSettings } from "./server.ts";

const defaultDir = path.join(
  path.dirname(path.fromFileUrl(import.meta.url)),
  "public",
);

const settings: WebServerSettings = {
  dirs: { default: defaultDir },
  hostname: "localhost",
  port: 8080,
};

await new WebServer(settings).start();

Now, running this module will reproduce the issue that you described:

% deno --version
deno 1.30.3 (release, aarch64-apple-darwin)
v8 10.9.194.5
typescript 4.9.4

% deno run --allow-net=localhost:8080 --allow-read=. main.ts
Web Server up and running on port 8080

On the first page load, the requested resources are received, as shown in the waterfall and log views of Chrome's dev tools network panel:

However, on a refresh of the page, the request stalls (the server doesn't respond) and the log shows "Pending...":

This is the issue that you described.

Back in the terminal, stop the server with ctrl+c.

In order not to block the request loop, you must not await inside it. One way to refactor your ./server.ts code is to split the handle method into two methods:

export class WebServer {
  settings: WebServerSettings;

  constructor(settings: WebServerSettings) {
    this.settings = settings;
  }

  async start() {
    const server = Deno.listen(this.settings);
    console.log(`Web Server up and running on port ${this.settings.port}`);

    for await (const connection of server) {
      this.handleConn(connection).catch((ex) => console.error(ex));
    }
  }

  private async handleConn(connection: Deno.Conn) {
    const httpConnection = Deno.serveHttp(connection);
    for await (const requestEvent of httpConnection) {
      requestEvent.respondWith(this.handleRequest(requestEvent.request))
        .catch((ex) => console.error(ex));
    }
  }

  private async handleRequest(request: Request): Promise<Response> {
    const url = new URL(request.url);
    let filepath = decodeURIComponent(url.pathname);

    const root: string = (filepath.match(/\/[^\/]*/) || [""])[0];
    const local = new Map(Object.entries(this.settings.dirs)).get(root) ||
      this.settings.dirs.default + root;

    filepath = local +
      filepath.replace(/^\/?$/, "/index.html")
        .replace(root, "");

    let file: Deno.FsFile;
    try {
      file = await Deno.open(filepath, { read: true });
    } catch {
      return new Response(
        ReasonPhrases.NOT_FOUND,
        { status: StatusCodes.NOT_FOUND },
      );
    }

    const contentType = mime.getType(filepath) || "application/octet-stream";

    return new Response(file.readable, {
      headers: {
        "Content-Type": contentType,
      },
    });
  }
}

In the modified code above, the await keyword is never used directly within any of the for await...of loops, which prevents blocking of the AsyncIterators which are responsible for responding to connections and requests.

Now, when the server is run, all network logs look like this, no matter how many times the page is refreshed:

As mentioned at the beginning: no logic in your code uses features of the low level HTTP APIs. You can just use the serve function instead:

./server.ts:

import {
  serve,
  type ServeInit,
} from "https://deno.land/std@0.177.0/http/server.ts";

import {
  ReasonPhrases,
  StatusCodes,
} from "https://deno.land/x/https_status_codes@v1.2.0/mod.ts";

import { mime } from "https://deno.land/x/mimetypes@v1.0.0/mod.ts";

export type WebServerSettings = ServeInit & {
  dirs: Partial<Record<string, string>> & { default: string };
};

export class WebServer {
  settings: WebServerSettings;

  constructor(settings: WebServerSettings) {
    this.settings = settings;
  }

  start() {
    const onListen = this.settings.onListen ??
      (({ port }) => console.log(`Web Server up and running on port ${port}`));
    return serve(this.handleRequest, { ...this.settings, onListen });
  }

  private handleRequest = async (request: Request): Promise<Response> => {
    const url = new URL(request.url);
    let filepath = decodeURIComponent(url.pathname);

    const root: string = (filepath.match(/\/[^\/]*/) || [""])[0];
    const local = new Map(Object.entries(this.settings.dirs)).get(root) ||
      this.settings.dirs.default + root;

    filepath = local +
      filepath.replace(/^\/?$/, "/index.html")
        .replace(root, "");

    let file: Deno.FsFile;
    try {
      file = await Deno.open(filepath, { read: true });
    } catch {
      return new Response(
        ReasonPhrases.NOT_FOUND,
        { status: StatusCodes.NOT_FOUND },
      );
    }

    const contentType = mime.getType(filepath) || "application/octet-stream";

    return new Response(file.readable, {
      headers: {
        "Content-Type": contentType,
      },
    });
  };
}