How to add web-tree-sitter to a NextJS project?

Question

I want to be able to generate ASTs by parsing code typed on my NextJS web app. I saw that I should be using web-tree-sitter to achieve such a thing.

I successfully used Docker to generate the tree-sitter-javascript.wasm file, moved it to be in the same directory as my Page.tsx file (where I want to call the parsing function), and loading the parser like this: (as stated on their npm package's site).

     locateFile(scriptName: string, scriptDirectory: string) {
       return scriptName;
     },
   });

And then defining the parser and the JavaScript grammar like this, just like it says

  const JavaScript = await Parser.Language.load('tree-sitter-javascript.wasm');
  parser.setLanguage(JavaScript);

I am getting this error:

./node_modules/web-tree-sitter/tree-sitter.js:1:1044
Module not found: Can't resolve 'fs'

I also read on the npm package's site that adding a webpack config that makes the bundler ignore Node's fs which doesn't exist in the browser. I'm trying to add it via next.config.js like this:

const nextConfig = {
    // Other next.js config
  
    // webpack: (config, options) => {
    //   config.resolve.fallback = {
    //     fs: false,
    //   }
  
    //   return config
    // }
  }
  
  module.exports = nextConfig

However, I'm still getting the Can't resolve fs error.

What should I do differently?

Answer 1

I managed to make web-tree-sitter work on my project. Here is how:

Add the Webpack config to next.config.js file:

module.exports = {
    webpack: (config, options) => {
    config.resolve.fallback = {
      fs: false,
    }

    return config
  },
}

Add the tree-sitter.wasm and tree-sitter-javascript.wasm files to the public/ folder.

That should work with the latest NextJS. And here is an example use with the app router (page.tsx):

"use client"

import React from "react"
import Parser from "web-tree-sitter"

const exampleCode = `
const MyComp = () => {
  const [count, setCount] = React.useState(0)
  return (
    <div>
      <h1 className="text-4xl">Hello World</h1>
      <button onClick={() => setCount(count + 1)}>Click Me</button>
      <p>Count: {count}</p>
    </div>
  )
}
`

export default function TreeSitterTest() {
  const [isReady, setIsReady] = React.useState(false)
  const [code, setCode] = React.useState(exampleCode)
  const [AST, setAST] = React.useState<Parser.Tree | null>(null)

  const parserRef = React.useRef<Parser>()

  React.useEffect(() => {
    ;(async () => {
      await Parser.init({
        locateFile(scriptName: string, scriptDirectory: string) {
          return scriptName
        },
      })

      const parser = new Parser()
      const Lang = await Parser.Language.load("tree-sitter-javascript.wasm")

      setIsReady(true)

      parser.setLanguage(Lang)

      parserRef.current = parser
    })()
  }, [])

  React.useEffect(() => {
    if (!isReady) return
    if (!parserRef.current) return

    const tree = parserRef.current.parse(code)
    console.log("rootNode:", tree.rootNode)

    setAST(tree)
  }, [code, isReady])

  return (
    <div>
      <h1 className="p-2 text-4xl">Tree Sitter Test</h1>

      <div className="flex-cole flex h-96 gap-1 p-1">
        <textarea
          className="rounder-sm flex-1 overflow-auto whitespace-pre-wrap border border-gray-300 p-2"
          value={code}
          onChange={(e) => setCode(e.target.value)}
        />
        <div className="rounder-sm flex-1 overflow-auto whitespace-pre-wrap border border-gray-300 p-2">
          {AST?.rootNode.toString()}
        </div>
      </div>
    </div>
  )
}

The project I implemented this on is OSS, so you can check it out directly there:

• Commit.
• Live URL.