Search code examples
javascriptnode.jsnpz-file

How to read .npz file in Javascript?


I'm trying to follow a tutorial on YT on how to write a basic neural network, but it's written in python. I'm trying to write everything using javascript and I haven't found a way to read the npz file into JS.

I have tried npyjs and tfjs-npy-node, but neither works. Using npy.js yields the error:

Only absolute URLs are supported

Using tfjs-npy-node yields any one of these errors, depending on how I pass it the file:

Expected provided filepath () to have file extension .npy

Error: assert failed

Error: Not a numpy file

Are there any functional libraries that work in Node and will read/parse the file without me needing to create an entire project dedicated to it? (looking at you, tensorflow)

Edit: Examples

npyjs: import npyjs from 'npyjs';

const n = new npyjs();
const dataPath = path.resolve('./data/mnist.npz')
console.log(dataPath)
let data = n.load(dataPath);

console.log('NPZ data:', data);

result:

D:\Projects\Programming\Personal\neural-networks-> tutorial\js\data\mnist.npz NPZ data: Promise { } D:\Projects\Programming\Personal\neural-networks-tutorial\js\node_modules\node-fetch\lib\index.js:1327 throw new TypeError('Only absolute URLs are supported'); ^

TypeError: Only absolute URLs are supported

I assume it's related to this issue, but following the "working" solution: import npyjs from 'npyjs';

const n = new npyjs();
const dataPath = path.resolve('./data/mnist.npz')
console.log(dataPath)
let buf = fs.readFileSync(dataPath);
let data = n.parse(buf.buffer)
console.log('NPZ data:', data);

gives:

SyntaxError: Unexpected token '', "!�&vt" is not valid JSON at JSON.parse () at npyjs.parse (file:///D:/Projects/Programming/Personal/neural-networks-tutorial/js/node_modules/npyjs/index.js:126:29)

tfjs-npy-node:

import {npz} from "tfjs-npy-node"
(same as before)
let data = await npz.load(dataPath);

gives:

Error: Could not load D:\Projects\Programming\Personal\neural-networks-tutorial\js\data\mnist.npz: No backend found in registry. at Object.load (D:\Projects\Programming\Personal\neural-networks-tutorial\js\node_modules\tfjs-npy-node\dist\src\npz.js:41:15)

Project Structure: root

|- data

-|- mnist.npz

|- program.js


Solution

  • Edit: Sometimes simply googling the answer instead oif a sub propblem is best: answer

    The answer for tfjs-npy-node is to install a different library @tensorflow/tfjs-backend-cpu and import import "@tensorflow/tfjs-backend-cpu" into the project. This allows it to read the file, but only shows teh tensors as far as I can tell.

    Here is what worked for me(based on this):

    import path from "path"
    import npyjs from 'npyjs'
    import JSZip from 'jszip'
    import fs from "fs"
    
    
    const dataPath = path.resolve('./data/mnist.npz')
    
    
    let contents = fs.readFileSync("./data/mnist.npz")
    
    const jsZip = new JSZip()
    const npzFiles = await jsZip.loadAsync(contents)
    const _npyjs_ = new npyjs()
    
    let data = Object.entries(npzFiles.files)
    
    for (const [npy_filename, npy_data] of Object.entries(npzFiles.files)) {
        if (!npy_filename.endsWith('.npy')) {
            console.error('error .npy')
        }
    
        const npy_array_buffer = await npzFiles.files[npy_filename].async("arraybuffer")
        console.log({ npy_filename, npy_data, npy_array_buffer: npy_array_buffer })
    
        data[npy_filename] = await _npyjs_.parse(npy_array_buffer)
    }