Search code examples
node.jsparallel-processingrequire

Nodejs required variable undefined if script file not run directly?

I apologise for the phrasing of the question - it's a bit difficult to sum up as a question - please feel free to edit it if you can clarify. Also, as this quite a complex and long query - thank you to all those who are putting in the time to read through it!

I have 4 files (listed with directory tree from project root) as part of a project I'm building which aims to scrape blockchains and take advantage of multiple cores do get the job done:

  1. ./main.js
  2. ./scraper.js
  3. ./api/api.js
  4. ./api/litecoin_api.js

main.js

const { scraper } = require('./scraper.js')
const blockchainCli = process.env.BLOCKSCRAPECLI || 'litecoin-cli'

const client = (args) => {
  // create child process which returns a promise which resolves after
  // data has finished buffering from locally hosted node using cli
  let child = spawn(`${blockchainCli} ${args.join(' ')}`, {
    shell: true
  })
  // ... wrap command in a promise here, etc
}

const main = () => {
  // count cores, spawn a worker per core using node cluster, add
  // message handlers, then begin scraping blockchain with each core...
  scraper(blockHeight)
}

main()

module.exports = {
  client,
  blockchainCli
}

scraper.js

const api = require('./api/api.js')
const scraper = async (blockHeight) => {
  try {
    let blockHash = await api.getBlockHashByHeight(blockHeight)
    let block = await api.getBlock(blockHash)
    // ... etc, scraper tested and working, writes to shared writeStream
}

module.exports = {
  scraper
}

api.js

const { client, blockchainCli } = require('../main.js')
const litecoin = require('./litecoin_api')

let blockchain = undefined

if (blockchainCli === 'litecoin-cli' || blockchainCli === 'bitcoin-cli') {
  blockchain = litecoin
}
// PROBLEM HERE: blockchainCli (and client) are both undefined if and
// only if running scraper from main.js (but not if running scraper
// from scraper.js)

const decodeRawTransaction = (txHash) => {
  return client([blockchain.decodeRawTransaction, txHash])
}

const getBlock = (blockhash) => {
  return client([blockchain.getBlock, blockhash])
}

const getBlockHashByHeight = (height) => {
  return client([blockchain.getBlockHash, height])
}

const getInfo = () => {
  return client([blockchain.getInfo])
}

const getRawTransaction = (txHash, verbose = true) => {
  return client([blockchain.getRawTransaction, txHash, verbose])
}

module.exports = {
  decodeRawTransaction,
  getBlock,
  getBlockHashByHeight,
  getInfo,
  getRawTransaction
}

So, I've taken out most the noise in the files which I don't think is necessary but it's open source so if you need more take a look here.

The problem is that, if I start the scraper from inside scraper.js by doing, say, something like this: scraper(1234567) it works like a charm and outputs the expected data to a csv file.

However if I start the scraper from inside the main.js file, I get this error:

Cannot read property 'getBlockHash' of undefined
    at Object.getBlockHashByHeight (/home/grayedfox/github/blockscrape/api/api.js:19:29)
    at scraper (/home/grayedfox/github/blockscrape/scraper.js:53:31)
    at Worker.messageHandler (/home/grayedfox/github/blockscrape/main.js:81:5)

I don't know why, when launching the scraper from main.js, the blockchain is undefined. I thought it might be from the destructuring, but removing the curly braces from around the first line in the example main.js file doesn't change anything (same error).

Things are a bit messy at the moment (in the middle of developing this branch) - but the essential problem now is that it's not clear to me why the require would fail (cannot see variables inside main.js) if it's used in the following way:

  • main.js (execute scraper()) > scraper.js > api.js

But not fail (can see variables inside main.js) if it's run like this:

  • scraper.js (execute scraper()) > api.js

Thank you very much for your time!

Solution

  • You have a circular dependency between main and api, each requiring in the other. main requires api through scraper and api directly requires main. That causes things not to work.

    You have to remove the circular dependency by putting common shared code into its own module that can be included by both, but doesn't include others that include it. It just needs better modularity.