Search code examples
node.jsherokuwebsocketpuppeteer

On heroku, puppeteer's Network.webSocketFrameReceived event is never triggered. Why?

I have built a small app that I deployed to heroku. Locally, the whole thing is working as expected. But when deployed, the Network.webSocketFrameReceived event is never triggered. It is a node app that runs on express with a minimal websocket server. The goal of the app is to open some url using headless chrome (i am using puppeteer here), record the websocket frames and parse them if they contain some specific fields, close connection when successful. Then move to next url.

async function openUrlAndParseFrames(page, url) {
  await new Promise(async function (resolve) {
    const parseWebsocketFrame = (response) => {
      console.log('parsing websocket frame...', response);
      let payload;
      
        try {
          // some parsing here
        } catch (e) {
          console.error(`Error while parsing payload ${response.response.payloadData}`)
        }
     
    }

    console.log('Go to url', url);

    await page.goto(url);
    const cdp = await page.target().createCDPSession();
    await cdp.send('Network.enable');
    await cdp.send('Page.enable');
    cdp.on('Network.webSocketFrameReceived', parseWebsocketFrame);

  });
}

Is it not possible to make this websocket connection on heroku using puppeteer? I never receive the "parsing websocket frame..." logs...

PS: I am aware of this special args I need to set for puppeteer to run on heroku

puppeteer.launch({ args: ['--no-sandbox', '--disable-setuid-sandbox'] });

Also I added the buildpacks heroku/nodejs and https://github.com/jontewks/puppeteer-heroku-buildpack

Solution

  • I found the answer myself. The real problem was, that the IP range (from Heroku) was blocked and I didn't even access the page I was trying to but was blocked with a 403 from CloudFront.

    I figured it out by logging the page content. const websiteContent = await page.content(); Which showed the error page html.

    After trying various things I decided to move away from Heroku and now successfully deployed to Google App Engine.