How to use Google App Engine Cron jobs in a cheaper way?

Question

I have a twitter bot which tweets out content a few times a day. here is my cron.yaml file.

cron:
- description: "twitter instagram scraper"
  url: /scrape/twitter_intra
  schedule: every 24 hours
  target: scraper
- description: "USD to LKR scraper"
  url: /scrape/exRates
  schedule: every 1 hours
  target: scraper
- description: "last 24 hour weather"
  url: /scrape/weather_last24hours
  schedule: every day 12:00
  target: scraper
- description: "tweet out last 24 hour weather"
  url: /tweet/weather_last24hours
  schedule: every day 13:00
  target: twitter
- description: "tweet out exchange Rate USD to LKR"
  url: /tweet/exRates
  schedule: every day 7:00
  target: twitter

here is an example of one request method,

app.get(`/tweet/weather_last24hours`, async (req, res, next) => {
    console.log(`Tweet!! last 24 hours`);

    try {
        //await tweetText('This is a test');
        const report = await getWeatherLast24Hours();
        const content = makeTweetLast24HourWeather(report);

        console.log(content);
        await tweetText(content);

        res.status(200)
            .set('Content-Type', 'text/plain')
            .send(`Completed Successfully...!`)
            .end();
    } catch (error) {
        next(error);
    }
});

right now It's working fine except it cost me 2.2$ a day because I have to keep an instance up all day by doing the following.

const PORT = process.env.PORT || 8080;
app.listen(PORT, () => {
    console.log(`App listening on port ${PORT}`);
    console.log('Press Ctrl+C to quit.');
});

I try removing this part of the code assuming that GCP cron job can up an instance on its own and run the task. Even though cron job itself was able to up the instance, Task was a failure with 500 Error code and following message.

This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.

error message in the logs. I try it a few times and got the same result.

1. Is there a solution to this, by starting an instance just before cron job executes?
2. Can newly introduced cloud scheduler can help?
3. am I doing something wrong?

Thank you in advance.

Answer 1

Your 500 is a warning to tell you something you already know: that bootup time will be longer due to needing to warm up a new instance. This is not necessarily a problem in itself, unless you notice that your tasks are not running properly.

---

To use Cloud Scheduler, consider decomposing your application into Cloud Functions. You can build a separate function for each of your 5 endpoints, plus a sixth that contains the shared logic (e.g. getWeatherLast24Hours()). You can then invoke them on a schedule using Cloud Scheduler.

The cost of running with Cloud Functions + Scheduler will be near-zero, so you'll want to do your own ROI evaluation to determine whether the development effort is worth the savings.