My Discord bot couldn't work for some time. In the Heroku logs I found out that after one dyno relocation, my bot was switching from down to restarting and from up to down constantly. When my dyno got relocated once again, this issue stopped, and although I had a few more relocations that day, the bot could restart successfully. What can I do about this issue and can I prevent this from happening?
2023-04-08T00:39:20.194824+00:00 heroku[worker.1]: Starting process with command `npm start`
2023-04-08T00:39:20.720188+00:00 heroku[worker.1]: State changed from starting to up
2023-04-08T00:39:23.488034+00:00 app[worker.1]:
2023-04-08T00:39:23.488052+00:00 app[worker.1]: > [email protected] start
2023-04-08T00:39:23.488052+00:00 app[worker.1]: > node index.js
2023-04-08T00:39:23.488053+00:00 app[worker.1]:
2023-04-08T00:39:25.626333+00:00 app[worker.1]: Ready! Logged in as user#6684
2023-04-08T01:04:25.911497+00:00 heroku[worker.1]: Relocating dyno to a new server
2023-04-08T01:04:25.915742+00:00 heroku[worker.1]: State changed from up to down
2023-04-08T01:04:25.917624+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T01:04:49.556990+00:00 heroku[worker.1]: Stopping all processes with SIGTERM
2023-04-08T01:04:59.529137+00:00 heroku[worker.1]: Process exited with status 143
2023-04-08T01:11:26.865257+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T01:11:26.867504+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T01:13:57.723927+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T01:13:57.729075+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T01:18:58.575661+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T01:18:58.578503+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T01:25:30.458128+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T01:25:30.461314+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T01:31:00.224672+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T01:31:00.229650+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T01:36:15.400305+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T01:36:15.404544+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T01:42:17.073988+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T01:42:17.077104+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T01:46:33.577191+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T01:46:33.579738+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T01:53:04.643819+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T01:53:04.646758+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T01:56:36.236763+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T01:56:36.239451+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T02:03:07.438558+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T02:03:07.441858+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T02:09:39.894195+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T02:09:39.896798+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T02:16:11.558150+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T02:16:11.561671+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T02:33:59.334484+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T02:33:59.337397+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T02:38:00.483471+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T02:38:00.485077+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T02:43:35.000000+00:00 app[api]: Build started by user [email protected]
2023-04-08T02:43:45.603384+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T02:43:45.607493+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T02:44:05.657278+00:00 app[api]: Deploy f085a386 by user [email protected]
2023-04-08T02:44:05.657278+00:00 app[api]: Release v92 created by user [email protected]
2023-04-08T02:44:06.032766+00:00 heroku[worker.1]: Restarting
2023-04-08T02:44:09.000000+00:00 app[api]: Build succeeded
2023-04-08T02:48:10.000000+00:00 app[api]: Build started by user [email protected]
2023-04-08T02:48:45.612585+00:00 app[api]: Deploy 88da9af7 by user [email protected]
2023-04-08T02:48:45.612585+00:00 app[api]: Release v93 created by user [email protected]
2023-04-08T02:48:45.887026+00:00 heroku[worker.1]: Restarting
2023-04-08T02:48:50.000000+00:00 app[api]: Build succeeded
2023-04-08T02:55:17.858114+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T02:55:17.860506+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T02:57:33.482229+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T02:57:33.485139+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T03:01:03.551038+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T03:01:03.553534+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T03:07:23.309498+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T03:07:23.311979+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T03:13:53.266567+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T03:13:53.269400+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T03:18:52.236456+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T03:18:52.238519+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T03:25:23.546873+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T03:25:23.550562+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T03:28:24.070008+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T03:28:24.074333+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T03:34:55.779645+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T03:34:55.781821+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T03:40:27.449073+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T03:40:27.451674+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T03:46:58.833299+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T03:46:58.834845+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T03:53:13.785321+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T03:53:13.790739+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T03:59:15.148336+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T03:59:15.150727+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T04:03:32.560565+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T04:03:32.563044+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T04:06:17.129993+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T04:06:17.132919+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T04:06:40.000000+00:00 app[api]: Build started by user xxxxx.com
2023-04-08T04:07:17.039633+00:00 app[api]: Deploy 88da9af7 by user [email protected]
2023-04-08T04:07:17.039633+00:00 app[api]: Release v94 created by user [email protected]
2023-04-08T04:07:17.304728+00:00 heroku[worker.1]: Restarting
2023-04-08T04:07:20.000000+00:00 app[api]: Build succeeded
2023-04-08T04:12:48.334033+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T04:12:48.338607+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T04:19:08.202862+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T04:19:08.206655+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T04:25:35.330505+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T04:25:35.333541+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T04:32:06.821998+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T04:32:06.823835+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T04:34:21.717834+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T04:34:21.723595+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T04:41:10.167573+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T04:41:10.169431+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T04:47:40.131086+00:00 heroku[worker.1]: State changed from starting to down
2023-04-08T04:47:40.133580+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T04:48:43.206855+00:00 heroku[worker.1]: Starting process with command `npm start`
2023-04-08T04:48:43.756477+00:00 heroku[worker.1]: State changed from starting to up
2023-04-08T04:48:45.059659+00:00 app[worker.1]:
2023-04-08T04:48:45.059674+00:00 app[worker.1]: > [email protected] start
2023-04-08T04:48:45.059674+00:00 app[worker.1]: > node index.js
2023-04-08T04:48:45.059675+00:00 app[worker.1]:
2023-04-08T04:48:46.808520+00:00 app[worker.1]: Ready! Logged in as user#6684
2023-04-08T06:29:33.841174+00:00 heroku[worker.1]: Relocating dyno to a new server
2023-04-08T06:29:33.849597+00:00 heroku[worker.1]: State changed from up to down
2023-04-08T06:29:33.852851+00:00 heroku[worker.1]: State changed from down to starting
2023-04-08T06:29:37.982119+00:00 heroku[worker.1]: Stopping all processes with SIGTERM
2023-04-08T06:29:57.450063+00:00 heroku[worker.1]: Process exited with status 143
2023-04-08T06:30:21.606144+00:00 heroku[worker.1]: Starting process with command `npm start`
2023-04-08T06:30:22.186549+00:00 heroku[worker.1]: State changed from starting to up
2023-04-08T06:30:24.877481+00:00 app[worker.1]:
2023-04-08T06:30:24.877493+00:00 app[worker.1]: > [email protected] start
2023-04-08T06:30:24.877494+00:00 app[worker.1]: > node index.js
2023-04-08T06:30:24.877494+00:00 app[worker.1]:
2023-04-08T06:30:26.393836+00:00 app[worker.1]: Ready! Logged in as user#6684
I ran into same problem recently, so tha task I was running was using alot of memory and I was on a performance L dedicated server, I reached out to support and here is what they said.
As the impacted dynos are on dedicated servers (Performance-L dynos), We have seen cases where it is possible that CPU-intensive tasks can sometimes cause the health check service on the server to determine something unhealthy. Also, aggressive use of temporary disk space can trigger the platform to trigger relocation of the dyno. Available disk space is one of the metrics the platform checks for the health of the backend server. For dyno types like Standard-2x, Since they are shared , they are provided with more disk space than dedicated instance dynos like Performace-L dynos.
I request you to check if you are using disk space aggressively or if any CPU-intensive tasks which are causing this to be seen in your worker dynos on a performance L and see if that can be addressed.
If a worker dynos is relocated during the processing of a job, the job may be interrupted and need to be retried. If switching to shared instances like Standard-2x dynos don't cause this, it would be ideal to move to these instances if you are meeting the application needs.
What I did to solve my problem was just to downgrade to a standard 2x server and that was it error the issue went away. If you do and still have same issue and you are running an intensive scripting task, try breaking them into different batches or contact heroku support