I have an interesting problem that I have a reproducer for. Using a container to compartmentalize this system and make it reproducible, I can have it run successfully on my powerful laptop, but when running on a slow raspberry Pi it fails.
::::::::::::::
A.service
::::::::::::::
[Unit]
Description=Service A
After=B.service
BindsTo=B.service
[Service]
Type=simple
Restart=always
RestartSec=1
ExecStartPre=/bin/sleep 1
ExecStart=/bin/sleep 100
ExecStartPost=/bin/sleep 1
TimeoutStartSec=10s
[Install]
WantedBy=multi-user.target
::::::::::::::
B.service
::::::::::::::
[Unit]
Description=Service A
After=C.service
BindsTo=C.service
[Service]
Type=simple
Restart=always
RestartSec=1
ExecStartPre=/bin/sleep 1
ExecStart=/bin/sleep 100
ExecStartPost=/bin/sleep 1
TimeoutStartSec=10s
[Install]
WantedBy=multi-user.target
::::::::::::::
C.service
::::::::::::::
[Unit]
Description=Service A
[Service]
Type=simple
Restart=always
RestartSec=1
ExecStartPre=/bin/sleep 1
ExecStart=/bin/sleep 100
ExecStartPost=/bin/sleep 1
TimeoutStartSec=10s
[Install]
WantedBy=multi-user.target
::::::::::::::
Dockerfile
::::::::::::::
FROM ubuntu:18.04
RUN DEBIAN_FRONTEND=noninteractive apt update && apt install -y systemd init socat
COPY *.service /etc/systemd/system/
#RUN systemctl enable A.service
ENTRYPOINT ["/sbin/init"]
::::::::::::::
run.sh
::::::::::::::
docker build -t service .
docker stop -t 0 service && docker rm service
docker run -d --name service --privileged --cap-add SYS_ADMIN service
#docker run -d --cpus="0.3" --name service --privileged --cap-add SYS_ADMIN service
sleep 3
docker exec -it service service A start
sleep 1
docker exec -it service service A status
docker exec -it service service B status
docker exec -it service service C status
What the intent here is that there are 3 services: A, B, and C. The dependency is as follows: A->B->C. When starting service A, B should be started which then in turn starts C. The services are dummy services in this case and I've tried adding delays pre and post service, but the problem persists.
On my powerful laptop, I can somewhat reproduce the issue by adding "--cpus=0.3" to the 'docker run' line.
Any ideas on what could be the culprit?
I have discovered that service has an interesting "feature":
# avoid deadlocks during bootup and shutdown from units/hooks
# which call "invoke-rc.d service reload" and similar, since
# the synchronous wait plus systemd's normal behaviour of
# transactionally processing all dependencies first easily
# causes dependency loops
if ! systemctl --quiet is-active multi-user.target; then
sctl_args="--job-mode=ignore-dependencies"
fi
Obviously, if systemctl is launched with --job-mode=ignore-dependencies, it is less likely to work :-).
As expected, the following sequence works:
docker run -d --name service --privileged --cap-add SYS_ADMIN service
docker exec -ti service systemctl start multi-user.target
docker exec -it service service A start
Obviously, the best option is to replace service A start by systemctl start A. BTW, service is specific to Ubuntu while systemctl is common to nearly any Linux distribution.
I think that any service manually started in a docker container is impacted by this issue.
However, I still don't explain why it works on your powerful laptop.