Assuming hosted environment where untrusted code is executed in worker nodes, how can one lock down the manager node to prevent compromise by untrusted worker nodes? How vulnerable are e.g. the gossip protocols, etc.?
Worker nodes do not have access to run code on the managers by default. This is by design, and anything violating that is a bug. This is only covers what swarm mode provides, so you still need to avoid things like sharing ssh keys between the workers and managers, and you do not want the docker socket on the managers exposed by the network. As long as you have not explicitly added any security vulnerabilities, and run your jobs on the workers, the biggest security concern with swarm mode is a DoS attack which can be isolated with memory and CPU limits and reservations on your services.