I have a couple backends written as single-file Silex applications - basically the example from https://silex.symfony.com/ just with a couple more routes. No classes, no frills.
Is there any way to replicate this tiny low effort structure in Symfony 4 or do I really have to blow each backend up to the full Symfony structure in the Symfony way?
You should look at the Symfony 4 MicroKernelTrait. This is going to be very similar to the single-file-app Silex model.
It is a bit more verbose than Silex but it will allow you to migrate up easily if you decide you need more of the framework.