I have installed WordPress and created permalinks for every page.
However, there is still a version available with example.com/?p=123 which represents the ID in the database.
Is it possible to disable these links so only my permalink is available?
I have seen that Google has indexed my permalinks and links with the ID - do I have to create a 301 redirect for every ID link to the permalink version?
If you have enabled permalinks in the settings, then WordPress will automatically redirect those URLs for you via its built in canonical API:
example.com/?p=123 => example.com/pretty-url (happens automatically)
Manually adding redirects will result in a redirect loop for those pages.
Google will eventually re-index your pages with the pretty permalinks you have set up.
It might help to ask Google to re-crawl the website via Google's Search Console.