After starting a new project in DjangoCMS 4.x, I added a static_placeholder as usual for the footer on my template and it didn't appear anymore on the structure sidebar, which were normally placed underneath the normal placeholders on DjangoCMS 3.x.
It took me a long time to figure out (actually after I opened a new bug on github and a dev explained it to me), that this is no bug or missing behaviour, but just the fact that static-placeholders got replaced by aliases on 4.x.
Below is my own answer for more details.
In DjangoCMS 4.x static placeholders were replaced by aliases to be able to support versioning as with the normal pages. You can add a footer alias to the template like described on the docs:
https://django-cms-docs.readthedocs.io/en/staging/introduction/02-templates_placeholders.html#static-aliases
<!-- some CMS template, e.g. default.html, or directly on base.html -->
{% load cms_tags sekizai_tags djangocms_alias_tags %}
<html>
<head>
<title>{% page_attribute "page_title" %}</title>
{% render_block "css" %}
</head>
<body>
{% cms_toolbar %}
{% placeholder "content" %}
<footer>
{% static_alias "footer" %}
</footer>
{% render_block "js" %}
</body>
</html>
After that, you can go to Aliases... and create/edit a new draft as you would with a page.
And of course, you must first publish a version first in order to see the footer if you're not logged in as an admin ;-)
And, as a side note, it's normal that they still don't appear on the structure sidebar underneath the "normal" placeholders when editing a page. Instead, they get their "own structure sidebar", when editing an alias. Which also didn't seem that obvious for me at the beginning.