Say you have a Python program that you can successfully package using only pyproject.toml. What are the downsides? Why use setup.py or setup.cfg in this case?
There is no downside in not having a setup.py. It is just that in some particular cases some elements of the packaging can not be expressed in a descriptive manner (which means without code) in setup.cfg or pyproject.toml. This can range from some custom dynamic package metadata to the handling of packaging for custom non-Python code and many other things.
My recommendation is:
Avoid using setup.py as much as possible, if you can do without this file completely, then do without it
Place the standard parts in pyproject.toml:
[build-system] section, it is strongly recommended to have this section independently of whether or not you have a setup.cfg file and/or setup.py script
[project] section, and avoid dynamic fields as much as possible
For the parts that are specific to setuptools, you can choose to have them in pyproject.toml (under the [tool.setuptools] section) or in setup.cfg, as you prefer
Related: