I'd like to be able to add or remove geometries as they appear/disappear in my environment.
The docs clearly state that all geometries must be added before the call to MultibodyPlant::Finalize(). Does this mean every time I add a new geometry, I have to re-create an entirely new plant/diagram/scene_graph and re-load all existing objects?
This strongly depends on the role.
Proximity: If you're using MultibodyPlant and are going to ask the plant to handle contact dynamics (e.g., MultibodyPlant needs to resolve contact), then the simplest answer is: you cannot change geometry after you finalize. Changes in geometry require new plants (as of today).
Illustration or Perception: If you're hoping to change the geometry for either of these roles, you can go into SceneGraph and manipulate those geometries to your hearts content before or after you finalize MultibodyPlant. If you're interested in this route, let me know in a comment and I'll elaborate on what this would look like in an edit.