Does Background Color Without Text Matter?
Regarding color contrast:
In this example, do I need to be careful of the blue background even though there is no copy directly on top of it? I.e. do I only need to be concerned with background color when there is copy on top of it, and I could hide the element with the background color from screen readers without causing an issue?
And in this one, do I need to be careful of the contrast between the light blue background and the button or am I only concerned with the contrast between the copy and the button? (I know the image is blurry. It's just an example.)
Both examples are fine as they are.
There are two things that are applicable here.
The first is contrast between text and the background for that text - you must have a contrast ratio of 4.5:1 for normal text and 3:1 for large text / bold text to be WCAG AA compliant. In the first example it is essentially black on white so it will pass easily.
The second is for controls. Buttons, inputs etc. should have a contrast ratio with their background of 3:1 minimum, no matter what state they are in (so if your above button turned white with black text on hover it probably wouldn't pass). In your second example your button is black on light blue so it certainly passes this also.
Also worth noting is that text within controls (your button) has the same 4.5:1 contrast requirement. Yet again white on black passes easily so you are fine.
Just check the contrast (almost certainly fine) on the red button with white text, reds and oranges can be deceiving in their contrast ratios (but as it is quite a dark red I am 99% sure you are fine just by looking at it).
For clarity your blue background in the first one could be 1% darker than the white box and it would be fine as it is not an interactive control that it surrounds.