I ran into this annoying issue and found it very difficult to find documentation on how to solve it. In my context I was using xvfb to test Anthropic's computer use functionality. Now that I've solved it, I'd like to share the fix with the world!
I tried everything to no avail until...
To fix your issue, locate the .X1-lock file that xvfb will have created and delete it.
rm -f .X1-lock
Voilà!