It depends.
If the private key is not password-protected on the client, yes, the server is vulnerable and can definitely be accessed. If you want to access the server without password, this is usually the case.
If it is password-protected, this is not the case. However, you would still have to enter your password upon connecting. Consequently, you don't ease accessing your server (if you don't cache the password), but you still increase the security of your server.
If you absolutely want to access a server without entering your password and without compromising the server if the machine is itself compromised, you want to use a HSM that will hold the private key securely (such as smart cards or YubiKey-like HSMs).
PS: This question would have been better suited for Server Fault.