There are two ways to access hardware:
If a user-mode process directly wants to access I/O without using system calls and it knows about a particular piece of hardware it can't access it through memory-mapped I/O as it's not in its address space, so a segmentation fault will occur. But what about using I/O ports to do so? As I/O ports are not in the memory address space, but are accessed by instructions executed directly executed by processor, what will happen, can the process access them or not?
Short version
No. For example, in the x86 architecture, in and out are privileged instructions that can only run in Ring 0. Since user-mode applications run at Ring 3, the CPU will fault when trying to execute that instruction.
Long version
In general, in an OS that enforces memory protection and process separation (such as Windows, Linux, Mac OS, but not DOS), user-mode applications don't have direct access to the hardware, and use system calls to ask the kernel to do the actions required on the their behalf. The kernel can then decide if the application has the right to do said action, if the action is safe to do etc.
If a user-mode application is able to do anything that can normally only be done by the kernel, it is a bug/backdoor and a huge security vulnerability in the kernel (or sometimes, the CPU itself).
On the x86 architecture, I/O port instructions are privileged in all operating modes except Real Mode. An attempt to execute any privileged instruction will generate a fault (specifically, #GP, General-Protection Fault, which corresponds to INT 0xD).
In order to allow processes like user-mode drivers to work, the OS is able to override this privileged status of I/O instructions in 2 ways:
IOPL (I/O privilege level) field in the FLAGS register (bits 12 and 13). This is a number from 0 to 3 that specifies the least privileged ring that can run I/O instructions. Setting IOPL to 3 will allow ALL processes to access ANY I/O port, which is insecure and defeats the purpose of protecting everything else like memory. For this reason, operating systems usually set this field to 0.IOPB (I/O Permissions Bitmap) bitfield in the Task State Segment (TSS) structure. This is simply a collection of 65536 bits, each specifying if an I/O port can be accessed (0) or not (1). If a user-mode driver needs to access, say, port 0xAB, then the OS can clear the corresponding bit in the TSS and allow the user-mode code to access only that port, even though IOPL=0. If the operating system doesn't include all 65536 bits, those missing are assumed by the CPU to be privileged and thus inaccessible.