I see for python BCC implementation the syscall __x64_sys_openat is used to attach a kprobe, however in libbpf implementation a kprobe is attached to sys_enter_openat. It seems both capture openat() syscall, I tested it with cat file.txt.
What is the difference between them? And which one is more reliable to use?
__x64_sys_openat is the name of some function in the Linux kernel, to which BCC attaches a kprobe.
sys_enter_openat is the name of a tracepoint in Linux, meaning that this is a (more or less) stable interface to which you can hook for tracing, including with an eBPF program. You can see the available tracepoints on your system by listing the entries under /sys/kernel/debug/tracing/events/. I think BCC also has a utility called tplist to help with it.
When given the choice, I would recommend hooking at tracepoints if possible, because they tend to be more stable than kernel internals: The parameters for __x64_sys_openat, or the name of that function, could change between different kernel versions for example; or the name would change on an other architecture, et cætera. However, the tracepoint is unlikely to change. Note that the instability of kernel's internals is somewhat mitigated for eBPF with CO-RE.
Then it is not always possible to hook to a tracepoint: You can only use one of the existing tracepoints from the kernel. If you want to hook to another random function where no tracepoint is present (and assuming this function was not inlined at compilation time - check this by looking for it in /proc/kallsyms), then you want to use a kprobe.
Sometimes you also need to pay extra attention to where you hook. For example, for security use cases (i.e. blocking a syscall), syscall tracepoints (or the corresponding kernel functions, obviously) are not always the best hooking points because they might leave you open to TOCTOU attacks. LSM hooks could be a good solution for that use case.