From list of LSM hook definitions https://github.com/torvalds/linux/blob/v5.18/include/linux/lsm_hook_defs.h#L179 I can see file_open hook.
I looked into definition of openat syscall https://elixir.bootlin.com/linux/v5.18/source/fs/open.c#L1233
Then I tried looked into functions that it calls, do_sys_op -> do_sys_openat. From there, it becomes harder to determine where the hook is actually called. And I'm actually not sure how security_file_open is called.
Is there a general way to determine if a syscall has an LSM hook?
Probably the easiest way is to use in kernel function graph profiler. It can be used via trace-cmd wrapper, or directly via tracefs.
$ trace-cmd record -p function_graph -g '*openat*' -F head /etc/mtab
$ trace-cmd report | less
head-157791 [001] 2350.203452: funcgraph_entry: | __x64_sys_openat() {
head-157791 [001] 2350.203453: funcgraph_entry: | do_sys_openat2() {
head-157791 [001] 2350.203453: funcgraph_entry: | getname() {
...................................................................................
head-157791 [001] 2350.201689: funcgraph_entry: | may_open() {
head-157791 [001] 2350.201689: funcgraph_entry: | path_noexec() {
head-157791 [001] 2350.201689: funcgraph_exit: 0.110 us | }
head-157791 [001] 2350.201689: funcgraph_entry: | inode_permission() {
head-157791 [001] 2350.201689: funcgraph_entry: | generic_permission() {
head-157791 [001] 2350.201689: funcgraph_exit: 0.120 us | }
head-157791 [001] 2350.201689: funcgraph_entry: | security_inode_permission() {
head-157791 [001] 2350.201689: funcgraph_exit: 0.110 us | }
head-157791 [001] 2350.201689: funcgraph_exit: 0.520 us | }
head-157791 [001] 2350.201690: funcgraph_exit: 0.970 us | }
head-157791 [001] 2350.201690: funcgraph_entry: | vfs_open() {
head-157791 [001] 2350.201690: funcgraph_entry: | do_dentry_open() {
head-157791 [001] 2350.201690: funcgraph_entry: | path_get() {
head-157791 [001] 2350.201690: funcgraph_entry: | mntget() {
head-157791 [001] 2350.201690: funcgraph_exit: 0.130 us | }
head-157791 [001] 2350.201690: funcgraph_exit: 0.370 us | }
head-157791 [001] 2350.201691: funcgraph_entry: | try_module_get() {
head-157791 [001] 2350.201691: funcgraph_exit: 0.120 us | }
head-157791 [001] 2350.201691: funcgraph_entry: | security_file_open() {
head-157791 [001] 2350.201691: funcgraph_entry: | __fsnotify_parent() {
head-157791 [001] 2350.201691: funcgraph_exit: 0.120 us | }
head-157791 [001] 2350.201691: funcgraph_entry: | __fsnotify_parent() {
head-157791 [001] 2350.201691: funcgraph_exit: 0.110 us | }
head-157791 [001] 2350.201691: funcgraph_exit: 0.540 us | }
head-157791 [001] 2350.201691: funcgraph_entry: | xfs_file_open() {
head-157791 [001] 2350.201692: funcgraph_entry: | generic_file_open() {
head-157791 [001] 2350.201692: funcgraph_exit: 0.110 us | }
head-157791 [001] 2350.201692: funcgraph_exit: 0.300 us | }
head-157791 [001] 2350.201692: funcgraph_entry: | file_ra_state_init() {
head-157791 [001] 2350.201692: funcgraph_entry: | inode_to_bdi() {
head-157791 [001] 2350.201692: funcgraph_exit: 0.090 us | }
head-157791 [001] 2350.201692: funcgraph_exit: 0.260 us | }
head-157791 [001] 2350.201692: funcgraph_exit: 2.410 us | }
head-157791 [001] 2350.201693: funcgraph_exit: 2.600 us | }
trace-cmd is just a wrapper over ftrace API, and the same could be achieved without it, only less convenient.
tracefs /sys/kernel/debug/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
cd /sys/kernel/debug/tracing
echo "do_sys_openat2" > set_graph_function # optional, only trace this function
echo 1 > /proc/sys/kernel/ftrace_enabled # enable ftrace
echo function_graph > current_tracer
cat trace_pipe # display trace events
echo nop > current_tracer
echo "" > set_graph_function
echo 0 > /proc/sys/kernel/ftrace_enabled # disable ftrace
Although, I suggest you use trace-cmd since it's much more convenient.
Another option would be perf ftrace command, which also uses ftrace with function_graph profiler.
$ perf ftrace -t function_graph cat /dev/null
....
10) 0.420 us | }
10) 0.560 us | }
10) | vfs_open() {
10) | do_dentry_open() {
10) | path_get() {
10) 0.060 us | mntget();
10) 0.220 us | }
10) 0.070 us | try_module_get();
10) | security_file_open() {
10) 0.080 us | __fsnotify_parent();
10) 0.200 us | }
10) | xfs_file_open() {
10) 0.080 us | generic_file_open();
10) 0.190 us | }
10) | file_ra_state_init() {
10) 0.070 us | inode_to_bdi();
10) 0.200 us | }
10) 1.410 us | }
10) 1.540 us | }
....
As an output, you'll get a function call graph from a kernel. Obviously, you will not see inlined or static functions in this graph.
Then you can just search functions that start with security_ prefix and in this call graph you'll be able to determine by which function it were called and in what order, since many syscalls trigger more than a single LSM hook.