Running performance tools using RBAC

I recently started a job that will involve a lot of performance tweaking.

I was wondering whether tools like eBPF and perf can be used with RBAC? Or will full root access be required? Getting root access might be difficult. We're mainly using fairly old Linux machines - RHEL 6.5. I'm not too familiar with RBAC. It home I have used Dtrace on Solaris, macOS and FreeBSD, but there I have the root password.

Solution

RHEL lists several profiling and tracing solutions for RHEL6 including perf in its Performance Tuning Guide and Developer Guide: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-analyzperf-perf https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/developer_guide/perf-using

Chapter 3. Monitoring and Analyzing System Performance of Performance Tuning Guide mentions several tools: Gnome System Monitor, KDE System Guard, Performance Co-Pilot (PCP), top/ps/vmstat/sar, tuned and ktune, MRG Tuna, and application profilers SystemTap, Oprofile, Valgrind (which is not real profiler, but cpu emulator with instruction and cache event counting), perf.

Chapter 5. Profiling of Developer Guide lists Valgrind, oprofile, SystemTap, perf, and ftrace.

Usually profiling of kernel or whole system is allowed only for root, or for user with CAP_SYS_ADMIN capability. Some profiling is limited by sysctl variables

kernel.perf_event_paranoid (documented in https://www.kernel.org/doc/Documentation/sysctl/kernel.txt):

perf_event_paranoid:

Controls use of the performance events system by unprivileged users (without CAP_SYS_ADMIN). The default value is 2.

 -1: Allow use of (almost) all events by all users
     Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
     Disallow raw tracepoint access by users without CAP_SYS_ADMIN
>=1: Disallow CPU event access by users without CAP_SYS_ADMIN
>=2: Disallow kernel profiling by users without CAP_SYS_ADMIN

kernel.kptr_restrict (https://www.kernel.org/doc/Documentation/sysctl/kernel.txt), which also change perf ability to profile kernel

kptr_restrict:

This toggle indicates whether restrictions are placed on
exposing kernel addresses via /proc and other interfaces.

More recent versions of ubuntu and rhel (7.4) has also kernel.yama.ptrace_scope http://security-plus-data-science.blogspot.com/2017/09/some-security-updates-in-rhel-74.html

... use kernel.yama.ptrace_scope to set who can ptrace. The different values have the following meaning:
# 0 - Default attach security permissions.
# 1 - Restricted attach. Only child processes plus normal permissions.
# 2 - Admin-only attach. Only executables with CAP_SYS_PTRACE.
# 3 - No attach. No process may call ptrace at all. Irrevocable until next boot.
You can temporarily set it like this:
 echo 2 > /proc/sys/kernel/yama/ptrace_scope

To profile a program you should have access to debugging it, like attaching with gdb (ptrace capability) or strace. I don't know RHEL or its RBAC so you should check what is available to you. Generally perf profiling of own userspace programs on software events is available for more cases. Access to per-process cpu hardware counters, profiling of programs of other users, profiling of kernel is more limited. I can expect that correctly enabled RBAC should not allow you or root to profile kernel, as perf can inject tracing probes and leak information from kernel or other users.

Qeole says in comment that eBPF is not implemented for RHEL6 (added in RHEL7.6; with XDP - eXpress Data Path in RHEL8), so you only can try ftrace for tracing or stap (SystemTap) for advanced tracing.