Search code examples
linuxlinux-kernelkernelreal-time

Cyclictest for RT patched Linux Kernel

Hello I patched the Linux kernel with the RT-Patch and tested it with the Cyclinctest which monitors latencies. The Kernel isn't doing good and not better than the vanilla kernel. https://rt.wiki.kernel.org/index.php/Cyclictest

I checked the uname for RT, which looks fine.

So I checked the requirements for the cyclinctest and it states that I have to make sure that the following is configured within the kernel config:

CONFIG_PREEMPT_RT=y
CONFIG_WAKEUP_TIMING=y
CONFIG_LATENCY_TRACE=y
CONFIG_CRITICAL_PREEMPT_TIMING=y
CONFIG_CRITICAL_IRQSOFF_TIMING=y

The Problem now arising is that the config doesn't contain such entries. Maybe there are old and the they may be renamed in the new patch versions (3.8.14)?

I found options like:

CONFIG_PREEMPT_RT_FULL=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_RT_BASE=y
CONFIG_HIGH_RES_TIMERS=y

Is that enought in the 3.x kernel to provide the required from above? Anyone a hint?

Solution

  • There's a lot that must be done to get hard realtime performance under PREEMPT_RT. Here are the things I am aware of. Entries marked with an asterisk apply to your current position.

    • Patch the kernel with PREEMPT_RT (as you already did), and enable CONFIG_PREEMPT_RT_FULL (which used to be called CONFIG_PREEMPT_RT, as you correctly derived).
    • Disable processor frequency scaling (either by removing it from the kernel configuration or by changing the governor or its settings). (*)
      • Reasoning: Changing a core's frequency takes a while, during which the core does no useful work. This causes high latencies.
      • To remove this, look under the ACPI options in the kernel settings.
      • If you don't want to remove this capability from the kernel, you can set the cpufreq governor to "performance" to lock it into its highest frequency.
    • Disable deep CPU sleep states
      • Reasoning: Like switching frequencies, Waking the CPU from a deep sleep can take a while.
      • Cyclictest does this for you (look up /dev/cpu_dma_latency to see how to do it in your application).
      • Alternatively, you can disable the "cpuidle" infrastructure in the kernel to prevent this from ever occurring.
    • Set a high priority for the realtime thread, above 50 (preferably 99) (*)
      • Reasoning: You need to place your priority above the majority of the kernel -- much of a PREEMPT_RT kernel (including IRQs) runs at a priority of 50.
      • For cyclictest, you can do this with the "-p#" option, e.g. "-p99".
    • Your application's memory must be locked. (*)
      • Reasoning: If your application's memory isn't locked, then the kernel may need to re-map some of your application's address space during execution, triggering high latencies.
      • For cyclictest, this may be done with the "-m" option.
      • To do this in your own application, see the RT_PREEMPT howto.
    • You must unload the nvidia, nouveau, and i915 modules if they are loaded (or not build them in the first place) (*)
      • Reasoning: These are known to cause high latencies. Hopefully you don't need them on a realtime system :P
    • Your realtime task must be coded to be realtime
      • For example, you cannot do file access or dynamic memory allocation via malloc(). Many system calls are off-limits (it's hard to find which ones are acceptable, IMO).
      • cyclictest is mostly already coded for realtime operation, as are many realtime audio applications. You do need to run it with the "-n" flag, however, or it will not use a realtime-safe sleep call.

    The actual execution of cyclictest should have at least the following set of parameters:

    sudo cyclictest -p99 -m -n