Can't provoke Priority Inversion in C++

Question

I'm trying to provoke Priority Inversion on a small C++ program for demonstration purposes but I can't: The low priority thread that holds the mutex is not preempted and keeps running on the critical section. This is what I'm doing:

// let's declare a global mutex
pthread_mutex_t my_mutex;
  ...

int main(int argc, char **argv) {
  ...
  pthread_t normal_thread;
  pthread_t prio_thread;

  pthread_mutexattr_t attr;
  pthread_mutexattr_init (&attr);
  pthread_mutexattr_setprotocol (&attr, PTHREAD_PRIO_NONE);  // ! None !
  pthread_mutex_init(&my_mutex, &attr);

  // create first normal thread (L):
  pthread_create(&normal_thread, NULL, the_locking_start_routine, NULL);

  // just to help the normal thread enter in the critical section
  sleep(2);

  // now will launch:
  // * (M) several CPU intensive SCHED_FIFO threads with priority < 99
  // * (H) one SCHED_FIFO thread that will try to lock the mutex, with priority < 99

  // build Real Time attributes for the Real Time threads:
  pthread_attr_t my_rt_att;
  pthread_attr_init(&my_rt_att);

  // it was missing in the original post and it was also wrong:
  // even setting the SchedPolicy you have to set "InheritSched"
  pthread_attr_setinheritsched(&my_rt_att, PTHREAD_EXPLICIT_SCHED)

  pthread_attr_setschedpolicy(&my_rt_att, SCHED_FIFO);
  struct sched_param params;

  params.sched_priority = 1;
  pthread_attr_setschedparam(&my_rt_att, &params);

  pthread_create(&prio_thread, &my_rt_att, the_CPU_intensive_start_routine, NULL) 

  params.sched_priority = 99;
  pthread_attr_setschedparam(&my_rt_att, &params);

  // create one RealTime thread like this:
  pthread_create(&prio_thread, &my_rt_att, the_locking_start_routine, NULL)  //coma was missing

  ...
}

void *the_locking_start_routine(void *arg) {
  ...
  pthread_mutex_lock(&my_mutex);
  // This thread is on the critical section
  // ... (skipped)
  pthread_mutex_unlock(&my_mutex);
  ...
}

... But it doesn't work, I can't have my desired Priority Inversion.

This is what happens:

As I understand, with a scheduller like Linux's CFS, a non-real time thread (SCHED_OTHER) will not run until there isn't any Real Time thread (SCHED_FIFO or SCHED_RR) in runnning state. But I have achieved this threads running simultaneously:

• (L) One non-real time (SCHED_OTHER) thread locking the mutex and consuming CPU
• (M) several Real Time threads (SCHED_FIFO , & priority > 0) CPU intensive and non-waiting to lock the mutex
• (H) One Real Time thread (SCHED_FIFO , & highest priority ) waiting for the lock

There are more Real Time CPU intensive threads (M) running than the amount of CPUs of my system ... but the non-real time thread holding (L) the lock is still consuming CPU and finishes it's work and releases the mutex before the "M" threads finish consuming CPU.

Why isn't the low priority thread preempted, the application dead-locked and I can't get priority inversion?

I'm using g++ 4.5.2 on a Ubuntu Desktop 11.04 with kernel 2.6.38-13.

Answer 1

1. Are you running the program as root?

2. What are your values of these sysctl parameters? Here are mine from an Ubuntu box. The default is to give real time only 0.95 seconds out of a 1 second slice:

   kernel.sched_rt_period_us = 1000000
   kernel.sched_rt_runtime_us = 950000
   

   This prevents the real-time domain from taking all the CPU. If you want real real time, you have to disable these parameters.

See: http://www.kernel.org/doc/Documentation/scheduler/sched-rt-group.txt

If you set sched_rt_runtime_us to -1, you disable this safety mechanism.