Problem: C++ application changes directory permissions on shutdown

Question

I'm in the (ever-going) process of diagnosing a baffling problem with an application we use at work.

First, some notes about this application:

• Required to run as the root user
• Runs on the Solaris 10 operating system
• Compiled for C++14
• Normal shutdown is conducted by receiving SIGTERM
• Writes a log file (explicitly sets permissions to 660) to a data directory (with default 770 permissions)

The application runs fine and does everything it's supposed to do, up until the point it terminates. Upon termination, the application is changing the permissions on the data directory from 770 to 660.

My coworkers are as baffled as I am. Even our system administrator doesn't understand why this is happening.

Things I've tried:

• Print statements: The application reports the directory permissions are 770 until the exit or return statements
• Check the logging: The logging mechanism is shared with several other applications, none of which have this issue
• Running as myself: The directory's permissions are not changed on termination
• Change umask to 027: The directory's permissions are still changed to 660
• Check system logs: The sudo and messages logs do not show any calls to chmod for the directory (except those made to change the permissions back)

Due to the nature of the application, I cannot provide any of the code here for review/inspection. Further, many of the standard diagnostics tools are unavailable on the system in question.

However, I'm hopeful the gurus here can provide insight into what might be causing this problem, where to look going forward, or (ideally) how to fix it.

Answer 1

You can use the following simple dTrace script to get a stack trace from any process that calls chmod() (or any of its variants such as fchmod() or fchmodat()):

#!/usr/sbin/dtrace -s

syscall::fchmodat:entry
{
    printf{ "\nExecname: %s\n", execname );
    ustack();
}

You can filter by execname to only print chmod call stacks from your executable with

#!/usr/sbin/dtrace -s

syscall::fchmodat:entry
/ execname == "yourExecName" /
{
    ustack();
}

You can add more or less stack frames with ustack( 10 ); to print, for example, 10 stack frames. If you want longer or shorter function names in the stack trace, you can specify the string length with ustack( 10, 50 ); to print 10 stack frames with each function name printing up to 50 characters.

If your binary has been completely stripped of symbol names you may not get function names, only addresses.

As it's a C++ binary, you might have to demangle the function names.

Once you get a stack trace, you can start working on what exactly is happening.