I have a C++ and a Python code communicating with each other via reading and writing files in a directory, many times over. They also delete some of the files after reading the data. This is being done on Ubuntu 17.10.
However, after doing this sequence multiple times (>6000 - 7000 times), it comes to a point where I get the following error:
Couldn't open the directory: Too many open files
This error message is always delivered from the C++ code.
For the C++ code:
The data write is done using:
std::string opfilepath("/some/path/");
ofstream opfile;
opfile.open (opfilepath);
opfile<<"some stuff"<<endl;
opfile.close()
Number of files in a directory is counted using:
DIR* dirFile = opendir( path );
int count = 0;
if ( dirFile )
{
struct dirent* hFile;
errno = 0;
while (( hFile = readdir( dirFile )) != NULL )
{
if ( !strcmp( hFile->d_name, "." )) continue;
if ( !strcmp( hFile->d_name, ".." )) continue;
if ( strstr( hFile->d_name, ext.c_str() ))
count++;
}
}
closedir( dirFile );
Reads file using:
std::ifstream file("/some/path");
if (file.is_open())
{
//do something
}
file.close();
And files are deleted using:
boost::filesystem::remove("/some/file.txt")
For the Python code:
Files are written using:
f = open("/some/file.txt", "a")
f.write("some stuff")
f.close()
Files are read using:
fp = open(path)
lines = fp.read().splitlines()
fp.close()
And files are deleted using:
os.remove("/some/path/and/file.txt")
All the files are opened and closed in a modular way and synchronization does not seem to be a problem. But I keep seeing Too many open files, and always after a fixed number of iteration.
Any ideas why this might be happening?
Couldn't open the directory: Too many open files
This means that your process has exceeded the max open files parameter that was allocated to it, see that using ulimit -a, or ulimit -n. It could be possible that your process legitimately needs more fd's - if that is the case increase ulimit -n and start the process in the same shell, else follow the below mentioned.
If you want to see the trend of open file descriptors see it using ls /proc/<pid>/fd | wc -l, it will give you insight on how the process is doing with open files.
This issue seems like file descriptor leak, so to take an easy stab on it use lsof -p <pid> to see all open files, which can provide useful hint.
If nothing works, and the problem seems convoluted then go with valgrind profiling and use track-fd=yes flag, valgrind --track-fds=yes /path/to/cppBin, to see exact stacks causing fd leak.