Upon trying to write to a piece of (portable) C++ code that uses shared_memory_segment in order to write to "shared memory", I encountered boost::interprocess::bad_alloc several times. From the Boost documentation:
This exception is thrown when a memory request can't be fulfilled.
So, I must be allocating too little memory. Here follows the code (only for writing, since reading here is irrelevant):
shared_memory.h:
#include <boost/interprocess/managed_shared_memory.hpp>
#include <boost/interprocess/containers/vector.hpp>
#include <boost/interprocess/containers/string.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/sync/named_mutex.hpp>
#include <string>
#include <exception>
namespace my_shared_memory
{
typedef boost::interprocess::allocator<char, boost::interprocess::managed_shared_memory::segment_manager> CharAllocator;
typedef boost::interprocess::basic_string<char, std::char_traits<char>, CharAllocator> IPCString;
typedef boost::interprocess::allocator<IPCString, boost::interprocess::managed_shared_memory::segment_manager> StringAllocator;
typedef boost::interprocess::vector<IPCString, StringAllocator> ShmVector;
bool write_to_memory(std::string wsuid, std::string loop_val, std::string should_intercept, std::string post_data) ;
const std::string shm_prefix = "shm_";
const std::string mutex_prefix = "mtx_";
}
shared_memory.cpp:
#include "shared_memory.h"
namespace apl_shared_memory
{
bool write_to_memory(std::string wsuid, std::string loop_val, std::string should_intercept, std::string post_data)
{
bool ret_val;
std::string shm_name = shm_prefix + wsuid;
std::string mtx_name = mutex_prefix + wsuid;
boost::interprocess::named_mutex named_mtx{boost::interprocess::open_or_create, mtx_name.c_str()};
size_t size = (sizeof(loop_val) + loop_val.size() + sizeof(should_intercept) + should_intercept.size() + sizeof post_data + post_data.size()) * 5;
try
{
named_mtx.lock();
boost::interprocess::shared_memory_object::remove(shm_name.c_str());
boost::interprocess::managed_shared_memory segment(boost::interprocess::create_only, shm_name.c_str(), size);
CharAllocator charallocator (segment.get_segment_manager());
StringAllocator stringallocator(segment.get_segment_manager());
IPCString shm_loop_val(charallocator);
IPCString shm_should_intercept(charallocator);
IPCString shm_intercepted_data(charallocator);
shm_loop_val = loop_val.c_str();
shm_should_intercept = should_intercept.c_str();
shm_intercepted_data = post_data.c_str();
segment.destroy<ShmVector>("ShmVector");
ShmVector *shmVector = segment.construct<ShmVector>("ShmVector")(stringallocator);
shmVector->clear();
shmVector->push_back(shm_loop_val);
shmVector->push_back(shm_should_intercept);
shmVector->push_back(shm_intercepted_data);
named_mtx.unlock();
ret_val = true;
} catch(const std::exception& ex)
{
ret_val = false;
named_mtx.unlock();
boost::interprocess::shared_memory_object::remove(shm_name.c_str());
}
named_mtx.unlock();
return ret_val;
}
}
Yes, I realize I don't need unlock calls on all three places.
Problem seems to be in the line:
size_t size = (sizeof(loop_val) + loop_val.size() + sizeof(should_intercept) + should_intercept.size() + sizeof post_data + post_data.size()) * 5;
I thought it was overkill to add times 5 but that is apparently not the case. This code works fine on Debian 9 with gcc 6.3.0 and Windows 10 with Microsoft Visual Studio 2015 Community. However, on MacOS with Xcode 10.1, I get boost::interprocess::bad_alloc when I try to insert even the first element in the vector. The solution seems to multiply the size with 10 instead of 5 but this just appears wasteful.
I added segment.get_free_memory calls and determined that for a vector that contains the following three strings (without the quotes)
"true" "true" ""
I need 512 bytes with Xcode. The sizeof operator returned 32, 24 and 32 for IPCString, std::string and ShmVector, respectively. Therefore, it seems that I need 32+4 bytes for one "true" string and 32 for the empty string. Adding the ShmVector there, I need 36+36+32+32=136 bytes for the structures themselves. I calculated the size using sizeof(std::string), which is 24 here (an oversight), so that gives me (28+28+24)*5 = 400, which is not enough here.
My question here is how to determine how much memory I need for the segment? The data I want to write to the segment is known when the function is called and thus is its size.
EDIT: I changed the size to be:
size_t size = (sizeof(IPCString) + loop_val.size() + sizeof(IPCString) + should_intercept.size() + sizeof(IPCString) + post_data.size() + sizeof(ShmVector)) * 2 + 500;
So far, so good. I always have less than 500 bytes of free space after I'm finished writing to the segment. I've tried with various data sizes, ranging from less than 100 bytes to 3 megabytes. I've tried without multiplying by 2 but, in that case, when I try to insert large data chunks, the program crashes as 500 additional bytes does not provide sufficient leeway.
If there is no answer in a couple of days, I'll post this edit as an answer.
After some research and testing, I came up with the solution below. The key point here is to use boost::move (or, perhaps std::move, don't know if there will be difference here). Because we're using boost::move, there will be no copies of the IPCString objects and, therefore, no need to multiply by 2. Remember, because of how we defined IPCString, it will be allocated inside the shared memory segment.
Added overhead is necessary (perhaps because of alignment or some other overhead that's added or both) but there is always some space left. Because I could happen to send several megabytes, the overhead of 500 bytes is pretty small.
I am using std::string's size() method because it returns the size of the string in bytes.
The code follows:
shared_memory.h:
#pragma once
#include <boost/interprocess/managed_shared_memory.hpp>
#include <boost/interprocess/containers/vector.hpp>
#include <boost/interprocess/containers/string.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/sync/named_mutex.hpp>
#include <string>
#include <vector>
#include <exception>
namespace apl_shared_memory
{
typedef boost::interprocess::allocator<char, boost::interprocess::managed_shared_memory::segment_manager> CharAllocator;
typedef boost::interprocess::basic_string<char, std::char_traits<char>, CharAllocator> IPCString;
typedef boost::interprocess::allocator<IPCString, boost::interprocess::managed_shared_memory::segment_manager> StringAllocator;
typedef boost::interprocess::vector<IPCString, StringAllocator> ShmVector;
bool write_to_memory(std::string wsuid, std::string loop_val, std::string should_intercept, std::string post_data) ;
const std::string shm_prefix = "shm_";
const std::string mutex_prefix = "mtx_";
}
shared_memory.cpp:
#include "shared_memory.h"
#include "logger.h"
namespace apl_shared_memory
{
bool write_to_memory(std::string wsuid, std::string loop_val, std::string should_intercept, std::string post_data)
{
bool ret_val;
std::string shm_name = shm_prefix + wsuid;
std::string mtx_name = mutex_prefix + wsuid;
boost::interprocess::named_mutex named_mtx{ boost::interprocess::open_or_create, mtx_name.c_str() };
// Add the size of all the structures we're putting in the shared memory region and add some for the overhead.
size_t size = (3*sizeof(IPCString) + loop_val.size() + should_intercept.size() + post_data.size() + sizeof(ShmVector)) + 500;
try
{
named_mtx.lock();
boost::interprocess::shared_memory_object::remove(shm_name.c_str());
boost::interprocess::managed_shared_memory segment(boost::interprocess::create_only, shm_name.c_str(), size);
CharAllocator charallocator(segment.get_segment_manager());
StringAllocator stringallocator(segment.get_segment_manager());
IPCString shm_loop_val(charallocator);
IPCString shm_should_intercept(charallocator);
IPCString shm_intercepted_data(charallocator);
shm_loop_val = loop_val.c_str();
shm_should_intercept = should_intercept.c_str();
shm_intercepted_data = post_data.c_str();
segment.destroy<ShmVector>("ShmVector");
ShmVector *shmVector = segment.construct<ShmVector>("ShmVector")(stringallocator);
shmVector->clear();
shmVector->reserve(3);
// push_back will call a copy-constructor. But, if we send a rvalue reference (i.e. if we move it), there will be no copying.
shmVector->push_back(boost::move(shm_loop_val));
shmVector->push_back(boost::move(shm_should_intercept));
shmVector->push_back(boost::move(shm_intercepted_data));
ret_val = true;
}
catch (const std::exception& ex)
{
ret_val = false;
boost::interprocess::shared_memory_object::remove(shm_name.c_str());
}
named_mtx.unlock();
return ret_val;
}
}
If anyone believes I did something wrong, please comment.