boost::managed_mapped_file cannot allocate all of grown space

Question

I'm trying to grow a memory mapped file, I successfully grow it, but I can't allocate all the extra space I've requested - I just get a std::bad_alloc instead.

Here is an example that shows the effect with g++ on Linux (I've seen the same on my 'real' code on MSVC too):

#include <memory>
#include <sstream>

#include <boost/interprocess/managed_mapped_file.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/containers/vector.hpp>
#include <boost/filesystem/operations.hpp>
#include <boost/uuid/uuid_io.hpp>
#include <boost/uuid/random_generator.hpp>

namespace
{
using MMapManager = boost::interprocess::basic_managed_mapped_file<
    char,
    boost::interprocess::rbtree_best_fit<boost::interprocess::null_mutex_family,
                                         boost::interprocess::offset_ptr<void>,
                                         16u>,
    boost::interprocess::iset_index>;

using MMapAllocatorType = boost::interprocess::allocator<
    std::size_t,
    MMapManager::segment_manager>;

using MMapContainerType = boost::interprocess::vector<
    std::size_t,
    MMapAllocatorType>;

// I've measured this at 256 bytes for my example configuration, but it's not
// documented anywhere, so let's overcompensate
constexpr auto ManagedFileOverhead = 1024u;

boost::filesystem::path getTemporaryFilePath()
{
    auto ss = std::stringstream{};
    ss << "MMap_test_" << boost::uuids::random_generator{}();

    return boost::filesystem::temp_directory_path() / ss.str();
}
}

int main()
{
    // Create memory mapped file, initially for 100 items
    auto capacity = 100u;
    const auto size = (capacity * sizeof(std::size_t)) + ManagedFileOverhead;
    const auto path = getTemporaryFilePath();

    auto file = std::make_unique<MMapManager>(
        boost::interprocess::create_only,
        path.string().c_str(),
        size);
    auto data = file->construct<MMapContainerType>("data_")(file->get_segment_manager());

    // Fill with stuff
    data->reserve(capacity);
    for (auto i = 0u; i < capacity; ++i) {
        data->push_back(i);
    }

    // Let's grow to hold 162 items (100 * golden ratio)
    capacity = 162u;
    const auto newFileSize = (capacity * sizeof(std::size_t)) + ManagedFileOverhead;
    const auto oldFileSize = boost::filesystem::file_size(path);
    const auto extraBytes = newFileSize - oldFileSize;

    // Unmap from the process, and grow
    file.reset();
    MMapManager::grow(path.string().c_str(), extraBytes);

    // Reopen it to re-map it into this process
    file = std::make_unique<MMapManager>(
        boost::interprocess::open_only,
        path.string().c_str());
    data = file->find<MMapContainerType>("data_").first;

    // Allocate it all
    data->reserve(capacity); // Bang, you're dead

    // Close down
    file.reset();
    boost::system::error_code ec;
    boost::filesystem::remove(path, ec);

    return EXIT_SUCCESS;
}

Setting the reserve (after growing) to 155 items works, just one more triggers a std::bad_alloc.

Why does this not work? Does growing incur extra management overhead within the mapped file causing me to run out of space earlier than expected?

Answer 1

You're simply assuming too much about the allocators.

Growing the mapped file will happen in place. Growing the vector does not. So, while you would need only extraBytes after increasing the reserved size, during the reserve you need enough space to hold both the old and the new allocations.

Prove it either by using:

MMapManager::grow(path.string().c_str(), oldFileSize + extraBytes);

Or by clearing the old container first:

{
    auto file = std::make_unique<MMapManager>(boost::interprocess::open_only, path.string().c_str());
    file->destroy<MMapContainerType>("data_");
    auto data = file->construct<MMapContainerType>("data_")(file->get_segment_manager());
}

MMapManager::grow(path.string().c_str(), extraBytes);