I have a dynamic library (libdatamodel.so) file that is statically linked with my new/delete overrides. There are a handful of allocations that are not getting picked up with my overrides and I'm not sure why.
I used a debugger to get a stacktrace from one of the calls which should be using my overrides, but is not.
Stacktrace:
thread #124, name = 'run-vizqlserver', stop reason = breakpoint 7.1
frame #0: 0x00007fa70f1e9800 libc.so.6`__GI___libc_malloc
frame #1: 0x00007fa70fad6e88 libstdc++.so.6`operator new(sz=952) at new_op.cc:50
frame #2: 0x00007fa5c5701fed libdatamodel.so`void std::vector<FieldName, std::allocator<FieldName> >::_M_assign_aux<boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > > >(boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > >, boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > >, std::forward_iterator_tag) [inlined] __gnu_cxx::new_allocator<FieldName>::allocate(unsigned long, void const*) at new_allocator.h:104
frame #3: 0x00007fa5c5701fcb libdatamodel.so`void std::vector<FieldName, std::allocator<FieldName> >::_M_assign_aux<boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > > >(boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > >, boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > >, std::forward_iterator_tag) [inlined] std::allocator_traits<std::allocator<FieldName> >::allocate(std::allocator<FieldName>&, unsigned long) at alloc_traits.h:360
frame #4: 0x00007fa5c5701fcb libdatamodel.so`void std::vector<FieldName, std::allocator<FieldName> >::_M_assign_aux<boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > > >(boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > >, boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > >, std::forward_iterator_tag) [inlined] std::_Vector_base<FieldName, std::allocator<FieldName> >::_M_allocate(unsigned long) at stl_vector.h:170
frame #5: 0x00007fa5c5701fcb libdatamodel.so`void std::vector<FieldName, std::allocator<FieldName> >::_M_assign_aux<boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > > >(boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > >, boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > >, std::forward_iterator_tag) [inlined] FieldName* std::vector<FieldName, std::allocator<FieldName> >::_M_allocate_and_copy<boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > > >(unsigned long, boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > >, boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > >) at stl_vector.h:1224
frame #6: 0x00007fa5c5701fcb libdatamodel.so`void std::vector<FieldName, std::allocator<FieldName> >::_M_assign_aux<boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > > >(this=0x00007fa4cc9bdd30, __first=bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > > @ rbx, __last=bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > > @ r15, (null)=<unavailable>) at vector.tcc:273
frame #7: 0x00007fa5c56ff372 libdatamodel.so`AbstractQuery::GetOutputFields() const [inlined] void std::vector<FieldName, std::allocator<FieldName> >::_M_assign_dispatch<boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > > >(this=<unavailable>, __first=<unavailable>, __last=<unavailable>) at stl_vector.h:1336
frame #8: 0x00007fa5c56ff36a libdatamodel.so`AbstractQuery::GetOutputFields() const [inlined] void std::vector<FieldName, std::allocator<FieldName> >::assign<boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<FieldName, std::allocator<FieldName> > > >, void>(__first=<unavailable>, __last=<unavailable>) at stl_vector.h:508
frame #9: 0x00007fa5c56ff36a libdatamodel.so`AbstractQuery::GetOutputFields() const [inlined] AbstractQuery::GetOutputVector(this=<unavailable>, c=<unavailable>) const at AbstractQuery.cpp:210
Is this a problem with std::vector not calling my new() or is it something to do with boost inlining the call to the allocator? I am unable to make heads or tails of the boost template types.
For completeness, here is how I'm confirming that my new/delete are really defined in my library
$ nm -C libdatamodel.so | egrep "operator (new|delete)"
000000000017f1c0 T operator delete[](void*)
000000000017f1d0 T operator delete[](void*, std::nothrow_t const&)
000000000017f1a0 T operator delete(void*)
000000000017f1b0 T operator delete(void*, std::nothrow_t const&)
000000000017f150 T operator new[](unsigned long)
000000000017f190 T operator new[](unsigned long, std::nothrow_t const&)
000000000017f100 T operator new(unsigned long)
000000000017f140 T operator new(unsigned long, std::nothrow_t const&)
After a lot of experimentation and research I found two or three good solutions to this problem. My original issue is that the memory overriding that worked on Windows did not work on Linux. Windows will resolve any symbols that can be resolved within the library before the runtime loader is even in the picture. On Linux this isn't true unless you use one of the -Bsymobolic linker flags.
I chose not to alter the default linking for fear of unintended side effects. So instead I moved my memory overrides to their own library and linked them directly to the executable. Since my original library is opened with dlopen the runtime loader puts it at the back of the list for symbol resolution. Linking my memory overrides directly to the exe puts them high on the list and everything works as intended. I could have also used LD_PRELOAD to achieve this goal, but opted to do it all at build time.