I sometimes forgot to do cleanup and afraid if the resources of them resides inside the GPU memory.
Things I use: shader programs, vertex array objects and buffer objects, textures
All OpenGL resources are supposed to be automatically released on destruction of OpenGL context (all shared contexts, to be precise). So practically, there should be no risk in leaking GPU memory when closing OpenGL context with some unreleased objects as far as your application does not trigger some video driver bug.
System will also take care on releasing all resources of closed application, even if some OpenGL contexts have been forgotten to be closed. Otherwise, it would be a total nightmare debugging 3D applications if GPU would keep allocated resources on an application crash.
To prove the idea - just write a simple test application allocating large portions of GPU memory (textures/VBOs) and track memory usage via external tools. Luckily, Task Manager in Windows 10 has been significantly improved and shows detailed GPU memory statistics.
From design point of view, however, it sounds like a bad idea tolerating incomplete clean-ups as the same release procedures used in other renderer code will cause real problems.