I have different meshes with different VBOs, some may have normals, some not, etc. Every mesh also has its VAO with all VBOs being bound.
Then I draw all meshes with instancing. I plan to use a shared global VBO of mat4 to store dynamically calculated transformations every frame. Every VAO also needs to point at this shared VBO additionally. Also the number of every mesh instances may vary.
But I guess we want to reduce the amount of data uploading commands to GPU, that's why I want to accumulate all the matrices in a contiguous memory and send it in a single glBufferSubData command.
Different batches of different instanced meshes want to use different segments of the shared VBO to read the matrices from. So I need to update VAOs each frame as well.
The question is: how should I perform this in a better way? And is such an architecture actually a good one? I guess I should use glBindVertexBuffer for the shared VBO on each VAO, so I update the offset and size of the segments, and VAOs are lightweight, but is it really a standard solution?
You should not be concerned with updating VAOs. In fact, you should not have one VAO per mesh at all; have one VAO per vertex format (aka: the stuff set by glVertexAttribFormat and glEnable/DisableVertexAttrib), and try to make all of your meshes use the same vertex format. Setting buffer binding state is much cheaper than setting vertex format state.
So the idea ought to be that you bind a VAO for a vertex format, then iterate through all of the objects that use that format, using glBindVertexBuffer as needed for their individual data.