https://glm.g-truc.net/0.9.4/api/a00210.html
I'm trying to find the signed angle between two vectors in 3d space.
I can't seem to find what the "reference vector" is in this function, mathematically. Therefore, I don't know how to use the function. Thanks!
There is no such concept as signed angle between two vectors in 3D. If there is no any selected direction, you can get only angle in range 0..Pi (for example, using scalar product, that is invariant to argument order).
Imagine that you drew two vectors A and B (starting from one point) on a glass sheet. Look from one side of glass - you see that you need rotate A by 45 degrees to B. Look from another side - - you see that you need rotate A by -45 degrees to B.
If you have got some reference vector (axis), it helps to define orientation of plane that contains two vectors (positive normal direction), and you can determine also angle sign.
For example, sequence of vectors A, B, Ref may form 'right' oriented triplet (imagine 1st, 2nd, 3rd thumbs on the right hand), so you should turn vector A in positive direction to make it coinciding with vector B, around axis Ref (if Ref is perpendicular both A and B).
Or this sequence may be 'left' triplet, so you should turn vector A in negative direction to make it coinciding with vector B.
(Sorry for my ugly English)