I'm working on a VR game, which has some climbing mechanics in it. The player will be able to grab onto edges and I want to use dynamic edge detection instead of what I'm doing right now, which is placing colliders on the edges that I want the player to be able to grab. Doing this takes a long time, and doesn't allow for any procedural mesh generation.
I know that something like this can be achived with shaders, but since they don't return any data, thats not an option. Are there any other ways I could do it? Even a way to mark the edges in the mesh instead of using colliders would be great.
Here's how I would do it. This is an opinionated take, not necessarily a best practice. Your mileage may vary.
Forget colliders unless your levels are small. They're too expensive here. I like to use colliders for classic physics rigidbody collisions. I don't find they're always the best match for motive interactions like walking or climbing etc, esp. in Unity, and often using a collider to detect a simple line-plane intersection is overkill.
Analyze your mesh (at runtime, say, during level load) and pull out grabbable edges/faces. Here's one way to do it. These will be navigation waypoints for your climb. Note that you have to define what constitutes a "grabbable" edge or face, which to me would mostly depend on the face orientation, which you can get from the face normal.
If your levels are large or complex, further group those faces into a volume tree structure (oct-tree, k-d, something to help you locate edges in the vicinity).
As the player climbs, test for intersections between the player's hand and nearby climbable faces. You may be able to rely on the player's actions here and perform your most time-consuming tests only when he "clutches" what he believes to be a climbable edge.
Under this system, climbing is essentially a mini-platformer. The hand must "jump" from one platform to another platform, enabling further vertical movement.