Accessing the packet data including the IP header from a UDP packet in C#

The title is kind of self explanatory, I'd like to access the IP header from all UDP packet I'm receiving. Why? There are some flags I'd like to use to be able to de-fragment UDP packets and merge them back together, such as the MF flag (more fragments).

If there is a better way to merge UDP packets together, let me know, or if there's a way to access a packet's IP header without requiring administrative permissions, also let me know, or if I'm totally wrong on what I'm saying, let me know too.

What I've tried

Using a raw socket, had to use admin permissions, or flat out just crashed

What I didn't want to try

Pcap.Net, it's windows only as far as I know.

Solution

Turns out the OS does it for me, the actual max size is 64K, but if you need packets that long, you'd most likely need to rethink your whole system. Thanks for Steffen for letting me know, some people told me otherwise, and got me confused, so I hope this self-answer gets at least one less person confused.