Let's say for example we have a routine
int unw_getcontext(unw_context_t *ucp)
The unw_getcontext() routine initializes the context structure pointed to by ucp with the machine-state of the call-site. The exact set of registers stored by unw_getcontext() is platform-specific
What must be a unit test for such a routine? Of course I can use some system calls to fetch CPU registers and stuff like this and then compare it with what the routine returns. But it seems to be so much work for a unit test. What does testing theory say about testing such API?
I would say it depends a bit more on the context.
You should test everything that you depend on or that you promise in the public API (also hide everything that's not tested).
If this method is part of your library you should make sure it does everything you say in the documentation. That is set registers to specific values and validate that the function returns them correctly.
If this is internal and you only depend on part of it it would be ok to only test that part and modify the documentation to say other things are not tested and should not be used (unless tested properly). Say you only need this to get the instruction pointer so you can jump back from some other part of the code, then only test this part of the behavior.
I wouldn't skip tests completely. The documentation correctly specifies that there's some platform specific component to it. It would be nice if there's a test to fail if the code requires something specific that is missing on some strange architecture. It can also serve as a nice template for users of more exotic architectures to add their own tests, making the code more robust overall.