I know there are a lot of questions on how to do frequency or voltage scaling from a particular operating system, but I'm not interested in that. I'm from compilers background and would like to know how a compiler could perform frequency scaling if it has information on the application being compiled.
Specifically, I would like to know whether a specific assembly instruction is used to modify the frequency or voltage used by a cpu or is it something different.
Voltage and frequency scaling is typically controlled by a combination of hardware and software.
Beyond that it isn't possible to given a totally generic answer since the details are hardware-specific, so most of the below applies to Intel x86 hardware. Other "big" CPUs are likely similar, but something like a microcontroller is going to be quite different.
So the "software" part of the above might give you some hope that you can control it from your application, but the software side of things is generally restricted to the kernel, or in some cases processes running as root. In particular, most of the frequency control on recent Intel can be adjusted using MSR registers, and you could access this from userland using the msr kernel module and the rdmsr and wrmsr commands.
This isn't a recommended way of doing things, because you are changing MSRs that the kernel is also setting, so things may go haywire, your changes may be arbitrarily undone, etc.
The recent trend in x86 is to push more of the "frequency control" loop into hardware, which can respond more quickly and has access to fine-grained metrics about application behavior that the application doesn't. On Intel this is the so-called HWP (hardware p-states) and is embodied in the intel_pstate drive for Linux (the default on recent Intel) and similar stuff on Windows. Here, software is relegated to giving more general hints to hardware, but generally doesn't control the frequency directly.
Finally, about voltage specifically: this usually isn't even directly controllable in a general way on x86 - you set what you can via the MSRs, and the voltage is set by the hardware to an appropriate level. Certain motherboards or "enthusiast" CPUs may offer drivers that can set the voltage directly at runtime, but these are far from universal.